Striim for BigQuery Documentation

Install the wal2json plugin for the operating system of your PostgreSQL host as described in https://github.com/eulerto/wal2json.

Edit postgressql.conf, set the following options, and save the file. The values for max_replication_slots and max_wal_senders may be higher but there must be one of each available for each instance of PostgreSQL Reader. max_wal_senders cannot exceed the value of max_connections.

wal_level = logical
max_replication_slots = 1
max_wal_senders = 1

Edit pg_hba.conf and add the following records, replacing <IP address> with the Striim server's IP address. If you have a multi-node cluster, add a record for each server that will run PostgreSQLReader. Then save the file and restart PostgreSQL.

local  replication  striim  <IP address>/0  trust
local  replication  striim                  trust

Restart PostgreSQL.

Enter the following command to create the replication slot (the location of the command may vary but typically is /usr/local/bin in Linux or C:\Program Files\PostgreSQL\<version>\bin\ in Windows.

pg_recvlogical -d mydb --slot striim_slot --create-slot -P wal2json

If you plan to use multiple instances of PostgreSQL Reader, create a separate slot for each.

Create a role with the REPLICATION attribute for use by Striim and give it select permission on the schema(s) containing the tables to be read. Replace ****** with a strong password and myschema with the name of your schema.

CREATE ROLE striim WITH LOGIN PASSWORD '******' REPLICATION;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO striim;

Go to your RDS dashboard, select Parameter groups > Create parameter group.

For the Parameter group family, select the aurora-postgresql item that matches your PostgreSQL compatibility option (for example, for PostgreSQL 11, select aurora-postgresql11).

For Type, select DB Cluster Parameter Group.

For Group Name and Description, enter aurora-logical-decoding, then click Create.

Click aurora-logical-decoding.

Enter logical_ in the Parameters field to filter the list, click Modify, set rds.logical_replication to 1, and click Continue > Apply changes.

In the left column, click Databases, then click the name of your Aurora cluster, click Modify, scroll down to Database options (you may have to expand the Additional configuration section), change DB cluster parameter group to aurora-logical-decoding, then scroll down to the bottom and click Continue.

Select Apply immediately > Modify DB instance. Wait for the cluster's status to change from Modifying to Available, then stop it, wait for the status to change from Stopping to Stopped, then start it.

In PSQL, enter the following command to create the replication slot:

SELECT pg_create_logical_replication_slot('striim_slot', 'wal2json');

Create a role with the REPLICATION attribute for use by PostgreSQLReader and give it select permission on the schema(s) containing the tables to be read. Replace ****** with a strong password and (if necessary) public with the name of your schema.

CREATE ROLE striim WITH LOGIN PASSWORD '******';
GRANT rds_replication TO striim;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO striim;

Go to your RDS dashboard, select Parameter groups > Create parameter group, enter posstgres-logical-decoding as the Group name and Description, then click Create.

Click postgres-logical-decoding.

Enter logical_ in the Parameters field to filter the list, click Modify, set rds.logical_replication to 1, and click Continue > Apply changes.

In the left column, click Databases, then click the name of your database, click Modify, scroll down to Database options (you may have to expand the Additional configuration section), change DB parameter group to postgres-logical-decoding, then scroll down to the bottom and click Continue.

Select Apply immediately > Modify DB instance. Wait for the database's status to change from Modifying to Available, then reboot it and wait for the status to change from Rebooting to Available.

In PSQL, enter the following command to create the replication slot:

SELECT pg_create_logical_replication_slot('striim_slot', 'wal2json');

Create a role with the REPLICATION attribute for use by PostgreSQLReader and give it select permission on the schema(s) containing the tables to be read. Replace ****** with a strong password and (if necessary) public with the name of your schema.

CREATE ROLE striim WITH LOGIN PASSWORD '******';
GRANT rds_replication TO striim;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO striim;