Additional Settings
Note
The options on this page cannot be changed after you start the pipeline.
If BigQuery is running in a free trial or the Google Cloud Platform free tier, uncheck Use streaming mode.
How do you want to write changes to BigQuery?
Write continuous changes as audit records (default; also known as APPEND ONLY mode): BigQuery retains a record of every operation in the source. For example, if you insert a row, then update it, then delete it, BigQuery will have three records, one for each operation in the source (INSERT, UPDATE, and DELETE). This is appropriate when you want to be able to see the state of the data at various points in the past, for example, to compare activity for the current month with activity for the same month last year.
With this setting, Striim will add two additional columns to each table, STRIIM_OPTIME, a timestamp for the operation, and STRIIM_OPTYPE, the event type, INSERT, UPDATE, or DELETE. Note: on initial sync with SQL Server, all STRIIM_OPTYPE values are SELECT.
Write continuous changes directly (also known as MERGE mode): BigQuery tables are synchronized with the source tables. For example, if you insert a row, then update it, BigQuery will have only the updated data. If you then delete the row from the source table, BigQuery will no longer have any record of that row.
How would you like to handle schema changes?
NOTE: In this release, this feature is not supported when the source is MariaDB, Oracle CDB, Oracle PDB, Oracle 19c, or any version of SQL Server, so for pipelines with those sources the option will not appear.
Select what you want Striim to do in BigQuery when a table or column is added to or a table is dropped from the source database:
Do not propagate changes and continue (default): Striim will take no action. Any data added to new tables will not be synced to BigQuery. Any data added to a new column will not be synced to BigQuery as the column will not exist in the target. Tables dropped from the source will continue to exist in BigQuery.
Pause the pipeline: Striim will pause the pipeline. After making any necessary changes in the source or BigQuery, restart the pipeline.
Propagate changes to BigQuery: In BigQuery, Striim will create a new table, add a column, or drop a table so that the target matches the source. Sync will continue without interruption. (Note that if a column is dropped from a source table, it will not be dropped from the corresponding BigQuery target table.)
Use streaming mode to write continuous changes to BigQuery?
Use streaming mode(enabled by default): We recommend using this method when you need low latency. If your uploads are infrequent (for example, once an hour, you may wish to disable streaming mode. If BigQuery is running in a free trial or the Google Cloud Platform free tier, disable this option or upgrade to a paid account.
What is your PostgreSQL replication slot?
Appears only when your source is PostgreSQL.
Enter the name of the slot you created or chose in Set up your PostgreSQL source. Note that you cannot use the same slot in two pipelines, each must have its own slot.
Schema evolution tracker table
If for "How would you like to handle schema changes" you selected Propagate changes to BigQuery, enter the name of the table you created or chose in Set up your PostgreSQL source.