Spanner PG Dialect Writer

Batch Policy

String

eventCount: 1000, Interval: 60s

The batch policy includes eventcount and interval (see Setting output names and rollover / upload policies for syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, all remaining events are sent to the target.

With the default setting, data will be written every 60 seconds or sooner if the buffer accumulates 1000 events.

When Preserve Source Transaction Boundaries is False, Primary key updates are sent as a single-event batch. When Preserve Source Transaction Boundaries is True, a batch with many primary key updates may take some extra time to execute as a read row operation must be performed.

When the batch policy event count or interval is reached, or Spanner's "Mutations per commit" limit (see Spanner > Documentation > Resources> Quotas & limits > Limits for creating, reading, updating, and deleting data) is reached, a transaction is created (see Spanner > Documentation > Guides > Transactions overview.

When Preserve Source Transation Boundary is True, you may disable this property with the setting -1 (see Using Preserve Source Transaction Boundary with Spanner Writer).

CDDL Action

String

Process

See Handling schema evolution.

Checkpoint Table

String

public.chkpoint

To support recovery (see Recovering applications, a checkpoint table using the following DDL must exist in the target database:Recovering applications

CREATE TABLE public.chkpoint (
  "ID" character varying NOT NULL,
  "SOURCEPOSITION" bytea,
  PRIMARY KEY ("ID") );

If the table does not exist and recovery is enabled, Striim will create the chkpoint table in the public schema automatically.

If necessary you may use different schema and/or table names, in which case change the value of this property. The schema and table names must be lowercase.

Database Name

String

The name of the Spanner database.

Excluded Tables

String

If Tables uses a wildcard, data from any tables specified here will be omitted. Multiple table names (separated by semicolons) and wildcards may be used exactly as for Tables.

Ignorable Exception Code

String

By default, if the target DBMS returns an error, Striim terminates the application. Use this property to specify one or more error codes (see Cloud Spanner > Documentation > Reference > Code) to ignore, separated by semicolons, for example, NOT_FOUND;ALREADY_EXISTS. (You may also specify error numbers from legacy documentation.)

Ignored exceptions will be written to the application's exception store (see CREATE EXCEPTIONSTORE).

This property is ignored when Preserve Source Transaction Boundary is True.

Instance ID

String

Specify the instance ID for the databases containing the tables to be written to. (Note: the instance ID may not be the same as the instance name.)

Parallel Threads

Integer

See Creating multiple writer instances (parallel threads).

Preserve Source Transaction Boundary

Boolean

False

See Using Preserve Source Transaction Boundary with Spanner Writer.

Private Service Connect Endpoint

String

If using Private Service Connect with Google Spanner, specify the endpoint created in the target Virtual Private Cloud, as described in Private Service Connect support for Google cloud adapters.

Project ID

String

To use a service account key other than the one associated with the Spanner instance's project, specify its project ID here. Otherwise leave blank.

Service Account Key

String

The path (from root or the Striim program directory) and file name to the .json credentials file downloaded from Google (see Service Accounts). This file must be copied to the same location on each Striim server that will run this adapter, or to a network location accessible by all servers. The associated service account must have the Cloud Spanner Database User or higher role for the instance (see Cloud Spanner Roles).

To use a service account key other than the one associated with the Spanner instance's project, specify a value for the Project ID property.

Tables

String

Specify the name(s) of the table(s) to write to, in the format <schema>.<table>. Tables are created in the database specified in Database Name.

All tables must have primary keys.

If the source is Database Reader and its Create Schema property is True, matching schemas will be created in Spanner if they do not exist already (see the discussion of Create Schema in Database Reader properties for more details). Otherwise, the schemas and tables must exist in the target when the application is started, and if a specified target table does not exist, the application will terminate with an error. To skip writes to missing tables without terminating, specify NOT_FOUND as an Ignorable Exception Code.

The target table name(s) specified here must match the case shown in the Spanner UI. See Naming conventions.

When the target's input stream is a user-defined event, specify a single table.

When the input stream of the target is the output of a DatabaseReader, IncrementalBatchReader, or SQL CDC source (that is, when replicating data from one database to another), it can write to multiple tables. In this case, specify the names of both the source and target tables. You may use the % wildcard only for tables, not for schemas or databases. If the reader uses three-part names, you must use them here as well. Note that Oracle CDB/PDB source table names must be specified in two parts when the source is Database Reader or Incremental Batch reader (schema.%,schema.%) but in three parts when the source is Oracle Reader or OJet ((database.schema.%,schema.%). Note that SQL Server source table names must be specified in three parts when the source is Database Reader or Incremental Batch Reader (database.schema.%,schema.%) but in two parts when the source is MS SQL Reader or MS Jet (schema.%,schema.%). Examples:

source.emp,target.emp
source.db1,target.db1;source.db2,target.db2
source.%,target.%
source.mydatabase.emp%,target.mydb.%
source1.%,target1.%;source2.%,target2.%

When a target table has a commit timestamp column, by default its value will be Spanner's current system time when the transaction is committed. To use a different value, use ColumnMap. For example, to use the time the source transaction was committed in Oracle: ORADB1.%,spandb1.% ColumnMap (Ts @metadata(DBCommitTimestamp))

See Mapping columns for additional options.