Skip to main content

Incremental Batch Reader

You can use Incremental Batch Reader to get new data from a supported database or data warehouse. The most common use case for Incremental Batch Reader is to continuously update targets with new source data after initial load using Database Reader has completed, particularly when a CDC reader is not available for the source or when you choose not to use the available CDC reader.

For property references, see:

To read incrementally from Spanner, see Spanner Batch Reader.

Three properties control incremental reading:

  • Check Column specifies the column Incremental Batch Reader will use to identify which data is new.

  • Start Position specifies the Check Column value from which reading will start.

  • Polling Interval controls how often Incremental Batch Reader checks for new data. The first batch will begin with the Start Position value. Each successive batch will begin with the table rows immediately after the last row included in the previous batch.

Warning

Differences between Incremental Batch Reader and CDC readers

SQL CDC readers capture INSERT, UPDATE, and DELETE operations, so if a writer supports Merge mode, all changes in the source can be replicated in the target. Incremental Batch Reader treats both INSERT and UPDATE operations as inserts and does not capture deletes, so a writer in Merge mode will not be able to replicate all changes to the source in the target, and in Append Only mode deletes in the source will not be captured in the target.

Incremental Batch Reader sample code

The following would read rows from TABLE1 with UUID column values equal to or greater than 1234: 

CREATE SOURCE OraSource USING IncrementalBatchReader  ( 
  Username: 'striim',
  Password: '********',
  ConnectionURL: '192.0.2.:1521:orcl',
  Tables: 'MYSCHEMA.TABLE1',
  CheckColumn: 'MYSCHEMA.TABLE1=UUID',
  StartPosition: 'MYSCHEMA.TABLE1=1234'
) 
OUTPUT TO OraSourceOutput;

If IncrementalBatchReader sends duplicate records to a DatabaseWriter target, by default the application will terminate. This can happen, for example, when recovery is enabled (see Recovering applications), there are multiple rows with the same CheckColumn timestamp, and only some of them were written before a system failure,  To avoid this, specify the appropriate IgnorableException in the target: for example, for CosmosDBWriter, RESOURCE_ALREADY_EXISTS.Recovering applications