Building pipelines from SQL Server
You can read from SQL Server as follows, and write to any target supported by Striim. Typically, you will set up data pipelines that read from SQL Server in two phases—initial load, followed by continuous replication—as explained in this concept article on Pipelines.
For initial load, you can use Database Reader to create a point-in-time copy of the existing source SQL Server dataset at the target, as described in SQL Server initial load.
After initial load has completed, you can start continuous replication by continuously reading the new data created in the same SQL Server dataset after the initial load was started, and then writing this new source data to the target.
For real-time continuous replication of new source data using CDC, you can use MSSQL Reader or MSJet, as described in SQL Server continuous real-time replication using CDC.
If you choose not to enable CDC on the source database, you can use Incremental Batch Reader to read the new source data at regular intervals, as described in SQL Server continuous incremental replication, allowing for continuous updates in near real time. Incremental Batch Reader differs from the CDC readers in several ways, including not capturing DELETE operations at the source (see Differences between real-time and incremental replication).
You can use automated pipeline wizards to build data pipelines from SQL Server for which Striim manages the lifecycle, or you can manually create separate applications for initial load and continuous replication and manage the lifecycle of the pipeline yourself.
Before building a pipeline, you must complete the steps described in SQL Server initial setup.
Using an automated pipeline wizard: if you want to build real-time data pipelines from SQL Server using CDC and write to a supported target, we recommend that you use an automated pipeline wizard with a SQL Server (MSSQL Reader) or MSJet source. These wizards perform all the following steps automatically:
Create two applications: one for initial load and the other for CDC-based continuous replication.
Create a schema and tables in the target matching those to be synced with the source.
Run the initial load application to perform an initial load of existing data from the source to the target.
When initial load completes, run the CDC application to replicating replicate new data using CDC.
Not using an automated pipeline wizard: if your use case or policies do not allow using an automated pipeline, create separate applications for initial load and continuous replication:
Before performing initial load, get the current maximum log sequence number as described in Switching from initial load to continuous replication of SQL Server sources.
Create a schema and tables in the target and perform initial load: use a wizard with a Database Reader source. (Alternatively you may do this with a native or third-party utility, such as the SQL Server Import and Export Wizard.)
Perform an initial load when the schema and tables already exist in the target: use a wizard with a Database Reader source.
Switch from initial load to continuous replication: see Switching from initial load to continuous replication of SQL Server sources.
Replicate new data: use a wizard with an SQL Server (MSSQL Reader) or MSJet source (for continuous real-time replication) or an Incremental Batch Reader source (for continuous incremental replication).
Alternatively, instead of using wizards, you can create applications using Flow Designer, TQL, or Striim's REST API.