Building pipelines from Db2 for z/OS
You can read from Db2 for z/OS as follows, and write to any target supported by Striim. Typically, you will set up data pipelines that read from Db2 for z/OS in two phases—initial load, followed by continuous replication—as explained in this concept article on Pipelines.
For initial load, you can use Database Reader to create a point-in-time copy of the existing source Db2 for z/OS dataset at the target, as described in Db2 for z/OS initial load.
After the initial load has completed, you can start continuous replication by continuously reading the new data created in the same source Db2 dataset after the initial load was started, and then writing this new source data to the target.
For real-time continuous replication of new source data using CDC, you can use Striim Connect as described in Db2 for z/OS continuous real-time replication.
If you choose not to enable CDC on the source database, you can use Incremental Batch Reader to read the new source data at regular intervals, allowing for continuous updates in near real time. Using Incremental Batch Reader differs from the Striim Connect approach in several ways, including not capturing DELETE operations at the source (see Differences between real-time and incremental replication).
Before building a pipeline, you must complete the steps described in Db2 for z/OS initial setup.
Striim does not have automatic pipeline wizards for Db2 for z/OS sources, so you must create separate applications for initial load and continuous replication, and after initial load completes, manually switch to continuous replication.
Create a schema and tables in the target and perform initial load: use a wizard with a Db2 for z/OS source.
Replicate new data: for continuous real-time replication, see Db2 for z/OS continuous real-time replication; for continuous incremental replication, use a wizard with an Incremental Batch Reader source.
Alternatively, instead of using wizards, you can create applications using Flow Designer, TQL, or Striim's REST API.
Db2 for z/OS CDC architecture overview
Striim provides change data capture (CDC) support for IBM Db2 for z/OS through its integration with Striim Agent for Db2 for z/OS. Striim Agent for Db2 for z/OS captures committed DML operations from Db2 for z/OS and streams them in Avro-serialized format to a Kafka topic for downstream processing.
As shown in the diagram above, Striim Agent for Db2 for z/OS captures the data changes from the source Db2 server, converts the source transaction to wire format, serializes it into binary as an Apache Avro message, and publishes it to an Apache Kafka topic in conjunction with a Confluent Schema Registry. Striim Connect retrieves relevant messages from the Kafka topic and uses the Avro schemas from the Schema Registry to deserialize them.
Striim Agent for Db2 for z/OS also utilizes either a PostgreSQL or Oracle database as a repository for metadata about input and output objects, as well as for configuring the replication workflows and their components. This may be the same PostgreSQL or Oracle database used by Striim for its metadata repository. (Derby is not supported as an Striim Agent for Db2 for z/OS metadata repository host.)