Building pipelines from MariaDB
You can read from MariaDB and write to any target supported by Striim. Typically, you will set up data pipelines that read from MariaDB in two phases—initial load, followed by continuous replication—as explained in this concept article on Pipelines.
For initial load, you can use Database Reader to create a point-in-time copy of the existing source MariaDB dataset at the target, as described in MariaDB initial load.
After initial load has completed, you can start continuous replication by continuously reading the new data created in the same Oracle dataset after the initial load was started, and then writing this new source data to the target.
For real-time continuous replication of new source data using CDC, you can use MariaDB Reader or MariaDB Xpand Reader, as described in MariaDB continuous real-time replication using CDC.
If you choose not to enable CDC on the source database, you can use Incremental Batch Reader to read the new source data at regular intervals, as described in MariaDB continuous incremental replication, allowing for continuous updates in near real time. Incremental Batch Reader differs from the CDC readers in several ways, including not capturing DELETE operations at the source (see Differences between real-time and incremental replication).
You can use automated pipeline wizards to build data pipelines from MariaDB for which Striim manages the lifecycle, or you can manually create separate applications for initial load and continuous replication and manage the lifecycle of the pipeline yourself.
Before building a pipeline, you must complete the steps described in MariaDB initial setup.
Using an automated pipeline wizard: if you want to build real-time data pipelines from MariaDB using CDC and write to a supported target, we recommend that you use an automated pipeline wizard with a MariaDB or MariaDB Xpand source. These wizards perform all the following steps automatically:
Create two applications: one for initial load and the other for CDC-based continuous replication.
Create a schema and tables in the target matching those to be synced with the source.
Run the initial load application to perform an initial load of existing data from the source to the target.
When initial load completes, run the CDC application to replicating replicate new data using CDC.
Not using an automated pipeline wizard: if your use case or policies do not allow using an automated pipeline, create separate applications for initial load and continuous replication:
Before performing initial load, get the current global transaction ID as described in Switching from initial load to continuous replication of MariaDB sources.
Create a schema and tables in the target and perform initial load: use a wizard with a Database Reader source. (Alternatively you may do this with a native or third-party utility, such as mysqldump.)
Perform an initial load when the schema and tables already exist in the target: use a wizard with a Database Reader source.
Switch from initial load to continuous replication: see Switching from initial load to continuous replication of MariaDB sources.
Replicate new data: use a wizard with a MariaDB or MariaDB Xpand source (for continuous real-time replication) or an Incremental Batch Reader source (for continuous incremental replication).
Alternatively, instead of using wizards, you can create applications using Flow Designer, TQL, or Striim's REST API.