Skip to main content

Building pipelines from MongoDB

You can read from MongoDB as follows, and write to any Striim target that can accept a JSONNodeEvent input stream. Typically, you will set up data pipelines that read from MongoDB in two phases—initial load, followed by continuous replication—as explained in this concept article on Pipelines.

  • For initial load, use MongoDB Reader in Initial Load mode to create a point-in-time copy of the existing source MongoDB dataset in the target, as described in MongoDB initial load.

  • After initial load has completed, you can start continuous replication by continuously reading the new data created in the same MongoDB dataset after the initial load was started, and then writing this new source data to the target. For real-time continuous replication of new source data using CDC, use MongoDB Reader in Incremental mode (as described in MongoDB continuous real-time replication using CDC). With MongoDB 4.1 and earlier, it will read change data from the oplog; with 4.2 or later, it will read from change streams.

Before building a pipeline, you must complete the steps described in MongoDB initial setup.

Striim does not have automatic pipeline wizards or support initial schema creation for MongoDB sources, so you must manually create the schemas and tables or collections in the target before starting the initial load, then manually switch to continuous replication. Since MongoDB Reader supports both initial load and continuous replication, you can handle both with a single application.

  1. Create a schema and tables or containers in the target corresponding to the database and containers in your MongoDB source.

  2. MongoDBToCosmosDBWIzard.png

    Perform an initial load: create an application using a wizard with a MongoDB source and select Initial load only as the type of app to create. If your target is not among the target choices when using a wizard, create the application using the Flow Designer, TQL, or Striim's REST API, setting Mode to InitialLoad and Quiesce on IL Completion to True, and enabling recovery.

  3. Switch from initial load to continuous replication: when initial load completes, undeploy the application, change Mode to Incremental, and redeploy and restart the application. Striim will begin writing new data to the target from the point where initial load completed.