Skip to main content

MongoDB initial setup

MongoDBReader reads data from the Replica Set Oplog, so to use it you must be running a replica set (see Deploy a Replica Set or Convert a Standalone to a Replica Set, or your MongoDB cloud provider's documentation) on each shard to be read. For more information, see Replication and Replica Set Data Synchronization.

For all versions of MongoDB, the user specified in MongoDB Reader's Username property must have read access to the config database (see MongoDB Manual > Config Database

For MongoDB Atlas, for both InitialLoad and Incremental mode, the user specified in MongoDB Reader's Username property must have the readAnyDatabase role (see Built-in Roles).

For other versions of MongoDB, In InitialLoad mode, the user specified in MongoDB Reader's Username property must have read access to all databases containing the specified collections..

In Incremental mode, for MongoDB 4.2 and later, the user specified in MongoDB Reader's Username property must have changeStream and find privileges on all the collections of the cluster. You may want to create a role with these two privileges (see MongoDB Manual > User-Defined Roles).

In Incremental mode, for MongoDB 4.1 and earlier, the user specified in MongoDB Reader's Username property must have read access to the local database and the oplog.rs collection. The oplog is a capped collection, which means that the oldest data is automatically removed to keep it within the specified size. To support recovery, the oplog must be large enough to retain all data that may need to be recovered. See Oplog Size and Change Oplog Size for more information.

To support recovery (see Recovering applications), for all versions of MongoDB, the replica set's oplog must be large enough to retain all the events generated while Striim is offline.

The following discussions of networking and security apply to both initial load and continuous replication.

Networking setup

The following applies to both initial load and continuous replication.

You need to establish proper network connectivity between your Striim environment and MongoDB. This involves configuring network access, firewall rules, and connection parameters to ensure reliable communication.

Ensure that the Striim server can connect to MongoDB. You need to configure security groups to allow access from your Striim instance.

Also consider network latency and bandwidth requirements, especially for high-volume CDC scenarios. For optimal performance, minimize the network latency between Striim and MongoDB.

For secure connections, you can configure SSL/TLS encryption between Striim and MongoDB. This requires setting up SSL certificates and configuring both MongoDB and Striim to use encrypted connections.

Security

Security configuration for MongoDB integration involves multiple layers, including authentication, authorization, network security, and data protection measures.

You must implement proper authentication mechanisms between Striim and MongoDB. This includes creating dedicated database users with minimal required privileges following the principle of least privilege. You should avoid using administrative accounts and instead create specific users for Striim operations with only the necessary permissions for the tables and operations required.

You should implement access control at multiple levels, including database-level permissions, schema-level access controls, and table-level privileges. You should regularly review and audit the permissions granted to Striim users and implement proper password policies and rotation procedures for service accounts.

Using SSL or Kerberos or X.509 authentication with MongoDB

If you have an on-premise MongoDB deployment with your own certificate authority, set the Security Config properties as discussed below.

To set up SSL in MongoDB, see MongoDB Manual > Configure mongod and mongos for TLS/SSL.

To set up Kerberos in MongoDB, see MongoDB Manual > Kerberos Authentication.

To set up X.509 in MongoDB, see MongoDB Manual > Use x.509 Certificates to Authenticate Clients.

To secure your authentication parameters, store the entire Security Config string in a vault (see use Using vaults). For example, assuming your Kerberos realm is MYREALM.COM, its Key Distribution Center (KDC) is kerberos.realm.com, the path to the SSL trust store is /cacerts, the path to the SSL keystore file is /client.pkcs12, and the password for both stores is MyPassword, the Striim console commands to store the Security Config string with the key SSLKerberos in a vault named MongoDBVault would be:

CREATE VAULT MongoDBVault;
WRITE INTO MongoDBVault (
  vaultKey: "SSLKerberos",
  vaultValue: "RealmName:MYREALM.COM;
    KDC:kerberos.myrealm.com;
    KeyStore:/keystore.pkcs12;
    TrustStore:/cacerts;
    trustStorePassword:MyPassword;
    KeyStorePassword:MyPassword"
);

Enter READ ALL FROM MongoDBVault; to verify the contents.

In TQL or the Flow Designer, you would then specify the Security Config as [[MongoDBVault.SSLKerberos]].

The following are examples for each authentication option.

Kerberos authentication

Without SSL:

CREATE VAULT MongoDBVault;
WRITE INTO MongoDBVault (
  vaultKey: "Kerberos",
  vaultValue : "RealmName:MYREALM.COM;
  KDC:kdc.myrealm.com"
);

With SSL:

CREATE VAULT MongoDBVault;
WRITE INTO MongoDBVault (
  vaultKey: "KerberosSSL",
  vaultValue : "RealmName:MYREALM.COM;
  KDC:kdc.myrealm.com;
  KeyStore:UploadedFiles/keystore.ks;
  KeyStorePassword:MyPassword;
  TrustStore:Platform/UploadedFiles/truststore.ks;
  TrustStorePassword:MyPassword"
);

If required, also specify:

  • JAAS_CONF: the path to and name of the uploaded JAAS configuration file, for example, UploadedFiles/jaas.conf

  • JAAS_ENTRY: the name of an entry in jaas.conf, for example, striim_user

  • KeyStoreType:PKCS12: specify this if the KeyStore is PKCS12 rather than JKS

  • any of the elements listed below for SSL

SSL

If you are using MongoDB on premise with SSL, use Security Config to configure key store and trust stores.

CREATE VAULT MongoDBVault;
WRITE INTO MongoDBVault (
  vaultKey: "SSL",
  vaultValue : "KeyStore:UploadedFiles/keystore.ks;
  TrustStore:UploadedFiles/truststore.ks;
  TrustStorePassword:MyPassword;
  KeyStorePassword:MyPassword"
);

If required, also specify:

  • SecureSocketProtocol: specify if a specific protocol is required

  • TrustStoreType:PKCS12: specify this if the TrustStore is PKCS12 rather than JKS

X.509 authentication
CREATE VAULT MongoDBVault;
WRITE INTO MongoDBVault (
  vaultKey: "X509",
  vaultValue : "KeyStore:UploadedFiles/keystore.ks;
  KeyStorePassword:MyPassword"
);

If required, also specify:

  • KeyStoreType:PKCS12: specify this if the KeyStore is PKCS12 rather than JKS

Connecting to MongoDB Atlas clusters using private endpoints or network peering

Striim provides support for connecting to MongoDB Atlas clusters using private endpoints or network peering.

  • Private endpoints: MongoDB Atlas supports private endpoints on dedicated clusters. For example, you may configure a private endpoint connection to a Mongo cluster in Atlas from Striim Platform installed in a Google Cloud Platform VM using a private aware endpoint.

  • Network peering: MongoDB Atlas supports network peering connections. Network peering establishes a private connection between your Atlas VPC and your cloud provider's VPC. The connection isolates traffic from public networks for added security.

There are no Striim-specific configuration steps required; the configuration steps you need to perform are in MongoDB Atlas.

See the following MongoDB Atlas doc topics: