Skip to main content

YugabyteDB operational considerations

YugabyteDB limitations

  • Only Tables with primary key are supported in CDC for YugabyteDB.

  • PK Update in YugabyteDB is received as DELTE followed by INSERT. Mon metrics of Striim’s Yugabyte Reader would not increment PK Update metrics, rather the DELETE & INSERT Metrics would be incremented for every PK-UPDATE.

  • Replica Identity Constraints

    • Existing Tables: The REPLICA IDENTITY of tables already present when the replication slot is created is immutable. You cannot change their identity once the slot exists.

    • New Tables: For tables created after slot creation, you can define the REPLICA IDENTITY dynamically, provided specific T-Server and Master flags are configured beforehand.

    • Required Configuration Flags

      • To support the automatic capture of dynamically created tables, the following flags must be set on both the Master and all T-Servers before the new tables are created:

      Flag

      Recommended Value

      Description

      ysql_yb_default_replica_identity

      FULL

      Sets the default replica identity for all new tables (ensures "before" images are captured).

      cdcsdk_enable_dynamic_table_support

      true

      Enables the CDCSDK module to automatically detect and include newly created tables.

      cdcsdk_publication_list_refresh_interval_secs

      120 (example)

      Controls how often the module scans for new tables. Default is 900 (15 mins).

  • Truncate operation - database does not allow user to truncate the table if they are enabled for change data capture.

  • When replicating Large Objects (TEXT, IMAGE, BLOB/CLOB), the following limits apply:

    • LOB Column Limit: Individual columns containing LOB data are restricted to 2MB each.

    • Maximum Row Size: The total size of a single record across all columns must not exceed 32MB.

    • Total Capture Limit: The aggregate amount of LOB data captured in the stream is capped at 200MB.

  • YB CDC reader would HALT due to “streaming closed” errors if any tables are dropped in the database After the replication slot has been created.

  • WAL RETENTION: To prevent data loss during Striim YB Reader downtime, YugabyteDB’s default WAL purging must be modified. WAL retention is not guaranteed unless the following parameters are adjusted across all cluster nodes:

    • cdc_wal_retention_time_secs: Set to 86400 (24 hours) to ensure logs persist during short-term CDC outages.

    • cdc_intent_retention_ms: 86400000, Should be increased proportionally to retain transaction intents for longer durations.

    Failure to adjust these settings may result in log truncation, requiring a full re-synchronization of the pipelines.