S3 Tables Writer
S3 Tables Writer is a Striim target adapter that writes change data and snapshot data from Striim sources into Apache Iceberg tables stored in Amazon S3 Tables. S3 Tables is an AWS-managed bucket type and storage service, purpose-built for Apache Iceberg, that provides an embedded catalog, higher transaction throughput than general-purpose S3 buckets, and automatic table maintenance such as metadata compaction and unreferenced-file removal.
S3 Tables Writer relies on two components outside of Striim: a compute engine that executes the underlying write operations, and a catalog that tracks namespaces and tables. Striim uses Amazon EMR as the compute engine and supports either of two catalogs: the catalog embedded in S3 Tables (the default) or AWS Glue Catalog.
Where S3 Tables Writer fits
S3 Tables Writer is a target-side component in a Striim pipeline. A source adapter (for example, Database Reader for Oracle, SQL Server, MySQL, or PostgreSQL; Salesforce Reader; ServiceNow Reader; MongoDB Reader; or Cosmos DB Reader) produces an output stream of insert, update, and delete operations. S3 Tables Writer consumes that stream, batches the events according to its upload policy, and writes them to Iceberg tables through an Amazon EMR Spark job, using a general-purpose S3 bucket as a staging area for batch data before it is committed to the target.
Feature summary
Capability | Details |
|---|---|
Supported sources | RDBMS sources such as Oracle, SQL Server, MySQL, and PostgreSQL; application adapters such as Salesforce and ServiceNow; and NoSQL sources such as MongoDB and Cosmos DB, for both snapshot and CDC loads. |
Authentication | Two distinct models depending on connection profile. AWS EMR and S3 (external staging) connection profiles accept an explicit Access Key / Secret Key pair, or fall back to the AWS default credentials provider chain. S3Tables and AWS Glue Catalog connection profiles authenticate only through the IAM role attached to the EMR cluster's instance profile. See Configure Striim to work with S3 Tables. |
Write modes | APPENDONLY (insert-only, snapshot loads) and MERGE (insert/update/delete, CDC loads), with an Optimized Merge option for partial-image CDC. |
Additional writing features | Automatic namespace and table creation during initial load (Initial Schema Creation); DDL propagation for supported source DDL operations (schema evolution); parallel-threaded snapshot writes; column mapping and key-column support for tables without a usable primary key. |
Supported staging areas | A folder inside a general-purpose S3 bucket, configured through a dedicated S3 connection profile and referenced by the writer's ExternalStagingLocation property. |
Resilience and recovery | Configurable connection retry per connection profile; recovery on restart with at-least-once processing semantics; table-level initial-load recovery so a completed table is not reprocessed after a restart. |
Performance | User-configurable batching (UploadPolicy) based on event count and time interval; parallel threads for snapshot loads; Optimized Merge to reduce CDC data volume. |
Programmability | Configurable through the Flow Designer or TQL using the S3TablesIcebergWriter adapter class, plus four connection profile types (AWS EMR, S3Tables, AWS Glue Catalog, S3). |
Metrics and auditing | Table-level write metrics (batch counts, timings, DDL history) and adapter-level metrics (write timestamp, target freshness, discarded events, connection retry information, queued batch size) — see S3 Tables Writer monitoring metrics. |
Key limitations | S3 Tables requires lowercase table, namespace, and column names, which breaks wildcard mappings against uppercase-identifier sources; CREATE TABLE DDL against the AWS Glue catalog requires an AWS-supplied EMR patch; Access Key and Secret Key are not exposed as configurable properties on the S3Tables and AWS Glue Catalog connection profiles; AWS region-level API rate limits can throttle S3 Tables calls under heavy concurrent use. See S3 Tables Writer operational considerations for the full list. |
Drivers and other third-party libraries | S3 Tables Writer uses AWS SDK for S3 and AWS SDK for EMR (version 2.32.13), Apache Iceberg Core and Apache Iceberg AWS (version 1.10.0), and Apache Spark Core and Apache Spark SQL (version 3.5.1). See the appendix for the full dependency list. |
Next steps
Continue to Create an S3 Tables Writer application to provision the required AWS infrastructure and build your first pipeline.
See S3 Tables Writer initial setup for the full AWS-side and Striim-side setup procedures for both catalog paths.
See S3 Tables Writer programmer's reference for the complete property and connection profile reference.
See S3 Tables Writer operational considerations for recovery behavior, best practices, troubleshooting, and limitations.
Primary use cases
Snapshot loads (initial load) — moving an initial copy of source tables into S3 Tables. Use APPENDONLY mode, optionally with parallel threads, to load data as a batch of insert operations.
Continuous CDC replication — keeping S3 Tables in sync with ongoing inserts, updates, and deletes from a source. Use MERGE mode so that captured operations are applied against the target Iceberg tables as they arrive.
Combined initial load + CDC — a common pattern (illustrated in the Getting started tutorial in Create an S3 Tables Writer application) that runs an APPENDONLY initial load application followed by a MERGE CDC application against the same target tables.
Higher transaction throughput than general-purpose S3 — use S3 Tables Writer to take advantage of S3 Tables' higher transactions-per-second ceiling for write-heavy Iceberg workloads.
Reduced table maintenance overhead — S3 Tables automatically performs metadata compaction and unreferenced-file removal, reducing the manual maintenance typically required for Iceberg tables on general-purpose S3.
Write modes
Write mode | Recommended for | Behavior |
|---|---|---|
APPENDONLY | Snapshot (initial) loads | Striim writes every incoming event as an insert operation against the target Iceberg tables. Supports parallel threads for faster snapshot loads. |
MERGE | CDC loads | Striim applies captured inserts, updates, and deletes from the source against the target Iceberg tables. Supports Optimized Merge for partial-image updates and deletes. |
MERGE with Optimized Merge | CDC loads where the source sends partial-image (change-only) events | Striim applies captured inserts, updates, and deletes using only the key fields and changed values for update and delete operations, instead of requiring a full row image. Reduces the data volume processed for update-heavy and delete-heavy CDC workloads. Enable by setting OptimizedMerge to true; valid only when Mode is MERGE. |
Note: Configuration Validation rejects incompatible combinations at compile time: ParallelThreads is only valid in APPENDONLY mode, and OptimizedMerge is only valid in MERGE mode. See S3 Tables Writer programmer's reference for the exact compiler error text.