GCP Lakehouse Writer

GCP Lakehouse Writer supports both REST and custom catalog modes.

REST catalog mode is recommended for new deployments because it:

Custom catalog mode remains supported and can be used when:

After you create Iceberg tables with one catalog endpoint type, continue using the same endpoint type for those tables. Switching catalog modes for existing tables is not supported and can lead to metadata inconsistencies.

When REST catalog mode is used, IcebergTablesLocation must use a bucket-root path such as gs://my-bucket or gs://my-bucket/. Do not use a subdirectory path such as gs://my-bucket/warehouse.

Custom catalog mode supports subdirectory paths.

Credential vending is available only in REST catalog mode. In this model, GCP Lakehouse Runtime Catalog returns short-lived, down-scoped storage credentials for Cloud Storage access. When credential vending is enabled, Managed Apache Spark's default service account does not need direct GCS bucket permissions for table data access; access is controlled through BigLake IAM roles.

The Striim CredentialVending setting must match the catalog configuration in Google Cloud (see GCP Lakehouse Runtime Catalog Connection Profile:

When credential vending is enabled, the GCP Lakehouse Runtime Catalog service account still needs storage permissions to the staging bucket for accessing Striim’s batch metadata. The GCP Managed Apache Spark runtime service account still needs the required BigLake permissions to access the catalog and obtain vended credentials. The runtime service account still needs storage permissions for the external staging bucket.

GCP Lakehouse Writer can create Iceberg namespaces and target tables during initial load when the source sends schema creation events.

For relational database initial loads, enable schema creation on the source. For Database Reader, set Create Schema to true. Auto schema creation can be performed from RDBMS sources such as Oracle, SQL Server, MySQL, PostgreSQL, and other JDBC-compatible databases supported by Database Reader.

During the snapshot load, the source sends DDLs for namespace creation followed by table creation. GCP Lakehouse Writer executes those DDLs one after another on the compute engine. After the DDLs complete, snapshot data is written to the target tables.

For source-to-Iceberg Spark data type mappings, see Supported data types in the GCP Lakehouse Writer programmer's reference.

The BigQuery query format depends on the catalog endpoint type.

SELECT * FROM dataset_name.table_name;

SELECT * FROM <project name>.<lakehouse catalog name>.<namespace>.<table name>;

SELECT * FROM my-gcp-project.my-biglake-catalog.sales_db.orders;

UploadPolicy controls when GCP Lakehouse Writer uploads and processes a batch. A batch is uploaded when either the eventcount threshold or the interval threshold is reached.

Upload policy affects performance. A low eventcount creates many batches, which multiplies batch overhead. A very high eventcount creates larger batch files and may delay upload and execution. Use a higher eventcount when there is sufficient source traffic and Spark cluster memory.

GCP Lakehouse Writer supports selected DDL operations received from source adapters.

GCP Lakehouse Writer executes snapshot and CDC batches as Managed Service for Apache Spark jobs. Each batch is processed as a Spark application and split into tasks across available executors. Performance depends on worker nodes, memory, batch size, and Spark configuration.

GCP Lakehouse Writer requires a GCS external staging location. The external stage temporarily stores batch data, metadata, and execution artifacts while Spark jobs are running. The staging location must be reachable from the GCP Managed Apache Spark cluster and should be separate from the Iceberg warehouse location used for target table data.

Set the staging folder with ExternalStagingLocation. You may provide an ExternalStageConnectionProfileName for the external stage. If the data lake and external stage use the same project and authentication information, you may reuse the same GCS Connection Profile.