GCP Lakehouse Writer initial setup

Choose the catalog endpoint type before creating tables. Use REST catalog mode for new deployments when its prerequisites are met. Use custom catalog mode for existing custom-catalog deployments or when you need standard BigQuery dataset.table query naming.

Do not switch catalog endpoint types for an existing target table.

Create or identify the GCS bucket that stores target Iceberg table data and metadata. Configure this location in GCP Lakehouse Writer as IcebergTablesLocation. See Configure Google Cloud Storage (GCS) buckets for Iceberg Writer.

For REST catalog mode, use the bucket root, for example gs://my-bucket or gs://my-bucket/. Do not use a subdirectory path.

Create a GCS folder for the external stage. The external stage should be reachable from the GCP Managed Apache Spark cluster and should be different from the Iceberg warehouse location.

If the staging bucket and data lake bucket are different, grant the required storage permissions on both buckets.

Enable the Lakehouse API in the Google Cloud project. As of July 2026 the API is still named BigLake API in Google Cloud.

Create a GCP Lakehouse Runtime Catalog resource that points to the Cloud Storage bucket. A single-bucket catalog supports one catalog per bucket, and the catalog name typically matches the bucket name. For example, for gs://my-bucket, the catalog name is usually my-bucket.

To create the catalog in Google Cloud Console:

A single-bucket catalog ties the catalog to a single bucket and typically ties the catalog name to the bucket name. Consider that constraint when planning the deployment.

Create or identify a GCP Managed Apache Spark cluster that uses a Spark 3.5.1-compatible image. Use image version 2.2-debian12; that image shorthand resolves to 2.2.42-debian12.

When creating the cluster:

The initialization script is bundled in the Striim Utilities zip file. In Striim, click Download Utilities under the About Striim icon, unzip the file, and locate setupIcebergDataprocV1.sh in Striim/tools/bin.

Upload setupIcebergDataprocV1.sh to a GCS location that the GCP Managed Apache Spark cluster can access. Use that gs:// URI as the initialization action when creating the cluster.

You can reuse an existing GCP Managed Apache Spark cluster if it has capacity for Iceberg Spark jobs and is not configured with Internal IP only. Check this in Google Cloud under Managed Service for Apache Spark cluster (Dataproc) > Configuration.

If the required Iceberg libraries are not already installed, connect to the master node with SSH and install the libraries as a super user:

sudo su
wget -P /usr/lib/striim/iceberg/ "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-runtime-3.5_2.12/1.10.0/iceberg-spark-runtime-3.5_2.12-1.10.0.jar";
wget -P /usr/lib/striim/iceberg/ "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-spark-extensions-3.5_2.12/1.10.0/iceberg-spark-extensions-3.5_2.12-1.10.0.jar";
wget -P /usr/lib/striim/iceberg/ "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp/1.10.0/iceberg-gcp-1.10.0.jar";
wget -P /usr/lib/striim/iceberg/ "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-gcp-bundle/1.10.0/iceberg-gcp-bundle-1.10.0.jar";
wget -P /usr/lib/striim/iceberg/ "https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-bigquery/1.10.0/iceberg-bigquery-1.10.0.jar";

Verify that the files are present:

cd /usr/lib/striim/iceberg/
ls

Expected files:

iceberg-gcp-1.10.0.jar
iceberg-gcp-bundle-1.10.0.jar 
iceberg-spark-extensions-3.5_2.12-1.10.0.jar 
iceberg-spark-runtime-3.5_2.12-1.10.0.jar
iceberg-bigquery-1.10.0.jar

GCP Managed Apache Spark images do not include the required Iceberg runtime libraries by default. Install the required libraries on all cluster nodes by using the Striim-provided setupIcebergDataprocV1.sh initialization script or by installing the jars manually on an existing cluster.

GCP Lakehouse Writer ships with Java client libraries used for GCS and Managed Service for Apache Spark connectivity.

Create or reuse the Google Cloud service accounts required by your deployment and obtain the JSON key files needed for the Connection Profiles. The required roles depend on catalog endpoint type and credential vending configuration.

When using CustomIcebergCatalogForBigQuery, the GCP Managed Apache Spark runtime service account requires:

When using IcebergRESTCatalog without credential vending, the GCP Managed Apache Spark runtime service account requires:

Write roles include the required read permissions, so separate read-only roles are not required for this write workload.

When using IcebergRESTCatalog with credential vending:

With credential vending enabled, the Spark runtime service account does not require direct GCS bucket permissions for table data. The GCP Lakehouse Runtime Catalog service account still requires roles/storage.objectUser on the bucket.

Regardless of catalog mode, the GCP Managed Apache Spark runtime service account requires:

If the staging area and data lake use the same GCS bucket, storage permissions can be combined. If they use different buckets, grant storage permissions on both buckets.

The user who creates the prerequisites may also need Google Cloud administrative roles, such as roles/dataproc.admin to create the cluster and roles/storage.admin to set up staging buckets.

Use the GCPManagedApacheSpark endpoint, shown as Google Cloud Managed Apache Spark in the UI. This Connection Profile configures connectivity to the GCP Managed Apache Spark cluster for job submission and monitoring.

The service account used in this Connection Profile must have the required permissions, including roles/dataproc.worker or equivalent permissions to submit and monitor jobs.

Use the GCS endpoint for both the data lake and the external staging area. You can reuse the same GCS Connection Profile for both when they share the same authentication and project configuration.

The service account used in the GCS Connection Profile must have the required storage permissions, including roles/storage.objectUser for read/write access to the data lake and staging buckets.

Use the GCPLakehouseRuntimeCatalog endpoint, shown as Google Cloud Lakehouse Runtime Catalog in the UI.

Use additional configuration on the GCP Lakehouse Runtime Catalog Connection Profile when you need to set catalog configuration that cannot be set with the standard profile properties.

If an additional configuration property maps to an existing Connection Profile property, the additional configuration value overrides the Connection Profile value.

Use additional configuration on the GCP Managed Apache Spark Connection Profile to provide Spark configuration properties. These settings can override the default Spark configuration on the cluster and can improve performance for specific workloads.

Common Spark performance properties include: