Skip to main content

BigQuery initial setup

In BigQuery, create a service account key. The account must have the following roles or permissions:

  • At the project level:

    • bigquery.jobUser role or bigquery.jobs.create permission, required to submit query jobs

    • bigquery.readSessionUser role or bigquery.readsessions.create and bigquery.readsessions.getData permissions, required for BigQuery Storage API access

  • At the dataset or project level:

    • bigquery.dataViewer role or bigquery.tables.get, bigquery.tables.getData, and bigquery.tables.list permissions, required to read table metadata and data

The following discussions of networking and security apply to both initial load and continuous replication.

Networking setup

The following applies to both initial load and continuous replication.

You need to establish proper network connectivity between your Striim environment and BigQuery. This involves configuring network access, firewall rules, and connection parameters to ensure reliable communication.

Ensure that the Striim server can connect to BigQuery. You need to configure security groups to allow access from your Striim instance.

Also consider network latency and bandwidth requirements, especially for high-volume CDC scenarios. For optimal performance, minimize the network latency between Striim and BigQuery.

Connections between BigQuery and Striim use TLS 1.2 or 1.3. No extra setup is required.

Security

Security configuration for BigQuery integration involves multiple layers, including authentication, authorization, network security, and data protection measures.

You must implement proper authentication mechanisms between Striim and BigQuery. This includes creating dedicated database users with minimal required privileges following the principle of least privilege. You should avoid using administrative accounts and instead create specific users for Striim operations with only the necessary permissions for the tables and operations required.

Connections between BigQuery and Striim use TLS 1.2 or 1.3. No extra setup is required.

You should implement access control at multiple levels, including database-level permissions, schema-level access controls, and table-level privileges. You should regularly review and audit the permissions granted to Striim users and implement proper password policies and rotation procedures for service accounts.