BigQuery Writer
BigQuery is a cloud-based data warehousing and analytics platform developed by Google. It allows users to store, analyze, and query large datasets in a fast and scalable way using SQL-like queries.
You can use Striim’s BigQuery Writer to write data from transactional databases such as Oracle and SQL Server, applications such as Salesforce and ServiceNow, NoSQL databases such as Cosmos DB and MongoDB, object stores such as Amazon S3 and Google Cloud Storage, and other supported sources into BigQuery. BigQuery Writer operates with low latency to support real-time analytics and data processing, business intelligence, machine learning, and other tasks.
BigQuery Writer summary
Supported sources | BigQuery Writer can write data from all sources supported by Striim. |
Authentication | BigQuery Writer authenticates its connection to BigQuery using a Google service account key. The adapter uses OAuth2 for authorization and TLS 1.2 to encrypt the connection. |
Supported writing methods | BigQuery Writer supports three writing methods, each of which uses a different Google API:
|
Supported write modes | BigQuery Writer supports two write modes:
|
Additional writing features |
|
Resilience and recovery |
|
Performance | Supports parallel threads (see Creating multiple writer instances) to increase throughput to the target. |
Programmability |
|
Metrics and auditing | Key metrics are available through Striim's monitoring features (see Monitoring Guide). |
Java client version | BigQuery Writer uses Google Cloud BigQuery Client for Java (google-cloud-bigquery) version 2.3.3. |
Notes on BigQuery terminology
Some of BigQuery's terms have different meanings than they do in the context of popular SQL databases.
Project: contains one or more datasets, similar to the way an Oracle CDB contains one or more databases.
Dataset: contains one or more tables, similar to an Oracle, PostgreSQL, or SQL Server schema.
Schema: defines column names and data types for a table, similar to a CREATE TABLE DDL statement in SQL.
Table: equivalent to a table in SQL.