BigQuery Writer runtime considerations

If you configure BigQuery Writer to exceed Google's quotas and limits, your application may halt. You can minimize the chances of this happening by configuring BigQuery Writer to use the Storage Write API. Quotas and limits are subject to change by Google without notice; numbers in this document are up to date as of July 2023. For the latest information, see BigQuery > Documentation > Reference > Quotas and limits.

When using the Storage Write API, if a single row contains approximately 5MB of data or more, the BigQuery Writer application may terminate. This is due to Storage Write API's AppendRows request size limit of 10MB (metadata may increase the request size to be twice that of the original data).

Arrays cannot be used as KeyColumns.

Due to limitations in Google BigQuery, when BigQuery Writer's input stream is the output of a Database Reader, Database Reader's RestartBehaviourOnILInterruption property is set to truncateTargetTableData or replaceTargetTableData, and BigQuery Writer's Streaming Upload property is True, initial load recovery may terminate or halt with an "entity not found" exception or "Could not perform a streaming insert" error. In this case, restarting the application after five to 30 minutes will typically resolve the problem.