Skip to main content

Using Private Service Connect with Google Cloud adapters

Google's Private Service Connect allows private services to be securely accessed from Virtual Private Cloud (VPC) networks without exposing the services to the public internet (for more information, see Virtual Private Cloud > Documentation > Guides > Private Service Connect). You can use Private Service Connect to access managed services across VPCs or to access Google APIs and services.

Connecting to services securely using Private Service Connect

In a cloud-based infrastructure, services often communicate with each other over a public internet connection, making them vulnerable to various security threats. For example BigQuery Writer currently uses a publicly available API bigquery.googleapis.com to communicate and ingest data into BigQuery service.In the same way storage.googleapis.com and spanner.googleapis.com are used by GCS Writer and Spanner Writer respectively.

These API calls are routed through the internet before reaching the actual BigQuery instance or GCS bucket in your VPC.

Private Service Connect provides a secure way to connect services privately over the GCP network.

PSC_new_diagram_1.png
PSC_new_diagram_2.png

Sample application

The following sample application configure a Private Service Connect endpoint for a BigQuery Writer target:

CREATE APPLICATION OracleToBQ RECOVERY 10 SECOND INTERVAL;

CREATE OR REPLACE SOURCE oracle_source_CDC Using OracleReader(
 Username:’*****’,
 Password:’*****’,
 ConnectionURL:'jdbc:oracle:thin:@//localhost:1521/xe',
 OnlineCatalog:true,
 FetchSize:'1',
 Tables: 'HR.EMPLOYEE'
) Output To sourcestream1;


CREATE OR REPLACE TARGET bq_target USING BigQueryWriter  ( 
  projectId: 'striimdev'
  ,ServiceAccountKey: '/path/to/serviceaccountkey.json'
  ,StandardSQL: 'true'
  ,Mode: 'MERGE'
  ,optimizedMerge: 'true'
  ,PrivateServiceConnectEndpoint: 'striimdevpsc'
  ,BatchPolicy: 'eventCount:1000'
  ,Tables: 'HR.EMPLOYEE, HR.EMPLOYEE KeyColumns(RONUM)'
)   
INPUT FROM sourcestream1;

Usage notes

Note the following requirements and limitations for Private Service Connect support:

  • The Private Service Connect endpoint details you provide to the adapter must already exist. The adapter will not create the endpoint.

  • The Private Service Connect endpoint you provide to the adapter must be reachable or routable from the network where the Striim application is running. If the provided Private Service Connect endpoint becomes not reachable, the adapter will halt.

  • The BigQuery Storage Write API is currently not supported with Private Service Connect endpoints.