Connecting Striim Cloud on Google Cloud Platform to managed data services
This topic explains how to enable private connectivity from Striim Cloud on Google Cloud Platform to managed Google Cloud services. Private Service Connect allows Striim Cloud to access these services without exposing traffic to the public internet.
There are two main approaches for connecting to managed Google Cloud services:
Google APIs via the
allgoogleapis
endpoint - For services like BigQuery, Cloud Spanner, and Cloud Storage that are accessed through Google APIs.Customer-managed databases via your infrastructure - For services like Cloud SQL that require routing through your own GCP project infrastructure.
Connecting to Google APIs
Use this approach for Google Cloud services that are accessed through standard Google APIs, including BigQuery, Cloud Spanner, and Cloud Storage (GCS). This method uses Google Cloud's built-in allgoogleapis Private Service Connect endpoint.
Overview
Google Cloud's Private Service Connect routes traffic to Google APIs privately over Google's network. This keeps data off the public internet while preserving full functionality for supported services such as BigQuery, Spanner, and GCS. The same flow applies across these services.
Supported services
BigQuery
Cloud Spanner
Cloud Storage (GCS)
Prerequisites
Access to the Striim Cloud console with Admin permissions to create endpoints and apps.
The Striim service must be in the Running state.
Implementation steps
This process creates a PSC endpoint that connects to Google's allgoogleapis service, providing private access to supported Google Cloud services.
Login to the Striim Cloud portal.
Select the Service that you want to create Private service connect from.
Under the Secure connection tab, choose Create Private Service Connect.
Use a unique name for this connection.
Leave the Service attachment field empty and Create.
An endpoint named for allgoogleapis will be created. Wait until its status is Running, then note the endpoint name to use in the Application connection configuration.
Connecting to customer-managed databases
Use this approach for managed Google Cloud databases like Cloud SQL that require routing through your own GCP project infrastructure. This method requires setting up Private Service Connect infrastructure in your GCP project.
Overview
This pattern routes traffic from Striim Cloud through a Private Service Connect attachment in your GCP project to reach managed databases like Cloud SQL. The connection uses your project's network infrastructure to provide private connectivity.
Supported services
Cloud SQL
Cloud PostgreSQL
Cloud MySQL
Prerequisites
A VPC network with one additional subnet for PSC.
IAM rule for the user with above services creation permission.
A CloudSQL database with a private IP.
Step-by-step process to setup the network pipeline in customer project
This process establishes the necessary infrastructure in your GCP project to route traffic privately from Striim Cloud to your managed databases.
Create a Ubuntu VM instance with followings:
A e2-medium should be sufficient for this solution.
Launch the instance in the same VPC network as database.
SSH to the instance and set up an IP forwarding rule to forward all incoming traffic to the database by using below command: (change the IP and Port accordingly for you target database).
Determine if forwarding is currently enabled or disabled. The command below outputs the value of the given parameter. A value of 1 indicates that the setting is enabled, while 0 indicates it is disabled. If you intend to configure forwarding, check that kernel parameter as well.
sudo sysctl net.ipv4.ip_forward
Open the file /etc/sysctl.conf using your preferred command-line editor, such as vi.
sudo vi /etc/sysctl.conf
Find the line corresponding with the type of forwarding you wish to enable, uncomment it, and set the value to 1. Alternatively, you can add the lines anywhere in the file.
After the changes have been saved, apply the changes by running the following command or by rebooting the machine.
sudo sysctl -p
Review the existing iptables rules. If you are on a fresh installation of Linux and do not have any preconfigured rules, the output of the below command should by empty.
sudo iptables-save
Configure iptables to allow port forwarding. This is the default setting for many systems.
sudo iptables -A FORWARD -j ACCEPT
Run following commands to setup the forwarding rule in the ip-table:
sudo iptables -t nat -A PREROUTING -p tcp --dport 5432 -j DNAT --to-destination sudo iptables -t nat -A POSTROUTING -j MASQUERADE sudo cat /proc/sys/net/ipv4/ip_forward sudo sh -c "echo 1 > /proc/sys/net/ipv4/ip_forward" sudo iptables-save
To make this rule persistence during reboot download iptables-persistent package: (press yes in the popup window to acknowledge).
sudo apt install iptables-persistent
To review iptables rule:
iptables -t nat -nvL
Go to the Compute Engine portal and choose Instance groups on the left panel.
Select Create instance group.
Choose New unmanaged instance group.
Enter an instance group name, location, zone and other information:
For location and zone, choose the same location and zone as the proxy instance that you created.
For network interface choose the same VPC, subnet and Instance VM that you created/used earlier for the proxy instance.
Choose Create.
Create a load balancer:
Choose TCP load balancer.
Select Only between my VMs, Single region only, Pass-through then choose continue.
Give it a name and choose the same region and network as the proxy VM instance.
In the backend configuration, choose the instance group you have created.
Create a health check probe for the port you assign for your database.
Select Frontend configuration.
Enter a name for the configuration.
Select the same network you have been using.
Choose an internal IP purpose, shared/non-shared.
For ports choose all or any port/s.
Disable global access.
Choose Create.
Go to the Load balancer portal.
Click on Private service connect on the left panel.
Select on Published service, then choose Create new.
In the config, choose the load balancer type that you created before and select the load balancer from the dropdown.
Give it a name.
In the subnet section, choose Reserve a subnet for PSC.
Enter a subnet name and a CIDR block with /24.
Choose connection preference Accept connections for selected projects.
Click on Add service to create this service.
After creating the service, select the service and copy the Service attachment. You will need to use this attachment to create PSC connection in the Striim Cloud console.
Create a Private service connection in the Striim Cloud project
Once the customer infrastructure is ready, create the corresponding PSC endpoint in Striim Cloud to establish the connection.
Login to the Striim Cloud console.
Select the service that you want to create a private service connect from.
Under the Secure connection tab, choose Create Private Service Connect.
Use a unique name for this connection.
Paste the private service attachment URL that you copied previously and click Create. The attachment looks like:
projects/striim-tools/regions/us-central1/serviceAttachments/psc-demo-service
You have now created a PSC connection endpoint. It is in a pending state until the request gets accepted.
Once the request gets approved from PSC published service portal, endpoint status will change to "Running".
Accept PLS connection request in customer project
The final step is to approve the connection request from your GCP project.
Go to the Private Service Connect portal, then the published service.
Select the published service.
You will see a request in the pending state.
You have a choice to Accept or Reject the request.
Reject will refuse the new connection request or disconnect the existing connection.