Using AWS PrivateLink
AWS PrivateLink is a robust and scalable solution designed to facilitate private connections between your VPC and various services, making them appear as though they reside within your VPC (for more information, see Amazon VPC > What is AWS Private Link?). Utilizing VPC endpoints, you can seamlessly integrate resources within your VPC with services that are compatible with AWS PrivateLink.
The following diagram illustrates the architecture of AWS PrivateLink. Service consumers establish interface VPC endpoints to interface with services presented by service providers. These providers create an endpoint service to make certain solutions—like EC2 and other non-fully managed AWS services—available within a specific region. Integral to this process is the service provider's designation of a load balancer during endpoint service setup. This load balancer captures and routes requests from the service consumers to the appropriate service.
Preparing to use AWS PrivateLink
Striim supports the use of AWS PrivateLink with the following sources and targets:
Amazon Kinesis
Amazon Managed Streaming for Apache Kafka (MSK)
Amazon Relational Database Service (RDS)
Amazon S3
Databricks
MongoDB Atlas
Snowflake
Endpoint service
Note
To configure an endpoint for Snowflake, contact Striim support.
You can perform the following prerequisites to use AWS PrivateLink:
Creating a VPC and an endpoint service
These sample steps create a VPC and service endpoint. Your environment and network requirements may differ.
Open the Amazon VPC console at https://console.aws.amazon.com/vpc/.
Create a VPC. See Amazon VPC > Create a VPC.
Create a route table and attach it to the VPC. See Amazon VPC > Configure route tables.
Set the route table as the main route table.
Create two subnets in different availability zones. See Amazon VPC > Create a subnet.
Create an internet gateway and attach it to the VPC. See Amazon VPC > Internet gateway.
Launch an EC2 instance. Create a key pair if required. See Amazon EC2 > Launch your instance.
Choose the correct VPC, assign a public IP address and choose the default security group.
Add the VPC > Security Group > Inbound Rule to accept SSH connections from your IP address.
Create an EC2 Target Group for your VPC. Choose the IP address type and TCP protocol.
Create an internal network load balancer. See Elastic Load Balancing > Create a network load balancer.
Create a service endpoint. See Amazon VPC > Create an endpoint service.
You can use the endpoint service name to create a VPC endpoint.
Configuring an endpoint for AWS-managed services
No database instance is running in your VPC yet. You can create and assign it in the previously created target group.
For more information about configuration properties specific to your database, see:
Using AWS PrivateLink with your Striim application
To use AWS PrivateLink with your Striim application, you should publish your service and provide the service name URI while creating the endpoint service. The Service Name will look similar to: com.amazonaws.vpce.us-east-1.vpce-svc-0f61034fede44196e
. The published service that you create should be in the same AWS Region as the service.
Note
You cannot create more than one endpoint using a single service name.
You are responsible for configuring the approval method (automatic or manual) for the service accessed though AWS PrivateLink. For manual approval, you should approve access when the PrivateLink connection is created. You provide the Endpoint DNS to access the published service when configuring your pipelines in the Striim application.
You can manage access to the connection by setting the proper permissions to the endpoint.
Once a private link is created for a service, it will be retained when the service is stopped. It will be deleted when the service is deleted. In case of disaster recovery, the PrivateLink may be disconnected because we cannot recreate the Private Service name, because it has a unique ID for the VPC that is being created for the Service Name.
Configuring an Amazon Kinesis provider
To create AWS PrivateLink for Amazon Kinesis, create a VPC endpoint with the following properties:
Service Category: AWS services.
Services: the interface for a given region.
VPC: the destination VPC.
Note
Make sure the Enable DNS Hostname option is enabled in your VPC.
Configuring an Amazon Managed Streaming for Apache Kafka provider
There are two possible topologies when using Private Link with MSK:
Using a different PrivateLink for each MSK broker
Using a single PrivateLink for all the brokers
For more discussion of these topologies, see AWS Big Data Blog > Secure connectivity patterns to access MSK.
To create AWS PrivateLink for Amazon MSK, create a VPC endpoint with the following properties:
Service Category: Other endpoint services.
Service Setting: provide the endpoint service name found in VPC > Endpoint Services > Service Name.
VPC: the destination VPC.
Note
Make sure the Enable DNS Hostname option is enabled in your VPC.
Routing users to your application
Amazon Route 53 is a a reliable and cost-effective way to route end users to Internet applications.
Configure the hosted zone in Amazon Route 53:
Create a Route 53 hosted zone for
kafka.<region>.amazonaws.com
if it does not exist.Create a Route 53 record which maps the VPC endpoint URL to the broker URL name.
Configuring an Amazon Relational Database Service provider
To create AWS PrivateLink for Amazon RDS, create a VPC endpoint with the following properties:
Service Category: Other endpoint services.
Service Setting: provide the endpoint service name found in VPC > Endpoint Services > Service Name.
VPC: the destination VPC.
Note
Make sure the Enable DNS Hostname option is enabled in your VPC.
Configuring an Amazon S3 provider
To create AWS PrivateLink for Amazon S3, create a VPC endpoint with the following properties:
Service Category: AWS services.
Service Setting: interface for a given region or global.
VPC: the destination VPC.
Note
Make sure the Enable DNS Hostname option is enabled in your VPC.
Configuring a MongoDB Atlas provider
To create AWS PrivateLink for MongoDB Atlas, create a VPC endpoint with the following properties:
Service Category: Other endpoint services.
Service Setting: provide the endpoint service name found in VPC > Endpoint Services > Service Name.
VPC: the destination VPC.
Note
Make sure the Enable DNS Hostname option is enabled in your VPC.