Documentation

Connect to a PrivateLink Apache Kafka Cluster

Overview

This page describes the steps that need to be taken to connect clients in one VPC to a PrivateLink enabled Kafka cluster in another VPC. In short, customers need to create a security group, an endpoint with the endpoint service name of the created Kafka PrivateLink cluster, and a Route 53 record to the endpoint created. Kafka Add-Ons require additional Route53 records.

Please refer to Connect to a PrivateLink Kafka cluster with AWS CloudFormation to automatically create the required resources using AWS CloudFormation templates to connect to a PrivateLink Kafka Cluster.

Retrieve Required Cluster Connection Information

  1. Log into the console, click the created cluster and enter the Connection Info page.
  2. The Connection Info page contains the specific connection details for PrivateLink. The endpoint created in the next section uses the endpoint service name shown below.
  3. In the Connection Info page, identify the region of the created PrivateLink Kafka cluster. For instance, the page shows the region is US_EAST_1, hence access to the AWS VPC Endpoint console in the US_EAST_1 region: https://us-east-1.console.aws.amazon.com/vpc/home?region=us-east-1#Endpoints:
  4. If the PrivateLink Kafka cluster and client are in different AWS accounts, please ensure the AWS Principal of the client AWS account is added to allow cross-account access to the endpoint. For more info, please read Managing Principal ARNs of a PrivateLink Kafka Cluster. 

Create Endpoint Security Group

A security group is required to allow traffic from your VPC to the PrivateLink endpoint. Your Instaclustr provided Kafka cluster already has a security group applied to ensure the security of your cluster, but an Endpoint cannot be created without having an associated security group. Therefore the security group being created here can be as permissive or as strict as you require. These steps will outline the recommended security group configuration.

  1. In the AWS Console, select the appropriate region for your VPC and navigate to the Security Groups page.
  2. Click “Create security group”.
  3. Under the heading “Security group name”, enter a name for the security group. For example, “companyName-kafka-privatelink-group”. The name cannot be edited after creation, thus it is recommended to use a naming convention which allows easy identification of the PrivateLink endpoint and destination cluster it will be associated with.
  4. Under the heading “Description”, enter a short description of the security group. For example, “Allows access to Kafka cluster via PrivateLink”.
  5. Under the heading “VPC”, search for and select the VPC which will connect to the Kafka cluster via PrivateLink.
  6. Under the Inbound rules heading, click “Add rule”. The rules created here permit traffic outbound from your VPC, inbound to the Kafka cluster over PrivateLink.AWS Endpoint Security Group Configuration Screen
  7. Select Custom TCP in the “Type” field. Enter 9091 in the “Port range” field (n.b. Port 9091 is the Kafka Broker listener port. When connecting via PrivateLink, this port is load balanced between all broker nodes in your cluster. This port will not change, and can be verified on your Kafka cluster’s Connection Info page.) Select Custom in the “Source” field, then enter the CIDR of the subnet you will be connecting to the PrivateLink endpoint from (ie. the CIDR within which your EC2 Instance resides). Optionally, add a description in the Description field.
  8. Repeat step 7, using ports 6001-6000+n in the Port range field, where n is the number of nodes in your Kafka cluster. For example, if you are using a 5 node Kafka cluster, the port range should be 6001-6005. n.b. If the number of nodes in the Kafka cluster is increased in the future, this port range will need to be extended to match the new number of nodes in the cluster. The purpose of the ports in this range is to provide a direct connection to a specific broker within the Kafka cluster, where the port to access a broker is 6000 + the broker’s ID, ie. broker with ID of 6 is accessible via port 6006.Configuration details of the endpoint security group for enabling connectivity to a Kafka cluster via PrivateLink
  9. Click “Create security group”, the security group should be created successfully. Take note of the security group name, as it will be required when creating the Endpoint in the following steps.
  10. If you have any Kafka add-ons enabled in the PrivateLink Kafka cluster, please add the corresponding ports as well. For Kafka Schema Registry/Karapace Schema Registry, the port is 8085 and the Custom CIDR is the same as the one in step 7.

    While for Karapace Rest Proxy, there are two rules to add. The first rule has 8083 ports and the other is 8091 – 8090+p, where p is the number of Karapace Rest Proxy nodes in PrivateLink Kafka Cluster. The Custom CIDR for both rules are the same as the one in step 7. The Karapace Rest Proxy Nodes are tagged “Karapace REST Proxy” in the Details page in Instaclustr console.Note: Due to the technical limitation of how Kafka REST Proxy operates, its use is not supported with PrivateLink.
    If the PrivateLink Kafka cluster has both add-ons, with 3 Karapace Rest Proxy nodes, expect the following inbound rules.

Create Endpoint and Route 53 Record

  1. On the AWS VPC Endpoint Console page, click “Create endpoint”.

    Note: As per this document, AWS doesn’t allow attaching policies to an endpoint for an endpoint service other than an AWS service. Hence, by default, AWS allows full access to a VPC endpoint created for PrivateLink. However, the access can be controlled by adding a condition key to the IAM policy from the consumer side to restrict endpoint creation from specific endpoint services of allowed accounts only. Below is an example of an IAM policy which only allows VPC endpoint creation from PrivateLink enabled kafka cluster’s account.

  2. On the creation page under “Service category”, select “Other endpoint services”
  3. In “Service Settings”, paste the endpoint service name from step 2 to the “Service name” field. Click “Verify service” and a green box will appear.
  4. For “VPC”, select the client’s VPC that will connect to the PrivateLink Kafka cluster. Note, the VPC of the client and PrivateLink Kafka cluster should be in the same region as we do not support inter-region access via PrivateLink.
  5. The configuration would then be as follows.
  6. For “Subnets”, please select one “Subnet ID” for each “Availability Zone”. Please note that we recommend that your clients (producers and consumers) be located in separate availability zones to support high availability should any one availability zone be unreachable.
  7. Select the security group that you created earlier. You will get the following configurations. Once done, click “Create endpoint”.AWS Create Endpoint screen showing configuration details for connecting to a PrivateLink enabled Kafka cluster
  8. You will be redirected to the main VPC endpoint page which will specifically show the created endpoint. Once the Status of the endpoint reaches Available, copy the first DNS under the “DNS names”.
  9. Head to the Route 53 record home page, enter Hosted Zone and select “Create hosted zones”. https://us-east-1.console.aws.amazon.com/route53/v2/hostedzones?region=us-east-1#
  10. Find the “URL” in Retrieve Required Cluster Connection Information step 2. Under “Hosted zone configuration”, enter the “URL” without the port and first part of the hostname into “Domain name”. For example for kafka.test.com:9091, enter test.com.
  11. Select “Private hosted zone” as “Type”.
  12. Select the corresponding region and VPC of the client used in step 4. 
  13. Ensure you have the following configurations, click “Create hosted zone”.
  14. You will be redirected to the created private hosted zone.
  15. Select “Create record”.
  16. Under “Record name”, place the rest of the “URL” from step 2 without 9091. For instance, the “URL” is kafka.test.com:9091 and we entered test.com in  step 14. Thus the remaining URL to be entered is kafka.
  17. Click the toggle “Alias” and select “Alias to VPC endpoint” from the dropdown box.
  18. Select the same region as the one in step 3 in the next dropdown box.
  19. In the last field, paste the endpoint DNS name from step 8.
  20. The resulting configuration is as follows.
  21. The newly created record will be in the hosted zone. You can now connect to the cluster with kafka.test.com:9091.
  22. If you have any Kafka add-ons enabled in the PrivateLink cluster, find the instaclustr domain in the Connection Info page of the add-on. In the following Karapace Rest Proxy example, the Instaclustr domain is 7a690416c0ae49f89530f6018ab60c0e.cu.dev.instaclustr.com.

  23. Create a new hosted zone with the Instaclustr domain in step 22 by repeating step 9 to 14.
  24. Next, create a new record in the hosted zone in step 23 with privatelink in the “Record name” by repeating step 15 to 20. You can now connect to the add-ons with https://privatelink.7a690416c0ae49f89530f6018ab60c0e.cu.dev.instaclustr.com:{8083/8085}.

Questions

Please contact [email protected] for any further inquiries.

By Instaclustr Support
Previous Article Connect to a Private Service Connect Apache Kafka cluster Next Article Connect to a PrivateLink Apache Kafka cluster with AWS CloudFormation
Need Support?
Learn More
Experiencing difficulties on the website or console?
Status page for known incidents
Already have an account?
Log In to the Console
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes
Free Trial