Connect to a PrivateLink Apache Kafka cluster with AWS CloudFormation

Overview

This page includes the required AWS CloudFormation templates for connecting to a PrivateLink Kafka Cluster and the general steps of using the AWS CloudFormation template. For a manual approach, please refer to Connect to a PrivateLink Kafka Cluster.

Table of Contents

AWS CloudFormation Template For PrivateLink Kafka Cluster without Kafka AddOns (JSON)

 

AWS CloudFormation Template For PrivateLink Kafka Cluster without Kafka AddOns (YAML)

 

Notes on AWS CloudFormation Template For PrivateLink Kafka Cluster with Kafka AddOns

The default templates include both Karapace Schema Registry/Kafka Schema Registry and Karapace Rest Proxy Kafka Add-Ons. Please modify accordingly by referencing the following table. The table applies to both JSON and YAML.

 

Kafka Add-Ons Parameters/Values to keep in the default template
Karapace Schema Registry / Kafka Schema Registry
  • SecurityGroupIngress: Keep the 8085 rule.
Karapace Rest Proxy
  • Parameters: Keep the RestProxyNodes
  • SecurityGroupIngress: Keep the 8083 and 8091-809n rule, where n is the RestProxyNodes

 

AWS CloudFormation Template For PrivateLink Kafka Cluster with Kafka AddOns (JSON)

 

AWS CloudFormation Template For PrivateLink Kafka Cluster with Kafka AddOns (YAML)

 

Section 1: Using the AWS CloudFormation template for PrivateLink Kafka Cluster without Kafka AddOns

  1. Download one of the templates above. For this example, we will use the JSON template for PrivateLink Kafka Cluster without Kafka AddOns, with the name instaclustr-kafka-privatelink.json.
  2. Login into your AWS Account and head to the AWS CloudFormation page. Please ensure the region is the same as the created PrivateLink Kafka Cluster. https://us-west-2.console.aws.amazon.com/cloudformation/home?region=us-west-2#/
  3. Select “Create Stack” and upload the saved template, instaclustr-kafka-privatelink.json. Select “Next”.
  4. On the next page, you would see a form similar to the following. Fill the “Stack Name” section. There are a total of 8 parameters to fill, see the relevant steps to get the corresponding values.
    • ClientCIDRIp: Step 8
    • ClientSubnets: Step 8
    • ClientVPC: Step 8
    • ClusterBrokers: Step 5
    • ClusterEndpointServiceName: Step 6
    • ClusterRegion: Step 5
    • HostedZoneExistingID: Step 14
    • HostedZoneName: Step 6
    • HostedZoneSubdomain: Step 6

  5. Log into Instaclustr Console, click the PrivateLink Kafka Cluster created and enter the Details page.
    Count the number of brokers for the “ClusterBrokers” variable. Additionally, the “Region” section is the value for the parameter, “ClusterRegion”. For example, US_EAST_1 is converted to us-east-1.  In the following example, “ClusterBrokers” is 3, while “ClusterRegion” is us-west-2.
  6. Next, enter the Connection Info page. The steps are similar to the ones outlined in Retrieve Required Cluster Connection Information in Connecting to a PrivateLink Kafka Cluster .
    • “Endpoint Service Name” is related to the parameter, “ClusterEndpointServiceName”, which is com.amazonaws.vpce.us-west-2.vpce-svc-0a5e785f4741a01f6.
    • With the advertised hostname kafka.test.com, shown in the “URL”, the “HostedZoneName” is test.com, while the “HostedZoneSubdomain” is kafka.
  7. Before moving on, please ensure you have added the principal ARN. Please refer to this document on the process.
  8. Find the client EC2 instances that will connect to the PrivateLink Kafka Cluster. For instance, companyName-client. Find the “ClientVPC” and the “ClientCIDRIp” in the VPC console.https://us-west-2.console.aws.amazon.com/vpc/home?region=us-west-2#vpcs:tag:Instaclustr:ClusterName=companyName-client.In this case, “ClientVPC” is vpc-0402d63e62fe0ceb0 and “ClientCIDRIp” is 10.1.0.0/16.The “ClientSubnets” can be found in the EC2 Console. https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=us-west-2#Instances:instanceState=running
    Please ensure the region is the same as the created PrivateLink Kafka Cluster.If there are multiple client instances, please be sure to have all the subnets comma separated. For instance, enter subnet-01534d20fd7c6021c,subnet-025aada7d71ff240f,subnet-011a91d95e541d1f0 as “ClientSubnets”

  9. Once all the values are filled, you should get the following:

  10. Enter “Next” and the following form should appear. Please add a tag with key Name and value as instaclustr-privatelink-resources, as this would help in debugging should issues arise.Please update the other permission and policies if needed, otherwise, leave it as default. When it is done, select Next.
  11. On the next page, ensure the details are correct and hit “Create Stack”.

  12. The page will be redirected back to the AWS CloudFormation page. The progress will be visible in the Events or Resource page. As shown below, the resource creation is currently in progress.
  13. Once completed, the progress will be displayed as CREATE_COMPLETE, which means the client can start connecting to the PrivateLink Kafka Cluster. You can now connect to the cluster with kafka.test.com:9091 and connect to Kafka Schema registry/Karapace Schema Registry/Karapace Rest Proxy with https://privatelink.7a690416c0ae49f89530f6018ab60c0e.cu.dev.instaclustr.com:{8083/8085}
  14. If there is an existing hosted zone, please find the hosted zone ID under Hosted zone details for the parameter, “HostedZoneExistingID”. For instance, if there is already an existing hosted zone “test.com”, the value of  “HostedZoneExistingID” will be Z01496251VVULYEUFH2GM. By specifying the value, the AWS CloudFormation template would not attempt to re-create another hosted zone with the same name.

Section 2: Using the AWS CloudFormation template for PrivateLink Kafka Cluster with Kafka AddOns

The process is the same as the steps listed in Section 1: Using the AWS CloudFormation template for PrivateLink Kafka Cluster without Kafka AddOns. For the purpose of this section, we will refer Section 1: Using the AWS CloudFormation template for PrivateLink Kafka Cluster without Kafka AddOns as Section 1.

  1. Download the PrivateLink Kafka Cluster with Kafka AddOns AWS CloudFormation template. Please remember to modify the template according to the existing add-ons on the PrivateLink Kafka cluster. In this example, we will use the default JSON template with both Karapace Schema Registry and Karapace Rest Proxy, named as instaclustr-kafka-privatelink-addon.json
  2. Follow step 2 and 3 in Section 1 to upload the template to AWS CloudFormation, and select “Next”.
  3. The form would be similar to the one in step 4 in Section 1. The difference is the additional two parameters: “InstaclustrDomain” and “RestProxyNodes”.

  4. For “InstaclustrDomain”, enter the Connection Info page of the PrivateLink Kafka cluster with Kafka Add-Ons. There should be tabs for each Kafka Add-Ons. Select one of them.
    For example, under Karapace Rest Proxy, the URL is https://karapace-proxy.7a690416c0ae49f89530f6018ab60c0e.cu.dev.instaclustr.com:8083. The “InstaclustrDomain” is 7a690416c0ae49f89530f6018ab60c0e.cu.dev.instaclustr.com.
  5. As for “RestProxyNodes”, enter the Details page of the PrivateLink Kafka cluster with Kafka Add-Ons, and count the number of nodes with the tag “Karapace REST Proxy”. In the following case, “RestProxyNodes” is 3.
  6. Follow step 4 to 8 in Section 1 to fill the other parameters, and hit “Next”.

  7. From hereon, follow the rest of Section 1 steps from step 10.
By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes