Connecting to a Private Service Connect Apache Kafka Cluster
This guide aims to provide basic instructions on how customers can connect to their Kafka Private Service Connect Cluster from their applications. See Creating a Private Service Connect Apache Kafka Cluster for more information on how Private Service Connect Kafka clusters work and how to create one.
Infrastructure deployment via GCP Console
Retrieve Required Cluster Connection Information
On the console connection information page, retrieve the published Service Attachment names:
These will be the service attachments to connect to in the client VPC.
Create Private Service Connect Endpoints
On the google cloud console, create Private Service Connect endpoints that connect to the above Service Attachments.
- On the GCP console, search for Private Service Connect.
- Under the Connected Endpoints tab, click on + CONNECT ENDPOINT to create an endpoint:
In the Connect Endpoint page, specify the following options:
- Target: Published Service
- Target Service: Service attachment path:
projects/<project name> /regions/<region>/serviceAttachments/<service attachment name above> - Endpoint name: choose a name that makes sense in your client VPC.
- Network: the network where your client applications are.
- Subnetwork: the subnetwork where your client applications are.
- IP address: create a new IP or using a pre-created IP address you like.
Enable global access: enable it if you wish to allow instances from another VPC to access the endpoint.
- Perform the above step for all Service Attachments.
Configure DNS Zone and Entries
Configure DNS entries so your client instances can connect to your cluster over the advertised hostnames you specified:
- (Optional) Create an internal Cloud DNS zone that is available in your client VPC.
- Create an entry that maps an Advertised Hostname to the IP of a Private Service Connect endpoint. Using the example above:
kafka-psc-test.instaclustr.com maps to X.X.X.X (the IP of a Private Service Endpoint) - Do the above step for each of the Private Service Connect endpoints.
- (Optional) You could also have one DNS name that routes randomly to one of the Private Service Endpoints, this DNS name could be used as bootstrap server for your client applications.
Summary
Above are instructions on how customers might set up their client-side connection to a Private Service Connect cluster. Other infrastructure modifications might be required, such as firewall rules and route table changes. Please contact [email protected] for further questions.
Infrastructure deployment via Terraform
Below are examples for deploying necessary infrastructure to connect to a Private Service Connect Kafka cluster.
Client VPC Network
The following Terraform providers are required:
|
1 2 3 4 5 6 7 8 9 |
provider "google" { project = "<client side project-id>" region = "<client applications region>" } provider "google-beta" { project = "<client side project-id>" region = "<client applications region>" } |
A typical VPC network would have the following Terraform definitions:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
resource "google_compute_network" "client-network" { name = "<network name>" auto_create_subnetworks = false # other settings for your network... } resource "google_compute_subnetwork" "subnet-a" { name = "subnet-a" ip_cidr_range = "<ip cidr range>" region = "<region>" network = google_compute_network.client-network.id } resource "google_compute_subnetwork" "subnet-b" { name = "subnet-b" ip_cidr_range = "<ip cidr range>" region = "<region>" network = google_compute_network.client-network.id } resource "google_compute_subnetwork" "subnet-c" { name = "subnet-c" ip_cidr_range = "<ip cidr range>" region = "<region>" network = google_compute_network.client-network.id } |
Note that the above examples are simply to provide reference points for the below Terraform resources, but actual configurations will vary based on customers’ needs. For instance, customers might choose to enable auto_create_subnetworks in their VPC.
More configs/subnets can be added, see google_compute_network and google_compute_subnetwork for more information.
IP Addresses
3 internal IP addresses are required because 3 forwarding rules are required to connect to the 3 published Service Attachments.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
resource "google_compute_address" "a-ip-address" { provider = google-beta name = "a-ip-address" address_type = "INTERNAL" subnetwork = google_compute_subnetwork.subnet-a.id } resource "google_compute_address" "b-ip-address" { provider = google-beta name = "b-ip-address" address_type = "INTERNAL" subnetwork = google_compute_subnetwork.subnet-b.id } resource "google_compute_address" "c-ip-address" { provider = google-beta name = "c-ip-address" address_type = "INTERNAL" subnetwork = google_compute_subnetwork.subnet-c.id } |
See google_compute_address for more information.
Forwarding Rules
Forwarding rules are what connect the client VPC to the published services. They serve as the bridge between client applications and Kafka. Three forwarding rules are required to connect to the three published Service Attachments.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
resource "google_compute_forwarding_rule" "a-forwarding-rule" { provider = google-beta project = google_compute_network.client-network.project name = "<forwarding rule name>" target = "<service attachment URL in subnet a>" network = google_compute_network.client-network.id ip_address = google_compute_address.a-ip-address.id load_balancing_scheme = "" no_automate_dns_zone = true } resource "google_compute_forwarding_rule" "b-forwarding-rule" { provider = google-beta project = google_compute_network.client-network.project name = "<forwarding rule name>" target = "<service attachment URL in subnet b>" network = google_compute_network.client-network.id ip_address = google_compute_address.b-ip-address.id load_balancing_scheme = "" no_automate_dns_zone = true } resource "google_compute_forwarding_rule" "c-forwarding-rule" { provider = google-beta project = google_compute_network.client-network.project name = "<forwarding rule name>" target = "<service attachment URL in subnet c>" network = google_compute_network.client-network.id ip_address = google_compute_address.c-ip-address.id load_balancing_scheme = "" no_automate_dns_zone = true } |
Cloud DNS (Optional)
Clients might want to set up a private cloud DNS zone that contains records to resolve the advertised hostnames (provided when provisioning the cluster) to the IPs used by the forwarding rules. This is so that the client applications can connect to Kafka using the advertised hostnames.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
resource "google_dns_managed_zone" "private-zone" { name = "<name>" dns_name = "<private DNS zone name >" description = "Example private DNS zone" visibility = "private" private_visibility_config { networks { network_url = google_compute_network.client-network.id } } } resource "google_dns_record_set" "a-record" { name = "<advertised hostname for PSC service attachment in subnet a>" type = "A" ttl = 300 managed_zone = google_dns_managed_zone.private-zone.name rrdatas = [google_compute_address.a-ip-address.address] } resource "google_dns_record_set" "b-record" { name = "<advertised hostname for PSC service attachment in subnet b>" type = "A" ttl = 300 managed_zone = google_dns_managed_zone.private-zone.name rrdatas = [google_compute_address.b-ip-address.address] } resource "google_dns_record_set" "c-record" { name = "<advertised hostname for PSC service attachment in subnet c>" type = "A" ttl = 300 managed_zone = google_dns_managed_zone.private-zone.name rrdatas = [google_compute_address.c-ip-address.address] } resource "google_dns_record_set" "c-record" { name = "<bootstrap DNS name, e.g. kafka-psc-bootstrap>" type = "A" ttl = 300 managed_zone = google_dns_managed_zone.private-zone.name rrdatas = [ google_compute_address.a-ip-address.address, google_compute_address.c-ip-address.address, google_compute_address.a-ip-address.address] } |
Alternatively, clients could insert above DNS entries at the instance/application level, e.g. putting them into /etc/hosts on a client instance.
Summary
Above are simple examples of how customers might set up their client-side connection to a Private Service Connect cluster. Other infrastructure modifications might be required, such as firewall rules changes. Please contact [email protected] for further questions.
By Instaclustr Support