Documentation

Creating a Private Service Connect Apache Kafka Cluster

This guide describes how to provision a Private Service Connect Kafka Cluster on the Instaclustr Platform. Private Service Connect is a feature provided by Google Cloud Provider that enables private connectivity between Google Cloud services and applications in your Virtual Private Cloud (VPC). By using PSC, you can establish a private data flow between Google Cloud services and your own VPC without exposing your data to the public internet, thereby increasing security. More information from GCP about PSC is available here.

Overview

A Kafka cluster with Private Service Connect feature enabled has its Kafka nodes accessible via Private Service Connect Service Attachments, which are resources that are used to create Private Service Connect published services. Kafka client applications can connect to the cluster from a VPC network using Private Service Connect Endpoints, which are created by deploying forwarding rules that reference the Private Service Connect Service Attachments. Traffic remains entirely within Google Cloud when connecting to a Private Service Connect Kafka cluster.

On a high level, client applications are connected to Kafka cluster using the below workflow:

Each Kafka PSC cluster has 3 published Private Service Connect Service Attachment which can be connected from the client VPC. Each Service Attachment is in its own private subnet inside the Kafka PSC Cluster Network.

Limitations

Enterprise add-ons arent yet supported, but will be supported soon with Kafka GCP Private Service Connect clusters..
We’re working to add support for these incrementally over time.
Private Service Connect Kafka clusters do not support mTLS at the moment, support for it will be added later.

Provisioning a cluster via Instaclustr Console

1 – Login to the Instaclustr Console and provision a new Kafka cluster and ensure you select the Private Service Connect feature, you will only be able to select this option if the provider selected is GCP.

2 – On the next page, specify the Kafka version, among other Kafka options, as you would when creating a standard Kafka cluster.

3 – On the same page, enable client-encryption and authentication if required. MTLS support and Enterprise Addons for PSC clusters are being implemented at the moment and will be available soon.

4 – On the next page, enter the Private Service Connect Service Attachment networks CIDR. This CIDR will decide the CIDRs of the subnets that hold the Private Service Connect Service Attachments. If no CIDR is provided, it defaults to the next available CIDR range after the Data Centre network.

5 – On the same page, enter the Advertised Hostnames. Advertised Hostnames are used by client applications to connect to the cluster. Each Advertised Hostname corresponds to one Private Service Connect endpoint, which in turn connects to one Service Attachment exposed by the cluster. For instance, if Advertised Hostnames provided are

us-central1a.kafka.test.com, us-central1b.kafka.test.com, us-central1c.kafka.test.com

Then client applications would connect to the cluster using these hostnames with the port number being 9091.

6 – On the same page, select and Kafka node size respectively. Shotover is a reverse proxy application that would reroute traffics coming from Kafka client to Kafka nodes, each Shotover instance serves as an aggregation of Kafka nodes that are in the same rack. As a general guideline, you should choose Shotover node sizes based on the expected amount of throughput the cluster would handle:

For throughput < 1GB/s, choose KSO-PRD-c2d-highcpu-4-40

For throughput between 1GB/s and 2GB/s, choose KSO-PRD-c2d-highcpu-8-40

For throughput > 2GB/s, choose KSO-PRD-c2d-highcpu-16-40

Note, the above serves as a general guideline, the actual node sizes required would vary based on several factors, such as number of Kafka nodes, average message size, number of partitions etc. You could always resize the Shotover instances size via the Console or API based on your actual workload.

7 –Under the Confirmation page, check that the configurations such as Advertised Hostnames are properly configured, accept Instaclustr terms and conditions, and click Create Cluster.

Please contact [email protected] if you have any questions.

Provisioning a cluster via Terraform

Below is an example for provisioning a Private Service Connect cluster via Terraform

terraform {
  required_providers {
    instaclustr = {
      source = "instaclustr/instaclustr"
      version = ">= 2.0.0, < 3.0.0"
    }
  }
}


provider "instaclustr" {
  terraform_key = "Instaclustr-Terraform <username>:<provisioning API key>"
}


resource "instaclustr_kafka_cluster_v3" "kafka-gcp-psc" {
  name = "<cluster_name>"
  private_network_cluster = true
  sla_tier = "<NON_PRODUCTION/PRODUCTION>"
  allow_delete_topics = true
  auto_create_topics = true
  bundled_use_only = false
  client_broker_auth_with_mtls = false
  client_to_cluster_encryption = true
  data_centre {
    cloud_provider = "GCP"
    name = "<DC name>"
    network = "<network>"
    provider_account_name = "<provider account name>"
    region = "<region>"
    node_size = "<nod>"
    number_of_nodes = 3
    private_connectivity {
       gcp_private_service_connect {
         gcp_private_service_connect_rack {
           advertised_hostname = "<rack 1 hostname>"
         }
         gcp_private_service_connect_rack {
           advertised_hostname = "<rack 2 hostname>"
         }
         gcp_private_service_connect_rack {
           advertised_hostname = "<rack 3 hostname>"
         }
         shotover_proxy {
           node_size = "<shotover node size>"
         }
       }
     }
  }
  default_number_of_partitions = 3
  default_replication_factor = 3
  kafka_version = "<kafka version>"
  kraft {
    controller_node_count = 3
  }
  pci_compliance_mode = false
}

terraform {

required_providers {

instaclustr = {

source = "instaclustr/instaclustr"

version = ">= 2.0.0, < 3.0.0"

}

provider "instaclustr" {

terraform_key = "Instaclustr-Terraform <username>:<provisioning API key>"

}

resource "instaclustr_kafka_cluster_v3" "kafka-gcp-psc" {

name = "<cluster_name>"

private_network_cluster = true

sla_tier = "<NON_PRODUCTION/PRODUCTION>"

allow_delete_topics = true

auto_create_topics = true

bundled_use_only = false

client_broker_auth_with_mtls = false

client_to_cluster_encryption = true

data_centre {

cloud_provider = "GCP"

name = "<DC name>"

network = "<network>"

provider_account_name = "<provider account name>"

region = "<region>"

node_size = "<nod>"

number_of_nodes = 3

private_connectivity {

gcp_private_service_connect {

gcp_private_service_connect_rack {

advertised_hostname = "<rack 1 hostname>"

}

gcp_private_service_connect_rack {

advertised_hostname = "<rack 2 hostname>"

}

gcp_private_service_connect_rack {

advertised_hostname = "<rack 3 hostname>"

}

shotover_proxy {

node_size = "<shotover node size>"

}

default_number_of_partitions = 3

default_replication_factor = 3

kafka_version = "<kafka version>"

kraft {

controller_node_count = 3

}

pci_compliance_mode = false

}

See https://registry.terraform.io/providers/instaclustr/instaclustr/latest/docs/resources/kafka_cluster_v3 for more information.

By Instaclustr Support

Previous Article Connecting to a Private Service Connect Apache Kafka Cluster Next Article Creating an Apache Kafka Cluster

Need Support?

Learn More

Experiencing difficulties on the website or console?

Status page for known incidents

Already have an account?

Need help with your cluster?

Contact Support

Why sign up?

To experience the ease of creating and managing clusters via the Instaclustr Console

Spin up a cluster in minutes

Free Trial