NetApp is excited to announce the latest enhancement to Instaclustr for Apache Kafka®: the integration of Google Cloud’s Private Service Connect (PSC).

This new feature provides our customers with a private connection from their application VPC to their Apache Kafka clusters running on the NetApp Instaclustr Managed Platform, leveraging Google Cloud Platform (GCP)’s PSC feature as a more secure alternative to using VPC peering or internet gateways.

What is Private Service Connect? 

GCP’s Private Service Connect is a service that makes use of endpoints and service attachments to allow service consumers to send traffic from their VPC network to services in the producer’s VPC network.  

In the case of our managed Kafka offering, that means you can have your Kafka clients in one VPC with unidirectional access to the Kafka cluster in another VPC without requiring the use of VPC Peering or transit gateways.  

PSC is now the better option for environments where unidirectional access is required or to reduce network management complexity. This complements our existing service for AWS PrivateLink for Kafka, now providing equivalent functionality for customers operating across multiple cloud providers. 

Main Benefits of Using GCP Private Service Connect with Instaclustr for Apache Kafka include: 

• Enhanced security and unidirectional access: With Private Service Connect, like with VPC Peering, your Apache Kafka traffic stays within Google Cloud’s network, reducing exposure to threats and vulnerabilities associated with the public internet. This aligns with the highest security standards, ensuring that sensitive data remains confidential and secure while in transit. However, as access is one way with PSC, it can only be initiated via the client VPC side, providing the additional benefit of removing the possibility of access from the managed cluster VPC to your application VPC. Because of this extra protection, the use of PSC and similar solutions is an increasingly common compliance requirement in large enterprises. 
• Simplified network architecture: PSC simplifies the network architecture by removing the need to manage VPC CIDR ranges to avoid overlaps or cross-network configurations. You can now connect to your Apache Kafka clusters with ease, using Google Cloud’s native tools and services. 
• Seamless integration: With a single click you can enable PSC on a new managed Apache Kafka cluster making it easy to set up and manage. You can start leveraging this feature with minimal changes to your existing Kafka clients and cluster. 

Fitting GCP PSC to Kafka was not a straightforward task since to route traffic from Kafka clients to Kafka brokers: 

• GCP Application Load Balancers couldn’t be used as they only support HTTP or HTTPS whereas Kafka needs TCP, and 
• GCP Network Load Balancers and Internal Protocol Forwarding couldn’t be used by themselves since to use them would have required exposing each Kafka broker individually with one forwarding rule (in NetApp’s VPC) and one mapped VPC Connect Endpoint (in your client VPC). 

So, we worked with the NetApp’s Shotover project–an open source data-layer proxy–to collaboratively add support for Kafka into the project. With the use of Shotover, each rack/zone has one GCP Network Load Balancer (NLB) and a dedicated Shotover node. The Service Attachment for the rack delivers traffic to the NLB which sends it on to the Shotover node for that rack, and that routes it on to the Kafka brokers.  

With this architecture, customers using GCP PSC only need 3 VPC Connect Endpoints per cluster regardless of number of brokers in that cluster. Without Shotover, GCP PSC enabled Kafka clusters would have required one VPC Connect Endpoint per Kafka broker, leading to a setup which would need to be manually updated in the client VPC each time as the Kafka cluster was horizontally scaled. 

On our managed platform, GCP Private Service Connect feature is offered as an Enterprise Feature. Enterprise Features (and Enterprise Add-Ons) attract enterprise tier pricing. Each PSC enabled cluster will require three dedicated Shotover proxy nodes, with multiple sizes available and pricing listed on the Instaclustr Console.  

Additionally, Run In Your Own Account (RIYOA) customers considering using PSC should be aware that it will result in additional charges for the NLBs and other charges as levied by GCP for PSC. If your enterprise security standards required Private Service Connect, we advise getting in contact with our Sales team to understand how PSC could work to address your security and compliance requirements. 

Enabling your new Instaclustr for Apache Kafka cluster to have PSC support can now be done with a single click. For enabling PSC support on existing clusters, we advise getting in touch with our support team. Our support documentation provides information on how to set up your managed Kafka cluster to support PSC  and lists steps to help you connect to a PSC enabled a cluster. For further information or help getting started, please reach out to our support team.