Apache Kafka Client ⇄ Broker Encryption & Mutual Authentication (mTLS)

Overview

Client ⇄ Broker Encryption & Mutual Authentication is a feature to configure Apache Kafka® clusters with mutual TLS (mTLS) authentication, an extension to TLS. TLS is a security protocol to authenticate parties when establishing a connection and then encrypts the connection for its duration. The key difference is that with mTLS enabled, the identity of both the client and cluster is authenticated when establishing a connection, whereas with TLS only the identity of the cluster is authenticated. 

This page provides some general information around TLS and mTLS as applicable to Apache Kafka as well as some specifics around their implementation for Instaclustr for Apache Kafka.

Background Knowledge 

TLS

What is TLS? 

Transport Layer Security (TLS) is a widely used encryption protocol for internet connections and is commonly used to encrypt HTTPS connections. TLS was formerly called SSL and this name continues to be used in configuration settings for Kafka and throughout the Kafka Project’s documentation. 

For TLS authentication the identity of the broker is authenticated by the client using a TLS certificate provided by the broker. The connection is then encrypted ensuring third parties cannot intercept communication between the client and cluster. 

Core Concepts

Public & Private Key Pair 

TLS is based on public key cryptography, an encryption technique using 2 keys.

1. Public Key: The public key is used to encrypt information.
2. Private Key: The private key is used to decrypt information 

The public key can be shared, and is used by the client to encrypt information sent to the cluster. The private key is kept secret and ensures only the cluster is able to decrypt and read any information sent by the client. 

TLS Certificate 

The TLS certificate contains the important information required to verify the identity of the cluster, including the public key, certificate authority and expiration date.  

TLS Handshake 

To establish a connection the client and cluster complete a process called a TLS handshake. First the client and cluster exchange messages to verify the client has a valid copy of the TLS certificate and the cluster’s identity is valid, then the cluster demonstrates it has the correct private key. Finally, an encryption method is decided upon and used to encrypt further communication between the 2 parties. 

How Does TLS Work?  

The following process authenticates a TLS connection: 

1. Client downloads the cluster’s TLS certificate 
2. Client connects to cluster 
3. Client verifies the cluster’s TLS certificate matches the cluster’s identity 
4. Client checks the cluster has the correct private key 
5. Client and the cluster exchange TLS encrypted information

mTLS

What is mTLS? 

Mutual TLS (mTLS) is an extension of TLS with mutual authentication. To authenticate via an mTLS connection both the client and server have their own key pair. The mTLS authentication then takes place bi-directionally with each party checking the other has the correct private key and a valid TLS certificate. 

Why use mTLS? 

Mutual authentication offers an additional validation check compared to TLS and hence provides enhanced security. TLS is a popular choice for public-facing servers, such as websites, where clients are often unknown to the server. Whereas mTLS is a popular choice for private services, such as Kafka clusters, where it’s important that only validated clients are able to successfully connect.  

mTLS prevents the following attacks: 

• Man-in-the-middle (MitM): MitM attacks involve an attacker eavesdropping or altering communication in transit. If the client and server exchange messages without verifying the sender they can fail to identify an attacker intercepting messages in the middle of the communication line. mTLS prevents MitM attacks by verifying both parties. 
• Spoofing Attacks: A spoofing attack involves an attacker falsifying information to pretend to be a valid client or server. Mutual authentication can prevent spoofing attacks by requiring both parties to validate TLS certificates. 
• Stolen/Leaked Credentials: User credentials can be leaked by phishing scams or data breaches. Requiring a valid client certificate prevents an attacker connecting to a server with leaked credentials. 
• Brute Force Attacks: Attackers can try connecting to a server by rapidly trying different passwords. mTLS is one option to prevent against this by requiring a valid certificate instead of a password. 

How Does mTLS Work? 

Compared to TLS, there are additional steps for authenticating mTLS. The following process authenticates a mTLS connection: 

1. Client creates a Key Pair 
2. Client generates a Certificate Signing Request (CSR) 
3. Client uploads CSR to the cluster for the cluster to sign with it’s own Certificate Authority (CA) 
4. Client downloads signed certificate from the cluster 
5. Client downloads the cluster’s TLS certificate 
6. Client connects to cluster 
7. Client verifies the cluster’s TLS certificate matches the cluster’s identity 
8. Cluster verifies the client’s signed certificate matches the client’s identity 
9. Client checks the cluster has the correct private key 
10. Cluster checks the client has the correct private key 
11. Client and cluster exchange TLS encrypted information 

Next Steps 

1. Creating an Apache Kafka cluster with mTLS authentication
2. Managing certificates on mTLS enabled Apache Kafka clusters 
3. Configuring the keystore to use mTLS authentication with Apache Kafka Client

Questions 

Please contact [email protected] for any further inquiries.