Managing Certificates on a Client ⇄ Broker Encryption & Mutual Authentication (mTLS) Enabled Apache Kafka Cluster

Overview

This article describes how to manage certificates for an Apache Kafka Cluster with Client Broker Encryption & Mutual Authentication (mTLS).  

See also

Prerequisites 

  • You need to have a running  Instaclustr Apache Kafka cluster with Client ⇄ Broker Encryption & Mutual Authentication (mTLS) enabled. Please refer to creating an Apache Kafka cluster with mTLS authentication for guidance if you require a new Kafka cluster.
  • This guide uses keytool – a key and certificate management utility which is bundled with the Java Development Kit (JDK). More information can be found on the keytool documentation. Another tool or service can be used instead, and many options are available. 

Managing Client Certificates Using Instaclustr Console

mTLS Kafka clients require a Kafka user to connect to a Kafka cluster as well as a certificate to authenticate the connection.

You need to generate a Certificate Signing Request (CSR) for the user and upload it to console for the server. The server will generate a signed user certificate using the Instaclustr Certificate Authority (CA).

This signing process allows the server to recognise the Kafka client’s certificate during the authenticating handshake for the user and is fundamental to the mutual aspect of mTLS. 

Add a User for mTLS

  1. Click on the created cluster and enter the Users page.

  2. Click the Add New User button to add a new user.

  3. Complete the form fields with the preferred Username, Initial Permissions and Authentication Method. As an example, this guide shows how to create a user named “mtls”, with standard Initial Permissions and the MTLS authentication method.
    There are two options for Authentication Method on clusters with mTLS enabled, ‘MTLS’ & ‘SASL’:

    • A user with ‘SASL’ authentication selected can establish connections through both the SASL and mTLS listeners. Note, the SASL Authentication Mechanism won’t be utilized for an mTLS connection. Instead, the connection will be authenticated using a signed client certificate.
    • A user with MTLS’ authentication can exclusively connect via the mTLS listener.

    Select either method based on your preferences when following this guide.
    SASL authentication can be enabled or disabled later by selecting Manage Password. Disabling SASL authentication, by removing the password, changes the authentication method to ‘MTLS’. Similarly enabling SASL authentication, by adding a password, changes the authentication method to ‘SASL’.

  4. Click Add User.

    Refer to Apache Kafka User Management for additional information on creating users and permission options. The username should match the Common Name (CN) provided while generating the CSR in the following section. 

Generate a Certificate Signing Request 

Generate a keystore for the client and then generate a CSR. We will use keytool for this guide. However, another tool or service could be used for keystore management.

  • Example command to generate a keystore with keytool in a terminal

  • Example command to generate a CSR with keytool in a terminal

Generate a Signed User Certificate

Lastly, you will need to provide Instaclustr CA with the CSR so that it can generate a signed user certificate which can be downloaded for use with your clients. Ensure that the Common Name (CN) matches the username when you add the signed user certificate to your client keystore.

  1. Click Manage Certificates on the row of the newly created user
     
  2. Click Add New Certificate to create a user certificate by uploading a CSR.

  3. Click SELECT A CLIENT CERTIFICATE FILE, navigate to the newly created CSR file, select it, and then click Upload. Console will validate the request, and it will appear in the CERTIFICATE SIGNING REQUEST box. Alternatively, you can paste the request straight into this box instead of uploading the file.Press the file icon to select a certificate file from your computer
  4. Enter the desired validity period of the signed user certificate in months and then click Create.Fill in the details of the certificate and then Press the Create button once you are done
  5. The newly created Signed User Certificate will now be listed on the Kafka User Certificate Management page.  
  6. Download the certificate by clicking Download.

  7. Returning to the Users page, you can see MTLS authentication is configured for your user.

Congratulations, you have now configured the certificates for mTLS authentication and are one step closer to a more secure infrastructure. See how to configure your local client keystore with an example on how to connect your client with mTLS authentication to complete securing your infrastructure. 

Managing Client Certificates Using Instaclustr APIv2 

Instaclustr APIv2 provides capabilities to sign your certificates via REST endpoints. See Generate a Certificate Singing Request above for more information on CSR generation. 

Refer to this documentation for more information and options to POST the CSR. 

Note: Each line in the CSR must be separated with a \r\n when passed in a JSON request body. 

The API can also be used to list the certificate IDs for a user and download a Kafka certificate using the certificate ID. See the linked documentation for more information on how to list the certificate ID and download the signed certificates  

Managing Client Certificates Using Instaclustr Terraform Provider v2 

Instaclustr provides Terraform capabilities with Instaclustr Terraform Provider v2, see Using the Instaclustr Terraform Provider for more information.  

For more information on the capabilities to sign and import the signed certificate as a resource, refer to this Instaclustr Terraform Documentation. 

Limitations 

  1. There is no way to revoke client certificates, but the equivalent goal (for example, preventing access to the topics on your cluster for a particular user) can be achieved by changing the associated users Kafka ACLs. 
  2. Client certificates need to be signed using the Instaclustr Certificate Authority (CA) for your cluster. Importing and using your own CA for certificate signing is not supported. 

Additional Resources 

Refer to the following resources for further information on mTLS with Apache Kafka: 

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes