Managing Apache Kafka Topics

Instaclustr has several different tools that you can use to manage Topics in Apache Kafka. These include a CLI tool, the Instaclustr API, and the Instaclustr Terraform Provider.

Table of Contents

Kafka Topic Management via CLI

Kafka comes with its own topic management tools that can be downloaded from here. For our offering of Kafka 2.6.1 and newer, you can use Kafka-topics.sh tool that they have provided. However, for our older offerings, you will need to use ic-Kafka-topics.

In the following instructions, we will assume the use of ic-Kafka-topics.

Introduction to Kafka Topic Management

Ic-Kafka-topics is a tool developed by Instaclustr that can be used to manage Kafka topics using a connection to a Kafka broker.  Ic-Kafka-topics is based on the standard Kafka-topics tool, but unlike Kafka-topics, it does not require a zookeeper connection to work.

The core “actions” supported by ic-Kafka-topics include:

  • list – list the topics available on the cluster
  • create – create a topic
  • describe – provide details of one or more topics
  • alter – change the properties of one or more topics
  • delete – delete one or more topics

Installing ic-kafka-topics

Ic-Kafka-topics is available as a tarfile, downloadable from the Instaclustr Console cluster connection page.  To install:

  1. Download the tarfile from the Instaclustr console
  2. Unpack the tarfile:
    tar -xf kafka_2.11-2.1.2.tgz
  3. Go to the bin directory of the Kafka installation
    > cd kafka_2.11-2.1.2/bin
  4. Run ic-kafka-topics.sh
    > ./ic-kafka-topics.sh

This should display help text for ic-Kafka-topic.sh:

Using ic-kafka-topics

Once you have downloaded and installed ic-Kafka-topics, you can use it to manage topics with your cluster.

To enable ic-Kafka-topics to connect to your cluster you will always need to provide two options to the tool:

Option Description
‑‑bootstrap-server provides the details of one of the nodes (brokers) in your Kafka cluster in the form host:port.  These details are available on the Instaclustr console in the cluster connections page.
‑‑properties-file a path to a file which contains details of your cluster username and password.  These details are also available from the customer support site.

A minimal properties file requires your userid and password in the following format:

(carefully note the placement of the backslashes and semicolon at the end of the lines)

If your Kafka cluster has client ⇆ broker encryption enabled your properties file should look like this, ensuring the password and truststore location are correct:

You can find out more about using certificates with Kafka clients including where to find them here.

All usage of the ic-Kafka-topics command will specify these details.  For example, to list the topics on your cluster, run:

The following sections describe each of the actions supported by the tool.  You can get help at any time at the command line by running ic-Kafka-topics.sh with no arguments.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

List

The List action provides a listing of the topics that are available on the cluster.  To use the List action, include the ‑‑list option

Option Description
‑‑list the list action

Example:

Create

The “Create” action is used to create a new topic.

For simple topic creation

For simple topic creation: use the Create action.  You must provide the following details:

Option Description
‑‑create the create action
‑‑topic <topic-name> the name of the topic your wish to create
‑‑partitions <num-of-partitions> the number of partitions you want the topic to have
‑‑replication-factor <replication-factor> the replication factor for the topic.  may not be greater than the number of brokers
‑‑config <key>=<value> (Optional) a custom config value to set for this cluster. This can be omitted to use default configuration settings. You can specify more than one config to be changed using multiple ‑‑config options

Example:

The following example will create a topic called “mytopic” with 9 partitions and a replication factor of 3.

Kafka will automatically assign partitions to available brokers.

Creating topics with a replica-assignment

If you want more control about the assignment of the new topic’s partitions to existing brokers, you may provide a manual partition to broker assignment using the ‑‑replica-assignment option.

Describe

The “describe” action is used to provide a detailed description of a topic.  You need to provide the following details

Option Description
‑‑describe the creation action
‑‑topic <TopicName> the name of the topic to be altered.  If this is not specified, all topics will be described.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Only configurations which are not set to defaults will be displayed.  This includes values inherited from broker default configs or values which have been set for the topic.

For more details see the description of the output format for the standard Kafka-topics tool here:

Example:

Alter

The “alter” action is used to set custom configurations on the topic.  You can configure additional custom configurations on a property using the ‑‑alter action.

To set a configuration on a topic you must specify the following options:

option description
‑‑alter the alter action
‑‑topic <topic-name> the name of the topic to be altered.  If this is not specified, all topics will be modified.  A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.
‑‑config <key>=<value> a config value to change.  You can specify more than one config to be changed using multiple ‑‑config options
‑‑delete-config <key> a config value to remove.  You may specifiy more than one config to be removed using multiple ‑‑delete-config options

Examples

The following command will alter the retention.ms configuration on the topic called “mytopic“.

The following command will alter two values (retention.ms and compression.type) on the topic called “mytopic

The following command will set retention.ms to 50000000 on all topics starting with the string “test

The following command will change the min.insync.replicas property value for this topic and override the default min.insync.replicas value of the broker.

Deleting a configuration

To remove a custom configuration from a topic, use ‑‑alter combined with the ‑‑delete-config option.  The ‑‑delete-config option should specify the name of the configuration to remove.  Removing a custom configuration will cause the value to revert to the default value.

Example:

The following command will remove the retention.ms configuration, reverting it to the default value.

Delete (Topic Deletion)

The Delete action is used to a delete a topic.  To delete a topic you must specify the following options:

option description
‑‑delete the delete action
‑‑topic <topic-name> the name of the topic to be altered.

A (java-style) regular expression can also be specified to select multiple topics.  Regular expressions must be enclosed in double quotations.

Examples

The following command will delete the topic called “mytopic“.

The following command will delete the all topics starting with “dev”.

Kafka Topic Management via Instaclustr API

Instaclustr provides Topic Management API for Kafka clusters to help you with managing topics. The Topic Management API supports the following functions:

  • List Topics
  • Create a Topic
  • Delete a Topic
  • Get Topic information (number of partitions and replication factor)
  • Get Topic configuration
  • Modify Topic configuration

These examples show how to use the features of the Instaclustr Topic Management API.

For each endpoint listed below, all requests must include basic authentication details:

List Topics

  1. To retrieve a list of topics currently available in the Kafka cluster, make a GET request to:

  2. On a successful request, the API should respond with a 200 status code and a JSON containing the list of topics. E.g.:

Create a Topic

  1. To create a new topic to the Kafka cluster make a POST request to:

    A request body should be included, specifying the topic name, number of partitions, and replication factor. E.g.:

    Number of partitions and the replication factor should be non-zero integers. The replication factor is limited by the number of available brokers in the clusters. Be aware of creating a topic with too many partitions, as it can slow down the cluster.

  2. On successful topic creation, the API should respond with a 201 status code and a short message. E.g.

    Once this message is received, the topic should be immediately available in the Kafka cluster.

Delete a Topic

  1. To delete an existing topic from the Kafka cluster make a DELETE request to:

    No request body is required. The topic to be deleted is specified in the URL.

  2. When the topic is successfully deleted, the API should respond with a 200 status code and a short message. E.g.

Get Topic Information

  1. To retrieve the topic information of a particular topic available in the Kafka cluster, make a GET request to:

  2. The API should respond with a 200 status code and a JSON containing the information of the requested topic. E.g.:

Get Topic Configuration

  • To retrieve the topic configuration of a particular topic available in the Kafka cluster, make a GET request to:

  • On a successful request, the API should respond with a 200 status code and a JSON containing the configuration of the requested topic. E.g.:

Modify a Topic Configuration

  1. To modify the configuration of a topic in the Kafka cluster, make a PUT request to:

    A request body should be included, specifying the configuration that you want to change. E.g.:

    We do not need to provide all of the available keys in the request body, only the keys of interest are required in the body.

    For available configuration keys and values, consult https://kafka.apache.org/23/documentation.html#topicconfigs . Alternatively, available configuration keys can also be inferred from the output of the get topic configuration endpoint. Note if you are using Kafka version that is newer than 2.3: Currently, we can’t change the message.format.version to a version that is newer than 2.3. Please use the Kafka CLI tools (kafka-configs.sh) to modify this configuration.

  2. The API should respond with a 200 status code and a short message. E.g.

Kafka Topic Management via Instaclustr Terraform Provider

Instaclustr also provides a way to manage Kafka ACL using Terraform. For more information and examples, have a look at

The documentation for the ACL resource is located in docs/resources/kafka_topic.md and the ACL data source is located in docs/data-sources/kafka_topic.md .

By Instaclustr Support
Need Support?
Experiencing difficulties on the website or console?
Already have an account?
Need help with your cluster?
Contact Support
Why sign up?
To experience the ease of creating and managing clusters via the Instaclustr Console
Spin up a cluster in minutes