Use Apache Kafka with the Command Line

Table of Contents

1. Prerequisites
2. Using client broker encryption (SSL)
3. Producing Messages
4. Consuming Messages
5. Putting Them Together

Introduction

In this example we will be using the command line tools kafka-console-producer and kafka-console-consumer that come bundled with Apache Kafka.

Prerequisites

The Apache Kafka package installation comes bundled with a number of helpful command line tools to communicate with Kafka in various ways. To get these tools you will need to download and install a Kafka release from here.

Using client ⇆ broker encryption (SSL)

If you have chosen to enable client ⇆ broker encryption on your Kafka cluster, see here for information on the certificates required to establish an SSL connection to your Kafka cluster.

Producing Messages

Client Configuration

Before creating a Kafka producer client, you first need to define the configuration properties for the producer client to use. In this example we provide only the required properties for the producer client. See here for the full list of configuration options.

Create a new file named producer.properties:

Ensure the truststore location and SCRAM password are correct. If your Kafka cluster does not have client ⇆ broker encryption enabled your configuration options should look like this:

Make sure the password is correct.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Run the Producer

Messages are produced to Kafka using the kafka-console-producer tool. This can be found in the bin directory inside your Kafka installation. In this example we provide only the required properties for the Kafka client. See here for more details on configuration options. Start the kafka-console-producer tool, making sure the broker-list and producer.config location are correct:

Once the tool has started, any text entered into the prompt will be sent to Kafka when you press Enter.

Consuming Messages

Client Configuration

As in the producer example, before creating a Kafka consumer client, you first need to define the configuration properties for the consumer client to use. In this example we provide only the required properties for the consumer client. See here for the full list of configuration options.

Create a new file named consumer.properties:

Ensure the truststore location and SCRAM password are correct. If your Kafka cluster does not have client ⇆ broker encryption enabled your configuration options should look like this:

Make sure the password is correct.

Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.

Run the Consumer

Messages from Kafka are consumed using the kafka-console-consumer tool. Like the producer, this can be found in the bin directory inside your Kafka installation. In this example we provide only the required properties for the Kafka client. See here for more details on configuration options. Start the kafka-console-consumer tool, making sure the bootstrap-server and consumer.config location are correct:

Any messages produced to Kafka while the consumer tool is running will be output to the console.

Putting Them Together

Now that we have a consumer and producer setup, it’s time to combine them.

Start the consumer

Start the consumer before starting the producer because by default consumers only consume messages that were produced after the consumer started.

Start the producer

Now that the consumer is setup and ready to consume messages, you can now start your producer.

If the consumer and producer are setup correctly the consumer should output the message sent by the producer shortly after it was produced:

Transparent, fair, and flexible pricing for your data infrastructure: See Instaclustr Pricing Here