Setting up Mirror Maker
Mirror Maker is a feature included in Kafka which allows for maintaining a replica of a Kafka cluster in a separate data centre. It uses the existing consumer and producer APIs to achieve this.
Prerequisites
The Apache Kafka package installation comes bundled with a number of helpful command line tools to communicate with Kafka in various ways. To get these tools you will need to download and install a Kafka release from here.
Using client ⇆ broker encryption (SSL)
If you have chosen to enable client ⇆ broker encryption, see here for information on the certificates required to establish an SSL connection to your cluster.
Setting up Mirror Maker
This example demonstrates how to set up a standalone Mirror Maker instance. To create a distributed Mirror Maker cluster, simply repeat the following steps on any number of machines. As long as the config files are the same, the Kafka clusters will ensure the load is balanced and fault tolerant across the Mirror Maker instances.
Consumer Configuration
The consumer is the part of Mirror Maker which reads data from the source cluster it is intending to replicate.
Create a file named consumer.properties with the following content:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
bootstrap.servers=123.456.789.101:9092,321.654.987:9092,54.208.136.190:9092 exclude.internal.topics=true client.id=mirror_maker_consumer group.id=mirror_maker_consumer ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 ssl.truststore.location = truststore.jks ssl.truststore.password = instaclustr ssl.protocol=TLS security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="361d4871ff1a5ef58deaf3b887b4898029faee9690e62c549078a1f51f18f755"; |
If your cluster does not have client ⇆ broker encryption enabled, your file should instead look like this:
|
1 2 3 4 5 6 7 8 9 |
bootstrap.servers=123.456.789.101:9092,321.654.987:9092,54.208.136.190:9092 exclude.internal.topics=true client.id=mirror_maker_consumer group.id=mirror_maker_consumer security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="361d4871ff1a5ef58deaf3b887b4898029faee9690e62c549078a1f51f18f755"; |
Make sure the bootstrap.server IPs, truststore location if using SSL, and password are correct.
Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.
Producer Configuration
The producer is the part of Mirror Maker that uses the data read by the and replicates it to the destination cluster.
Create a file named producer.properties with the following content:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
bootstrap.servers=18.221.218.17:9092,18.191.42.69:9092,13.59.254.173:9092 acks=1 batch.size=100 client.id=mirror_maker_producer ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 ssl.truststore.location = producer_truststore.jks ssl.truststore.password = instaclustr ssl.protocol=TLS security.protocol=SASL_SSL sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="361d4871ff1a5ef58deaf3b887b4898029faee9690e62c549078a1f51f18f755"; |
If your cluster does not have client ⇆ broker encryption enabled, your file should instead look like this:
|
1 2 3 4 5 6 7 8 9 |
bootstrap.servers=18.221.218.17:9092,18.191.42.69:9092,13.59.254.173:9092 acks=1 batch.size=100 client.id=mirror_maker_producer security.protocol=SASL_PLAINTEXT sasl.mechanism=SCRAM-SHA-256 sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ickafka" password="361d4871ff1a5ef58deaf3b887b4898029faee9690e62c549078a1f51f18f755"; |
Make sure the bootstrap.server IPs, truststore location if using SSL, and password are correct.
Note: To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.
Running Mirror Maker
Now that we have configured Mirror Maker’s producers and consumers, we are almost ready to run it.
Tweak Kafka memory options with the following command as the defaults are not enough to accomodate Mirror Maker:
|
1 |
export KAFKA_HEAP_OPTS='-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80' |
Run using the script found in the bin directory of your Kafka installation:
|
1 |
bin/kafka-mirror-maker.sh --consumer.config consumer.properties --producer.config producer.properties --num.streams 3 --whitelist=".*" |
The --whitelist parameter configures which topics are mirrored. It can be a comma separated list of topic names or a Java regular expression. The value “.*” copies all topics on the source cluster to the destination cluster. The “exclude.internal.topics=true” in consumer.properties ensures Mirror Maker does not copy any of Kafka’s internal topics. A blacklist option is also available.
The --num.streams option determines the number of consumers Mirror Maker should use. The more consumers you use the less time it takes for data to be replicated from the source cluster to the destination cluster.
Once it is running it will not output anything. The best way to test is to consume from the destination cluster to see if messages are being replicated.
Best Practices
It is a good idea to create any topics that are being mirrored on the destination cluster before starting Mirror Maker. Mirror Maker can create the topics automatically but they may not retain the exact same configuration as the originals.
If you are running multiple instances, it is recommended that all instances share the same group-id in the consumer properties.
Read our blog on Geo-replication with Apache Kafka Mirrormaker 2 (MM2) – Theory and Practice
By Instaclustr Support