.svg)
.svg)
We are covering how Kafka MirrorMaker operates, how to set it up, and how to test mirror data.
MirrorMaker 2.0 is the new replication feature of Kafka 2.4, defined as part of the Kafka Improvement Process - KIP 382. Kafka MirrorMaker 2 is designed to replicate or mirror topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify the configuration and scaling. MirrorMaker dynamically detects changes to source topics and ensures source and target topic properties are synchronized, including topic data, offsets, and partitions. The topic, together with topic data, offsets, and partitions, is replicated in the target cluster when a new topic is created in the source cluster.
Though Kafka is highly distributed and provides a high level of fault tolerance, disasters can still happen, and data can still become temporarily unavailable—or lost altogether. The best way to mitigate the risks is to have a copy of your data in another Kafka cluster in a different data center. MirrorMaker translates and syncs consumer offsets to the target cluster. That way, we can switch clients to it relatively seamlessly, moving to an alternative deployment on the fly with minor or no service interruptions.
Kafka producer clients often prefer to write locally to achieve low latency, but business requirements demand the data be read by different consumers, often deployed in multiple regions. This can easily make deployments complex due to VPC peering. MirrorMaker can handle all complex replication, making it easier to write and read local mechanisms.
Aggregation is also a factor in data pipelines, which might require the consolidation of data from regional Kafka clusters into a single one. That aggregate cluster then broadcasts that data to other clusters and/or data systems for analysis and visualization.
MirrorMaker uses a set of standard Kafka connectors. Each connector has its own role. The listing of connectors and their functions is provided below.
There are three ways to run MirrorMaker:
We recommend running MirrorMaker as a dedicated MirrorMaker cluster since it does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers. The cluster can be easily converted to a distributed cluster just by adding multiple replicas of the same configuration. A distributed cluster is required to reduce the load on a single node cluster and also to increase MirrorMaker throughput.
Set up a single node source, target Kafka cluster, and a MirrorMaker node to run MirrorMaker 2.
1. Clone repository:
https://gitlab.com/velotio/kafka-mirror-maker.git
2. Run the below command to start the Kafka clusters and the MirrorMaker Docker container:
| docker-compose up -d |
3. Login to the mirror-maker docker container:
| docker exec -it $(docker ps | grep "mirror-maker-node-1" | awk '{print $1}') bash |
4. Start MirrorMaker:
| connect-mirror-maker.sh ./mirror-maker-config.properties |
5. Monitor the logs of the MirrorMaker container—it should be something like this:
6. Create a topic at the source cluster:
| kafka-topics.sh --create --bootstrap-server source-kafka:9092 --topic test-topic --partitions 1 --replication-factor 1 |
7. List topics and validate the topic:
| kafka-topics.sh --list --bootstrap-server source-kafka:9092 |
8. Produce 100 messages on the topic:
| for x in {1..100}; do echo "message $x"; done | kafka-console-producer.sh --broker-list source-kafka:9092 --topic test-topic |
9. Check whether the topic is mirrored in the target cluster.
Note: The mirrored topic will have a source cluster name prefix to be able to identify which source cluster the topic is mirrored from.
| kafka-topics.sh --list --bootstrap-server target-kafka:9092 |
10. Consume 5 messages from the source kafka cluster:
| kafka-console-consumer.sh --bootstrap-server source-kafka:9092 --topic test-topic --max-messages 5 --consumer-property enable.auto.commit=true --consumer-property group.id=test-group —from-beginning |
11. Describe the consumer group at the source and destination to verify that consumer offsets are also mirrored:
| kafka-consumer-groups.sh --bootstrap-server source-kafka:9092 --group test-group --describe |
| kafka-consumer-groups.sh --bootstrap-server target-kafka:9092 --group test-group --describe |
12. Consume five messages from the target Kafka cluster. The messages should start from the committed offset in the source cluster. In this case, the message offset will start at 6.
| kafka-console-consumer.sh --bootstrap-server target-kafka:9092 --topic source-kafka.test-topic --max-messages 5 --consumer-property enable.auto.commit=true --consumer-property group.id=test-group —from-beginning |
We’ve seen how to set up MirrorMaker 2.0 in a dedicated instance. This running mode does not need a running Connect cluster as it leverages a high-level driver that creates a set of Connect workers based on the MirrorMaker properties configuration file.
Did you like the blog? If yes, we're sure you'll also like to work with the people who write them - our best-in-class engineering team.
We're looking for talented developers who are passionate about new emerging technologies. If that's you, get in touch with us.
We are covering how Kafka MirrorMaker operates, how to set it up, and how to test mirror data.
MirrorMaker 2.0 is the new replication feature of Kafka 2.4, defined as part of the Kafka Improvement Process - KIP 382. Kafka MirrorMaker 2 is designed to replicate or mirror topics from one Kafka cluster to another. It uses the Kafka Connect framework to simplify the configuration and scaling. MirrorMaker dynamically detects changes to source topics and ensures source and target topic properties are synchronized, including topic data, offsets, and partitions. The topic, together with topic data, offsets, and partitions, is replicated in the target cluster when a new topic is created in the source cluster.
Though Kafka is highly distributed and provides a high level of fault tolerance, disasters can still happen, and data can still become temporarily unavailable—or lost altogether. The best way to mitigate the risks is to have a copy of your data in another Kafka cluster in a different data center. MirrorMaker translates and syncs consumer offsets to the target cluster. That way, we can switch clients to it relatively seamlessly, moving to an alternative deployment on the fly with minor or no service interruptions.
Kafka producer clients often prefer to write locally to achieve low latency, but business requirements demand the data be read by different consumers, often deployed in multiple regions. This can easily make deployments complex due to VPC peering. MirrorMaker can handle all complex replication, making it easier to write and read local mechanisms.
Aggregation is also a factor in data pipelines, which might require the consolidation of data from regional Kafka clusters into a single one. That aggregate cluster then broadcasts that data to other clusters and/or data systems for analysis and visualization.
MirrorMaker uses a set of standard Kafka connectors. Each connector has its own role. The listing of connectors and their functions is provided below.
There are three ways to run MirrorMaker:
We recommend running MirrorMaker as a dedicated MirrorMaker cluster since it does not require an existing Connect cluster. Instead, a high-level driver manages a collection of Connect workers. The cluster can be easily converted to a distributed cluster just by adding multiple replicas of the same configuration. A distributed cluster is required to reduce the load on a single node cluster and also to increase MirrorMaker throughput.
Set up a single node source, target Kafka cluster, and a MirrorMaker node to run MirrorMaker 2.
1. Clone repository:
https://gitlab.com/velotio/kafka-mirror-maker.git
2. Run the below command to start the Kafka clusters and the MirrorMaker Docker container:
| docker-compose up -d |
3. Login to the mirror-maker docker container:
| docker exec -it $(docker ps | grep "mirror-maker-node-1" | awk '{print $1}') bash |
4. Start MirrorMaker:
| connect-mirror-maker.sh ./mirror-maker-config.properties |
5. Monitor the logs of the MirrorMaker container—it should be something like this:
6. Create a topic at the source cluster:
| kafka-topics.sh --create --bootstrap-server source-kafka:9092 --topic test-topic --partitions 1 --replication-factor 1 |
7. List topics and validate the topic:
| kafka-topics.sh --list --bootstrap-server source-kafka:9092 |
8. Produce 100 messages on the topic:
| for x in {1..100}; do echo "message $x"; done | kafka-console-producer.sh --broker-list source-kafka:9092 --topic test-topic |
9. Check whether the topic is mirrored in the target cluster.
Note: The mirrored topic will have a source cluster name prefix to be able to identify which source cluster the topic is mirrored from.
| kafka-topics.sh --list --bootstrap-server target-kafka:9092 |
10. Consume 5 messages from the source kafka cluster:
| kafka-console-consumer.sh --bootstrap-server source-kafka:9092 --topic test-topic --max-messages 5 --consumer-property enable.auto.commit=true --consumer-property group.id=test-group —from-beginning |
11. Describe the consumer group at the source and destination to verify that consumer offsets are also mirrored:
| kafka-consumer-groups.sh --bootstrap-server source-kafka:9092 --group test-group --describe |
| kafka-consumer-groups.sh --bootstrap-server target-kafka:9092 --group test-group --describe |
12. Consume five messages from the target Kafka cluster. The messages should start from the committed offset in the source cluster. In this case, the message offset will start at 6.
| kafka-console-consumer.sh --bootstrap-server target-kafka:9092 --topic source-kafka.test-topic --max-messages 5 --consumer-property enable.auto.commit=true --consumer-property group.id=test-group —from-beginning |
We’ve seen how to set up MirrorMaker 2.0 in a dedicated instance. This running mode does not need a running Connect cluster as it leverages a high-level driver that creates a set of Connect workers based on the MirrorMaker properties configuration file.
.svg)
.svg)