At Aiven, we strive to give our customers reliable, convenient, and responsive data tools. Many customers already enjoy our managed Kafka solution along with Kafka Connect as a service. Now, we are happy to announce that we’re providing Kafka MirrorMaker 2.0 as a managed service.
Aiven for MirrorMaker 2.0 is the most robust solution for data replication between Kafka clusters. By deploying it as a stand-alone service, you can free your replication workloads from others — this means that you can better allocate cluster resources and scale your processing workloads.
In this post, we’ll discuss the Kafka MirrorMaker 2.0 key features that’ll be included in our managed service, the kind of metrics you can get from your MirrorMaker clusters with our native metrics integration, and what’s in store for the future, e.g. ability to connect to external Kafka clusters.
Key features of MirrorMaker 2.0
MirrorMaker—a Kafka tool for data replication—has been around for many years and is widely used. The Kafka community has been aware of its downsides and limitations since, as well as the attempts to fix them with other replication tools. With version 2.4, MirrorMaker 2.0 (following KIP-382: MirrorMaker 2.0) was added to Apache Kafka.
Mirrormaker 2.0 comes with many improvements over its predecessor, such as:
- Preserved offset mappings between clusters. This means tooling for nearly transparent consumer migration is possible — critical for disaster recovery fail-over.
- Topic configuration is now synchronized from source to target clusters — removing the need for external tools.
- Record partitions are now preserved during the replication process.
- The ability to run multiple replication flows for a single MirrorMaker cluster and the prevention of replication cycles — allowing for complex replication topologies.
- Increased reliability, scalability and performance by leveraging the Kafka Connect framework.
Aiven support for MirrorMaker 2.0
Aiven supports MirrorMaker 2 with the Console UI, Aiven command line client, and Aiven Terraform provider.
Aiven metrics integration for MirrorMaker 2.0
To get the most out of your Aiven for MirrorMaker 2 cluster, we’re also providing access to a metrics integration with built-in dashboards to monitor a host of metrics related to replication, such as replication lag and replication latency.
Getting started with MirrorMaker 2.0
Getting started is pretty simple and can be covered in two primary steps. First — launch a MirrorMaker cluster from the Aiven Console, Terraform provider, REST API, or command line client. Second, tie it to your source and target clusters through an integration workflow.
You can find information about the finer points of setting up your first replication workflow in this help article. And now to the future. Aiven for MirrorMaker 2 is production-grade, but there is still room for improvements. Specifically, we’ll be adding the following features in the near future:
- Cross-project integration. Currently, it’s only possible to replicate data between Kafka services that belong to the same Aiven project. However, soon the support for integration between services in different projects will be added.
- Integration with external Kafka clusters. We will make it easy to replicate data between Aiven-managed and external Kafka clusters.
Kafka MirrorMaker 2.0 will allow Aiven users to seamlessly copy data between Apache Kafka clusters faster than ever! To get more information, go to our managed MirrorMaker 2.0 page. Meanwhile, subscribe to our changelog and blog RSS feeds, or follow us on LinkedIn or Twitter to stay up-to-date with product updates, as well as new services.