Aiven now offers Kafka MirrorMaker 2.0 as a service

Aiven now supports MirrorMaker 2.0 as a service. We'll outline some of the advantages that MirrorMaker 2.0 has over the original MirrorMaker. Spoiler: The difference is night and day.

24 June 2020
John HamminkDeveloper Advocate at Aiven

At Aiven, we strive to give our customers reliable, convenient, and responsive data tools. Many customers already enjoy our managed Kafka solution along with Kafka Connect as a service. Now, we are happy to announce that we’re providing Kafka MirrorMaker 2.0 as a managed service.

Aiven for MirrorMaker 2.0 is the most robust solution for data replication between Kafka clusters. By deploying it as a stand-alone service, you can free your replication workloads from others — this means that you can better allocate cluster resources and scale your processing workloads.

In this post, we’ll discuss the Kafka MirrorMaker 2.0 key features that’ll be included in our managed service, the kind of metrics you can get from your MirrorMaker clusters with our native metrics integration, and what’s in store for the future, e.g. ability to connect to external Kafka clusters.

Key features of MirrorMaker 2.0

MirrorMaker—a Kafka tool for data replication—has been around for many years and is widely used. The Kafka community has been aware of its downsides and limitations since, as well as the attempts to fix them with other replication tools. With version 2.4, MirrorMaker 2.0 (following KIP-382: MirrorMaker 2.0) was added to Apache Kafka.

Mirrormaker 2.0 comes with many improvements over its predecessor, such as:

  • Preserved offset mappings between clusters. This means tooling for nearly transparent consumer migration is possible — critical for disaster recovery fail-over.
  • Topic configuration is now synchronized from source to target clusters — removing the need for external tools.
  • Record partitions are now preserved during the replication process.
  • The ability to run multiple replication flows for a single MirrorMaker cluster and the prevention of replication cycles — allowing for complex replication topologies.
  • Increased reliability, scalability and performance by leveraging the Kafka Connect framework.

Aiven support for MirrorMaker 2.0

Aiven supports MirrorMaker 2 with the Console UI, Aiven command line client, and Aiven Terraform provider.

Console Screenshot mirroraker replication flow overview

Aiven metrics integration for MirrorMaker 2.0

To get the most out of your Aiven for MirrorMaker 2 cluster, we’re also providing access to a metrics integration with built-in dashboards to monitor a host of metrics related to replication, such as replication lag and replication latency.

Aiven metrics integration with built-in monitoring

The replication flow metrics run by MirrorMaker 2.0 are collected and can be exported via a metric integration like InfluxDB and visualized in Grafana.

Getting started with MirrorMaker 2.0

Getting started is pretty simple and can be covered in two primary steps. First — launch a MirrorMaker cluster from the Aiven Console, Terraform provider, REST API, or command line client. Second, tie it to your source and target clusters through an integration workflow.

You can find information about the finer points of setting up your first replication workflow in this help article. And now to the future. Aiven for MirrorMaker 2 is production-grade, but there is still room for improvements. Specifically, we’ll be adding the following features in the near future:

  • Cross-project integration. Currently, it’s only possible to replicate data between Kafka services that belong to the same Aiven project. However, soon the support for integration between services in different projects will be added.
  • Integration with external Kafka clusters. We will make it easy to replicate data between Aiven-managed and external Kafka clusters.

Wrapping up

Kafka MirrorMaker 2.0 will allow Aiven users to seamlessly copy data between Apache Kafka clusters faster than ever! To get more information, go to our managed MirrorMaker 2.0 page. In the meantime, subscribe to our changelog and blog RSS feeds, or follow us on LinkedIn or Twitter to stay up-to-date with product updates, as well as new services.

MirrorMaker 2.0 CTA

announcementskafka

Start your free 30 day trial

Test the whole platform for 30 days with no ifs, ands, or buts.

Aiven logo

Let‘s connect

Apache Kafka, Apache Kafka Connect, Apache Kafka MirrorMaker 2, Apache Cassandra, Elasticsearch, PostgreSQL, MySQL, Redis, InfluxDB, Grafana are trademarks and property of their respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.