Migrate an Apache Kafka® cluster to Aiven with MirrorMaker 2

Migrate an existing Apache Kafka® cluster to Aiven for Apache Kafka® using Aiven for Apache Kafka® MirrorMaker 2. The source can be a self-hosted, on-premises, or third-party managed cluster.

MirrorMaker 2 replicates topic data while producers and consumers keep using the source cluster. After replication catches up, migrate the schemas, ACLs, and connectors you need. Then switch your applications to the target Aiven service.

The procedures use the Aiven Console. You can also manage replication flows using the Aiven CLI, the Aiven API, or Terraform.

Important

MirrorMaker 2 replicates data asynchronously. Plan a cutover period for the final steps. During the cutover, stop writes to the source cluster and synchronize consumer group offsets. Verify the migration before you decommission the source cluster.

Prerequisites

Before you begin, you need the following:

• An Aiven project

• Connection details for the source cluster, including bootstrap servers, security protocol, and credentials or SSL certificates

• A MirrorMaker 2 user with the following permissions:

  • On the source cluster: READ access to the topics to replicate and DESCRIBE access to the consumer groups to migrate
  • On the target cluster: WRITE access to the replicated topics
  • On both clusters: READ, WRITE, CREATE, and DESCRIBE access to the MirrorMaker 2 internal topics

  For more information, see Permissions and internal topics.

• Network access from MirrorMaker 2 to the source cluster and from your applications to the target service, including required firewall rules or private connectivity, with enough bandwidth to sync your data

• Kafka clients compatible with the Apache Kafka version of the target service

• A list of the topics, consumer groups, schemas, ACLs, and Kafka Connect connectors to migrate

• A cutover plan that covers which producers, consumers, and connectors to switch, and how to roll back if needed

Define the source cluster endpoint

Create an external Apache Kafka integration endpoint so that MirrorMaker 2 can connect to the source cluster. Follow Integrate an external Apache Kafka® cluster and use a descriptive name, such as source-cluster.

Important

Aiven does not test connectivity when you create an external Apache Kafka endpoint. Ensure the firewall of the source cluster allows traffic from MirrorMaker 2.

To allow only specific addresses, set up static IP addresses for the MirrorMaker 2 service and allow them on the source cluster firewall. Static IP addresses incur an additional charge.

Note

If you run MirrorMaker 2 standalone, define the source cluster in the mm2.properties file instead.

Create the target services

Create an Aiven for Apache Kafka service as the target cluster and a MirrorMaker 2 service to replicate the data.

1. Create an Aiven for Apache Kafka service as the target cluster, for example target-kafka.

2. Apply the custom broker parameters that your applications depend on, such as message size limits or retention defaults. Aiven manages some broker parameters, so first review the available advanced parameters. To apply a parameter:

   1. In the target service, click Service settings.
   2. In Advanced configuration, click Configure.
   3. Click Add configuration options.
   4. Search for the parameter. For example, search for kafka.message_max_bytes to set the maximum message size.
   5. Set the value and click Save configuration.
   Important

   You cannot reset advanced parameters to their defaults or remove them after you add them. Most changes restart the service nodes one at a time, so apply them before you create the replication flow.

   For more information, see Manage Apache Kafka parameters.

3. Optional: Enable the Apache Kafka REST API to browse topic data in the Aiven Console. If your applications use a schema registry, enable it as well. For more information, see Enable schema registry and REST proxy.

4. Optional: If your clients use SASL, enable SASL authentication.

5. Optional: Enable automatic topic creation if your applications rely on it.

6. Create a MirrorMaker 2 service and integrate it with the target Aiven for Apache Kafka service. Enter a cluster alias that identifies the target cluster, such as kafka-target. You cannot change the alias after you create the integration.

Create a replication flow

Add the source cluster to MirrorMaker 2 and create a replication flow to the target cluster.

1. In the MirrorMaker 2 service, click Integrations.

2. Under External integrations, click Cluster for replication.

3. Click the source endpoint, such as source-cluster, and click Continue.

4. Enter a cluster alias, such as kafka-source, and click Enable.

5. Click Replication flows > Create replication flow.

6. Configure the replication flow:

   • Source cluster: The source cluster alias, such as kafka-source.
   • Target cluster: The target cluster alias, such as kafka-target.
   • Topics: Click All topics, or enter Java regular expressions for the topics to replicate. For example, customer\..* matches topics that start with customer.. For more patterns, see Include or exclude topics in a replication flow.
   • Topic blacklist: The topics to exclude. Click Internal topics, then explicitly exclude _schemas and any custom schema registry storage topic. Migrate schemas using the import procedure instead of replicating registry storage topics.
   • Replication policy class: The naming of replicated topics:
     • org.apache.kafka.connect.mirror.DefaultReplicationPolicy prefixes replicated topics with the source cluster alias. For example, orders becomes kafka-source.orders.
     • org.apache.kafka.connect.mirror.IdentityReplicationPolicy keeps the original topic names.
   • Sync group offsets: Turn on to synchronize consumer group offsets. Required to resume consumers from their replicated positions.
   • Sync interval in seconds: How often MirrorMaker 2 synchronizes offsets.
   • Offset syncs topic location: The source or target cluster, depending on your permissions and configuration.
   • Exactly-once message delivery enabled: Turn on to use exactly-once delivery. Recommended for new replications.
   • Emit heartbeats enabled: Turn on to emit heartbeats to the target cluster.
   • Enable: Starts the replication when you create the flow.

7. Click Create.

8. In the target service, click Manage stream > Topics and confirm that the topics appear with the expected names.

After you enable the flow, MirrorMaker 2 checks the source cluster at regular intervals. It replicates new topics that match your patterns. For more options, see Set up an Apache Kafka® MirrorMaker 2 replication flow.

Note

MirrorMaker 2 copies topic configurations from the source topics to the target topics at regular intervals. By default, it does not copy min.insync.replicas, and it never copies unclean.leader.election.enable. It sets the replication factor of a target topic only when it creates the topic. To preserve target-specific settings, see Topic configurations on the target cluster.

Test with a sample topic

Before you migrate production topics, test the setup with a sample topic. Create a topic on the source cluster, such as pizza-orders, and stream data to it with the Aiven fake data producer. Include the topic in the replication flow and confirm that the data arrives on the target cluster. Stop the producer when you finish testing.

Monitor replication

Replication lag is the delay between the source and target clusters. Monitor it to find out when the target has caught up with the source.

1. Create a service integration that sends the MirrorMaker 2 metrics to a monitoring system. For more information, see Monitor replication execution.

2. Track the following metrics:

   • kafka_mirrormaker_summary.replication_lag: The lag for each replication flow, tagged with replication-flow. For Datadog, see Configure Apache Kafka® MirrorMaker 2 metrics sent to Datadog.
   • jmx.kafka.connect.mirror.record_count: The number of replicated records. A flat line means that no new records arrive. It does not show that the lag is 0.

A replication lag of 0 shows that replication has caught up. It does not confirm that schemas, ACLs, connectors, or consumer group offsets are in place.

Note

MirrorMaker 2 does not offer synchronous replication. Consumers can receive duplicate or out-of-order records, so make sure your applications handle both.

Migrate schemas

If your applications use a schema registry, migrate the schemas to the target service before you switch clients. To preserve schema IDs and versions, see Migrate schemas to Karapace. Confirm that the schema subjects, IDs, and versions are available on the target service.

Migrate ACLs

MirrorMaker 2 does not replicate Apache Kafka ACLs. Extract the ACL definitions from the source cluster and recreate them on the target service, together with the application credentials. Confirm that each application has the permissions it needs for its topics. For more information, see Manage access control lists.

Migrate Kafka Connect connectors

MirrorMaker 2 replicates topic data but does not migrate Apache Kafka Connect connector configurations or runtime state. If your source environment uses Kafka Connect:

1. Identify the connectors to migrate from the source environment.
2. Ensure that your target Aiven for Apache Kafka® Connect service supports the connector plugins. For more information, see Available Apache Kafka® Connect connectors.
3. Prepare the target connector configurations, including compatible plugins, credentials, and connections to external systems. Don't create the connectors yet. Creating a connector starts its tasks, which can duplicate writes while the source connectors are still running.
4. Plan how to migrate offsets or runtime state from the source connectors, so that they don't process records twice.
5. During cutover, stop the source connectors and migrate their offsets or runtime state. Then create the target connectors.

Switch applications to Aiven

Switch your producers and consumers during a planned cutover period.

To let consumer groups resume from their replicated positions, turn on Sync group offsets in the replication flow before the cutover. MirrorMaker 2 translates committed source offsets and writes them to the target. It does this only while the consumer groups have no active consumers on the target.

Offset synchronization depends on the following:

• Sync interval: The default kafka_mirrormaker.sync_group_offsets_interval_seconds is 60 seconds.
• Checkpoints: MirrorMaker 2 emits offset checkpoints by default. If you turned them off with kafka_mirrormaker.emit_checkpoints_enabled, turn them back on.
• Offset syncs topic location: For more information, see Offset sync topic location.

To switch the applications:

1. Confirm that the topics, schemas, ACLs, and application credentials are available on the target service.

2. Stop the producers and source connectors that write to the source cluster.

3. Let the source consumers process the remaining records and commit their offsets, then stop them.

4. Confirm that the replication lag is 0 and the replication flow has no errors.

5. Keep MirrorMaker 2 running and wait for at least one sync interval.

6. Confirm that the migrated consumer groups have committed offsets on the target before you start the target consumers:

   kafka-consumer-groups.sh \
     --bootstrap-server TARGET_BOOTSTRAP_SERVERS \
     --command-config CLIENT_PROPERTIES_FILE \
     --describe \
     --group CONSUMER_GROUP

   Replace the following:

   • TARGET_BOOTSTRAP_SERVERS: The bootstrap servers of the target service.
   • CLIENT_PROPERTIES_FILE: The path to a client properties file with the connection settings of the target service.
   • CONSUMER_GROUP: The consumer group to inspect.

   Compare the topics, partitions, and committed offsets with the expected positions. Offsets on the target can differ from the source because MirrorMaker 2 translates them between clusters. If a group has no committed offsets on the target, do not start it there. First review its synchronization status in Offset sync status analysis.

7. Update your producers and consumers to use the target service, including the following:

   • Bootstrap servers and authentication details
   • Topic names, if the replication policy adds a prefix. To keep the original names, select org.apache.kafka.connect.mirror.IdentityReplicationPolicy before you start replication. Do not change the policy during cutover.
   • Schema registry endpoints, if applicable

   For connection details and examples, see Connect to Aiven for Apache Kafka.

8. Start the producers, consumers, and migrated connectors on the target service.

9. Produce and consume test records to confirm that the applications work and the consumers resume at the intended positions.

Tip

Test the offset behavior with a non-production consumer group before the cutover.

Verify and complete the migration

After you switch the applications, confirm the following:

• Producers and consumers write and read records on the target cluster.
• Consumer groups resume at the expected positions.
• Migrated connectors and other integrations work as expected.
• Applications no longer depend on the source cluster.

Keep the source cluster available until you finish verification and no longer need it for rollback. Then turn off the replication flow and decommission the source cluster.

Related pages

• Disaster recovery and migration
• Set up an Apache Kafka® MirrorMaker 2 replication flow
• Configuration and tuning for MirrorMaker 2
• Monitor replication execution
• Offset sync status analysis