Offset sync status analysis for Apache Kafka® MirrorMaker 2

Analyze offset synchronization status between source and target clusters with Aiven’s offset sync inspection tool for Aiven for Apache Kafka MirrorMaker 2. This tool monitors sync progress, provides insights into partition health, and flags any needed interventions. Run it locally after configuring the environment to review offset syncing, especially useful during large data migrations.

Prerequisites

Before you download and run the tool, ensure you have the following:

• Basic understanding of Apache Kafka shell tools
• A valid Aiven for Apache Kafka MirrorMaker 2 configuration file with the necessary source and target cluster connectivity fields
• Java 17 runtime environment (OpenJDK 17)
• Aiven CLI access
• Aiven user token
• Aiven for Apache Kafka SSL integration certificates, including:
  • Client truststore (JKS)
  • Client keystore (P12)
  • SSL keystore and truststore password

Download the offset sync inspection tool

Download the offset sync inspection tool to analyze offset syncing in Aiven for Apache Kafka MirrorMaker 2.

Run the tool

The offset sync inspection tool outputs data in CSV format. Use this output to check the sync status and detect any issues with the offset syncing process. Run the tool using the following command:

<extracted tgz>/bin/kafka-mirrormaker-offset-sync-inspector.sh --mm2-config <path-to-config-file>

Parameter:

• --mm2-config <path-to-config-file>: Specifies the path to the Aiven for Apache Kafka MirrorMaker 2 configuration file. This file contains the connection details for the source and target Apache Kafka clusters.

To view all available options, run the help command:

./bin/kafka-mirrormaker-offset-sync-inspector.sh --help

Log messages and actions

Below are common log messages generated by the tool, what they mean, and recommended actions. If any issues arise, create a support ticket or email Aiven Support.

Sync status message	Description	Action required	Inspect	Analyze
Successfully synced, operating normally.	Data and offsets are successfully migrated, allowing consumers to read from synced offsets with minimal redelivery.	No action required.	None.	None.
Intentionally not synced because the source partition is empty. Other partitions in this group are synced.	The source partition has no data, so Apache Kafka MirrorMaker 2 skips syncing it. Consumers start from the beginning when new data arrives.	No action required.	None.	None.
Intentionally not synced because the source partition is empty. Other partitions in this group are not synced.	The source partition has no data, so Apache Kafka MirrorMaker 2 skips syncing it. Consumers start from the beginning when new data arrives.	No action required.	None.	None.
Intentionally not synced because the source group is too old. Other partitions in this group are synced.	The source partition has newer data than the committed offset for the consumer group. Consumers start from the beginning when new data arrives.	No action required.	None.	None.
Erroneously not synced. Source partition has data, and other partitions for this group are synced.	There is an offset syncing issue. Consumers start from the beginning when new data arrives.	Create a support ticket.	Use kafka-get-offsets to compare the earliest and latest offsets in source and target topics. Check the source partition's throughput on the Apache Kafka dashboard. Inspect offset syncs in kafka-console-consumer to locate the topic-partition key. Enable TRACE logging for OffsetSyncStore in Apache Kafka MirrorMaker 2.	The partition has little or no data, causing dropped syncs. Zero throughput prevents syncs from being generated. Verify if MirrorSourceTask isn’t emitting syncs or if MirrorCheckpointTask is expiring them.
Erroneously not synced. Source partition has data, and other partitions for this group are not synced.	There is a systematic failure affecting all partitions in this topic. Consumers start from the beginning when new data arrives.	Create a support ticket.	Run per-partition diagnostics and compare offsets. Use kafka-get-offsets and inspect sync activity logs.	Investigate sync failures across partitions.