MirrorMaker 2 common parameters
MirrorMaker 2 (MM2) offers a suite of parameters to help with data replication and monitoring within Apache Kafka® ecosystems. This topic outlines common parameters you can adjust, along with tips for validating MM2's performance.
- Increase the value of
kafka_mirrormaker.tasks_max_per_cpu
in the advanced options. Setting this to match the number of partitions can enhance performance. - Ensure the interval seconds for the following settings match. You
can reduce these intervals for more frequent data synchronization:
- Advanced options:
kafka_mirrormaker.emit_checkpoints_interval_seconds
kafka_mirrormaker.sync_group_offsets_interval_seconds
- Replication flow:
Sync interval in seconds
.
- Advanced options:
- To exclude internal topics, add these patterns to your topic
blacklist:
.*[\-\.]internal
.*\.replica
__.*
connect.*
- Depending on your use case, consider adjusting these parameters:
kafka_mirrormaker.consumer_fetch_min_bytes
kafka_mirrormaker.producer_batch_size
kafka_mirrormaker.producer_buffer_memory
kafka_mirrormaker.producer_linger_ms
kafka_mirrormaker.producer_max_request_size
MirrorMaker 2 validation tips
To ensure MirrorMaker 2 is up-to-date with message processing, monitor these:
-
Consumer lag metric: Monitor the
kafka.consumer_lag
metric. -
Dashboard metrics: If MirrorMaker 2 stops adding records to a topic, the
jmx.kafka.connect.mirror.record_count
metric stops increasing, showing a flat line on the dashboard. -
Retrieve latest messages with `kt`: Use kt to retrieve the latest messages from all partitions with the following command:
kt consume -auth ./mykafka.conf \
-brokers SERVICE-PROJECT.aivencloud.com:PORT \
-topic topicname -offsets all=newest:newest | \
jq -c -s 'sort_by(.partition) | .[] | \
{partition: .partition, value: .value, timestamp: .timestamp}'