Skip to main content

Configuration and tuning for Aiven for Apache Kafka® MirrorMaker 2

Aiven for Apache Kafka® MirrorMaker 2 settings apply at three layers, which determine the restart impact of a change, how to tune performance, and how topics replicate.

Configuration layers​

Aiven for Apache Kafka® MirrorMaker 2 uses three configuration layers. Each layer controls a different part of the replication process and has a different restart impact.

  • Service configurations
  • Replication-flow configurations
  • Integration configurations

Service configurations​

Service configurations control the behavior of nodes and workers in the MirrorMaker 2 cluster.

For example:

  • Parameter: kafka_mirrormaker.emit_checkpoints_enabled
  • Description: Enables or disables periodic emission of consumer group offset checkpoints to the target cluster.
  • Impact:
    • Restarts workers
    • Restarts all connectors and tasks

Replication-flow configurations​

Replication-flow configurations control the behavior of connectors such as Source, Sink, Checkpoint, and Heartbeat connectors.

For example:

  • Parameter: topics
  • Description: Specifies a list of topics or regular expressions to replicate. For more information, see Topics included in a replication flow.
  • Impact:
    • Restarts the affected connectors
    • Restarts their tasks

Integration configurations​

Integration configurations refine how producers and consumers behave within connectors.

For example:

  • Parameter: consumer_fetch_min_bytes
  • Description: Sets the minimum amount of data the server returns for a fetch request.
  • Impact:
    • Restarts workers
    • Restarts all connectors and tasks
note

Many configuration parameters originate from KIP-382: MirrorMaker 2.0 configuration properties.

Some configuration parameters are commonly adjusted to improve replication throughput, consistency, or topic selection. The configuration layer determines where the parameter is set and what restarts when the value changes.

Task allocation​

Increasing the value of kafka_mirrormaker.tasks_max_per_cpu in the advanced configuration can improve throughput. Set this value close to the number of partitions when you need more parallelism.

Interval settings​

Aligning interval-based settings keeps replication activity consistent.

Topic exclusion​

Adding these patterns to the topic exclusion list prevents internal and system topics from being replicated:

  • .*[\-\.]internal
  • .*\.replica
  • __.*
  • connect.*

Producer and consumer settings​

These integration configuration parameters control how MirrorMaker 2 producers and consumers interact with the source and target Kafka clusters. Set them on the service integration resource. To update these settings, see Update integration configurations.

important
  • If you do not set a parameter, Kafka applies its built-in default.
  • The settings apply to MirrorMaker 2 integrations with both Aiven for Apache Kafka services and external Kafka clusters.
  • Exercise changes incrementally and cautiously, depending on the resources available in your service plan.

The following table lists the integration parameters, their Kafka defaults, and their maximum values.

ParameterDescriptionKafka defaultMaximum value
consumer_fetch_min_bytesMinimum amount of data the broker returns for a fetch request. Higher values reduce fetch frequency.1 byte—
consumer_fetch_max_bytesMaximum amount of data the broker returns for a fetch request.52,428,800 bytes (50 MiB)100 MiB
consumer_fetch_max_wait_msMaximum time the broker waits for enough data to fill a fetch request before it responds.500 ms600,000 ms (10 minutes)
consumer_max_partition_fetch_bytesMaximum amount of data per partition the broker returns in a single fetch response.1,048,576 bytes (1 MiB)100 MiB
consumer_max_poll_recordsMaximum number of records returned in a single poll request.500 records—
consumer_receive_buffer_bytesSize of the TCP receive buffer for the consumer. A value of -1 uses the OS default.65,536 bytes (64 KiB)100 MiB
consumer_request_timeout_msTimeout for consumer requests to the broker.30,000 ms600,000 ms (10 minutes)
producer_batch_sizeMaximum size of a record batch sent to a single partition.16,384 bytes (16 KiB)—
producer_buffer_memoryTotal memory available to the producer for buffering records.33,554,432 bytes (32 MiB)—
producer_linger_msTime the producer waits for additional records before sending a batch.0 ms—
producer_max_request_sizeMaximum size of a single producer request.1,048,576 bytes (1 MiB)—
producer_request_timeout_msTimeout for producer requests to the broker.30,000 ms600,000 ms (10 minutes)
producer_send_buffer_bytesSize of the TCP send buffer for the producer. A value of -1 uses the OS default.131,072 bytes (128 KiB)100 MiB

Topic configurations on the target cluster​

MirrorMaker 2 periodically copies topic configurations from source topics to target topics, unless you set kafka_mirrormaker.sync_topic_configs_enabled to false. It skips the properties in the default exclude list, or in the config_properties_exclude list of the replication flow if you set one. Aiven always excludes unclean.leader.election.enable.

Minimum in-sync replicas​

By default, min.insync.replicas is excluded, so MirrorMaker 2 doesn't copy it from source topics. If you set config_properties_exclude, your list replaces the default list. To keep min.insync.replicas excluded, include it in your list.

When min.insync.replicas is excluded, a target topic uses the first of the following values that applies:

  1. The min.insync.replicas value set on the target topic
  2. The kafka.min_insync_replicas value of the target service, which defaults to 1

To use a different value, set min.insync.replicas on the target topic or change kafka.min_insync_replicas on the target service.

Replication factor​

MirrorMaker 2 sets the replication factor of a target topic only when it creates the topic, using the replication_factor setting of the replication flow. It doesn't copy the replication factor from the source topic or change it.

Related pages