Exactly-once delivery in Aiven for Apache Kafka MirrorMaker 2

Exactly-once delivery in Aiven for Apache Kafka MirrorMaker 2 replicates each message exactly once between clusters, preventing duplicates or data loss.

Exactly-once delivery semantics

Exactly-once delivery semantics provide a transactional guarantee for message replication, ensuring that all messages in a batch are either fully committed to the target cluster or not replicated at all. This maintains data consistency across clusters.

note

Exactly-once delivery does not require ACL modifications by default. However, in external Apache Kafka setups where ACLs are applied to the TransactionalId resource, review and adjust these ACLs as needed to enable exactly-once delivery.

Prerequisites

• Aiven for Apache Kafka service with Aiven for Apache Kafka MirrorMaker 2 integration enabled
• Access to create or modify replication flows
• Access to one of the following, depending on your preferred method:
  • Aiven Console
  • Aiven CLI
  • Aiven Terraform Provider
  • Aiven API

Enable or disable exactly-once delivery

Aiven Console

1. In the Aiven Console, open your Aiven for Apache Kafka service with the Aiven for Apache Kafka MirrorMaker 2 integration.
2. Click Integrations.
3. Click the Aiven for Apache Kafka MirrorMaker 2 integration.
4. Click Replication flow in the sidebar.
   • To create a new flow, click Create replication flow .
   • To modify an existing flow, select Edit.
5. Set Exactly-once message delivery enabled to Enabled to turn it on or Disabled to turn it off.
6. Click Create or Save.

Aiven CLI

• Create a new replication flow with exactly-once delivery enabled:

  avn mirrormaker replication-flow create <service_name> \
     --source-cluster <source_cluster> \
     --target-cluster <target_cluster> \
     --replication_flow_config '{ "exactly_once_delivery_enabled": true }'

• Enable exactly-once delivery on an existing replication flow:

  avn mirrormaker replication-flow update <service_name> \
     --source-cluster <source_cluster> \
     --target-cluster <target_cluster> \
     --replication_flow_config '{ "exactly_once_delivery_enabled": true }'

• Disable exactly-once delivery on an existing replication flow:

  avn mirrormaker replication-flow update <service_name> \
     --source-cluster <source_cluster> \
     --target-cluster <target_cluster> \
     --replication_flow_config '{ "exactly_once_delivery_enabled": false }'

Parameters:

• SERVICE_NAME: Name of your Aiven for Apache Kafka MirrorMaker 2 service.
• SOURCE_CLUSTER: Alias of the source Aiven for Apache Kafka cluster.
• TARGET_CLUSTER: Alias of the target Aiven for Apache Kafka cluster.
• EXACTLY_ONCE_DELIVERY_ENABLED: Set to true to enable or false to disable exactly-once delivery.

Aiven API

Use the ServiceKafkaMirrorMakerCreateReplicationFlow API to create or update a replication flow with exactly_once_delivery_enabled in the configuration.

• Create a new replication flow with exactly-once delivery enabled:

  curl -X POST "https://console.aiven.io/v1/project/<project_name>/service/<service_name>/mirrormaker/replication-flows" \
     -H "Authorization: Bearer <API_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{
           "source_cluster": "<source_cluster>",
           "target_cluster": "<target_cluster>",
           "enabled": true,
           "exactly_once_delivery_enabled": true
         }'

• Update an existing replication flow to enable exactly-once delivery:

  curl -X PUT "https://console.aiven.io/v1/project/<project_name>/service/<service_name>/mirrormaker/replication-flows/<source_cluster>/<target_cluster>" \
     -H "Authorization: Bearer <API_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{
           "exactly_once_delivery_enabled": true
         }'

• Update an existing replication flow to disable exactly-once delivery:

  curl -X PUT "https://console.aiven.io/v1/project/<project_name>/service/<service_name>/mirrormaker/replication-flows/<source_cluster>/<target_cluster>" \
     -H "Authorization: Bearer <API_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{
           "exactly_once_delivery_enabled": false
         }'

Parameters:

• PROJECT_NAME: Name of your project.
• SERVICE_NAME: Name of your Aiven for Apache Kafka MirrorMaker 2 service.
• API_TOKEN: Your personal Aiven API token.
• SOURCE_CLUSTER: Alias of the source Aiven for Apache Kafka cluster.
• TARGET_CLUSTER: Alias of the target Aiven for Apache Kafka cluster.
• EXACTLY_ONCE_DELIVERY_ENABLED: Set to true to enable or false to disable exactly-once delivery.

Terraform

Find information about the aiven_mirrormaker_replication_flow resource in the Aiven Provider for Terraform documentation.

To enable exactly-once delivery:

1. Update or create the aiven_mirrormaker_replication_flow resource in your Terraform configuration file:

   • Add exactly_once_delivery_enabled = true to the resource block.
   • Specify the topics to replicate using patterns, such as .* for all topics.

   resource "aiven_mirrormaker_replication_flow" "example_flow" {
     project                       = "PROJECT_NAME"
     service_name                  = "SERVICE_NAME"
     source_cluster                = "SOURCE_CLUSTER"
     target_cluster                = "TARGET_CLUSTER"
     enable                        = true
     exactly_once_delivery_enabled = true
     topics                        = [".*"]
   }
   

2. Run the Terraform commands to apply the configuration:

   terraform init
   terraform plan
   terraform apply

Parameters

• PROJECT_NAME: Name of your Aiven project.
• SERVICE_NAME: Name of your Aiven for Apache Kafka MirrorMaker 2 service.
• SOURCE_CLUSTER: Alias of the source Aiven for Apache Kafka service.
• TARGET_CLUSTER: Alias of the target Aiven for Apache Kafka service.
• EXACTLY_ONCE_DELIVERY_ENABLED: Set to true to enable or false to disable exactly-once delivery.
• TOPICS: Pattern to define which topics to replicate. The default value, .*, replicates all topics.