Skip to main content

Configure Apache Kafka® metrics sent to Datadog

When creating a Datadog service integration, you can customize which metrics are sent to the Datadog endpoint using the Aiven CLI.

Supported metrics

The following metrics are currently supported for each topic and partition in Apache Kafka®:

  • kafka.log.log_size
  • kafka.log.log_start_offset
  • kafka.log.log_end_offset

All metrics are tagged with topic and partition, enabling independent monitoring of each topic and partition.

Variables

Replace the following placeholders in the code samples:

VariableDescription
SERVICE_NAMEAiven for Apache Kafka® service name
INTEGRATION_IDID of the integration between Aiven for Apache Kafka® service and Datadog

To find the INTEGRATION_ID parameter, run:

avn service integration-list SERVICE_NAME

Customize metrics for Datadog

Before customizing metrics, ensure a Datadog endpoint is configured and enabled in your Aiven for Apache Kafka service. For setup instructions, see Send metrics to Datadog. Format any listed parameters as a comma-separated list: ['value0', 'value1', 'value2', ...].

To customize the metrics sent to Datadog, you can use the service integration-update passing the following customized parameters:

  • kafka_custom_metrics: defining the comma-separated list of custom metrics to include (within kafka.log.log_size, kafka.log.log_start_offset and kafka.log.log_end_offset).

For example, to send the kafka.log.log_size and kafka.log.log_end_offset metrics, execute the following code:

avn service integration-update                                                \
-c 'kafka_custom_metrics=["kafka.log.log_size","kafka.log.log_end_offset"]' \
INTEGRATION_ID

After updating settings, view the collected metrics in your Datadog explorer.

Customize consumer metrics for Datadog

Apache Kafka Consumer Integration collects metrics for message offsets. To customize the metrics sent from this Datadog integration to Datadog, you can use the service integration-update passing the following customized parameters:

  • include_topics: Specify a comma-separated list of topics to include.

    note

    By default, all topics are included.

  • exclude_topics: Specify a comma-separated list of topics to exclude.

  • include_consumer_groups: Specify a comma-separated list of consumer groups to include.

  • exclude_consumer_groups: Specify a comma-separated list of consumer groups to exclude.

For example, to include topics topic1 and topic2, and exclude topic3, execute the following command:

avn service integration-update                                                  \
-c 'kafka_custom_metrics=["kafka.log.log_size","kafka.log.log_end_offset"]' \
-c 'include_topics=["topic1","topic2"]' \
INTEGRATION_ID

After updating settings, view the collected metrics in your Datadog explorer.