Skip to main content

Create diskless topics automatically using regular expressions

Configure an Aiven for Apache Kafka® service to create new topics as diskless topics by default when their names match configured regular expressions, without changing client applications.

Add one or more regular expressions to the service configuration to match topic names. When a client creates a topic, the service compares the topic name with the configured regular expressions. If the topic name matches any regular expression, the service creates the topic as a diskless topic.

When to use regular expressions​

Use regular expressions to automatically create new topics as diskless topics based on their names, without requiring every client or workflow to set diskless.enable=true. This is useful when:

  • Your clients don't support setting diskless.enable=true in the topic configuration, or you can't change the client code to include it.
  • Kafka Connect frameworks, especially CDC connectors, can create and alter topics dynamically and might not allow you to add diskless settings to the workflow.
  • Some clients reject unknown topic configurations before the request reaches the broker.
  • MirrorMaker 2 replicates source topics with their existing configurations.

Regular expressions let these clients and tools create diskless topics based on the topic name alone.

Behavior and limitations​

The service applies configured regular expressions only when a new topic is created. Existing topics do not change.

Topic matching​

For each new topic, the service checks the create-topic request and topic name in this order:

  • If the request sets diskless.enable=true, the service creates the topic as a diskless topic.
  • If the topic name matches any configured regular expression, the service creates the topic as a diskless topic.
  • Otherwise, the service creates the topic as a classic topic.

Excluded topics​

The service does not apply regular expressions to:

  • Internal Kafka topics, such as topics that start with __.
  • Compacted topics. If a topic is created with cleanup.policy=compact or cleanup.policy=compact,delete, the service creates it as a classic topic even if its name matches a regular expression.

Classic topic conflicts​

If a non-compacted topic name matches a configured regular expression, you cannot create it as a classic topic by setting diskless.enable=false. The request fails with a conflict error.

To create a classic topic, use a topic name that does not match any configured regular expression.

In the Aiven Console, creating a classic topic with a name that matches a configured regular expression also fails.

Other limitations​

  • Invalid regular expressions are ignored and do not match any topics. Other valid regular expressions continue to be evaluated.
  • The regular expressions have no effect when kafka_diskless.enabled is false.
  • Topic type is set when the topic is created and cannot be changed later.
  • Do not set a replication factor. If you must set it when creating diskless topics, use 1 or -1.

Prerequisites​

To configure default diskless topic regular expressions, you need:

  • An Aiven for Apache Kafka® service with diskless topics enabled.
  • A project role or permission that lets you update service configuration. For details, see Project roles and permissions.
note

For existing services, automatic diskless topic creation by regular expression is available after scheduled maintenance updates complete.

Configure default diskless topic regular expressions​

Set auto_diskless_topic_regexes in the kafka_diskless section of the Kafka service configuration. Add each topic-name regular expression as a separate value.

Before you add regular expressions, review these requirements:

  • You can add up to 32 regular expressions.
  • Each regular expression can contain up to 128 characters.
  • Regular expressions use Java java.util.regex.Pattern syntax.
  • Supported characters are alphanumeric characters and ^ $ - _ . * + ? \ [ ] | { } ( ).
  • Unsupported characters, such as # or whitespace, are rejected.
  • Regex features such as look-ahead assertions are not supported.

Test each regular expression before you apply it. Use specific regular expressions where possible. Avoid broad regular expressions such as .* unless most new non-internal, non-compacted topics must be diskless.

note

This option is not exposed in the Aiven Console for services with diskless topics enabled. Set it using the API or CLI.

Use the Aiven API to update the service configuration.

curl --request PUT \
  --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \
  --header "Authorization: Bearer TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "user_config": {
      "kafka_diskless": {
        "enabled": true,
        "auto_diskless_topic_regexes": ["events\\..*", "cdc\\..*"]
      }
    }
  }'

Replace:

  • PROJECT_NAME with the name of your Aiven project.
  • SERVICE_NAME with the name of your Aiven for Apache Kafka service.
  • TOKEN with your Aiven authentication token.
  • ["events\\..*", "cdc\\..*"] with the JSON-escaped regular expressions for topic names to create as diskless topics.

Changing the list of regular expressions might briefly interrupt topic creation requests. Kafka brokers remain online.

If you manage services or topics with infrastructure as code, make sure your topic naming conventions and regular expressions stay aligned. If you use the Aiven Terraform Provider, verify that your provider version supports auto_diskless_topic_regexes before adding this setting to your configuration.

After you save the configuration, Aiven applies the regular expressions to new topic creation requests. Existing topics keep their current type.

Regular expression examples​

GoalRegular expression
Create topics with the events. prefix as diskless topicsevents\..*
Create topics with the cdc. prefix as diskless topicscdc\..*
Create topics ending in .diskless as diskless topics.*\.diskless
Create topics that contain .events. as diskless topics.*\.events\..*

Matching uses full-match semantics, so leading ^ and trailing $ anchors are optional. Use .* to match variable text before or after a fixed value. For example, events matches only the topic name events. To match events.orders, use events\..*.

Verify the topic type​

After you create a topic that matches one of the regular expressions, verify that it was created as a diskless topic.

  1. In the Aiven Console, click your Aiven for Apache Kafka service.
  2. Click Topics.
  3. Check the Topic type column.

Topics that match a configured regular expression show Diskless.

Remove default diskless topic regular expressions​

To stop creating matching topics as diskless topics by default, set auto_diskless_topic_regexes to an empty array or null.

note

This option is not exposed in the Aiven Console for services with diskless topics enabled. Update it using the API or CLI.

curl --request PUT \
  --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \
  --header "Authorization: Bearer TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "user_config": {
      "kafka_diskless": {
        "enabled": true,
        "auto_diskless_topic_regexes": []
      }
    }
  }'

To clear the value instead of passing an empty array, set auto_diskless_topic_regexes to null.

This change affects only new topics. Existing topics are not changed.

This does not turn off diskless topics for the service. It only removes the automatic topic name matching behavior.

Related pages