Skip to main content

Create diskless topics automatically using regular expressions

Configure your Aiven for Apache Kafka® service to automatically create new topics as diskless topics when their names match configured regular expressions. This lets clients and connectors use diskless topics without setting diskless.enable=true in each create-topic request.

When a client creates a topic, the service compares the topic name against the configured regular expressions. If the name matches any expression, the service creates the topic as diskless.

When to use regular expressions​

Use regular expressions to automatically create new topics as diskless topics based on their names, without requiring every client or workflow to set diskless.enable=true. This is useful when:

  • Clients do not support setting diskless.enable=true in the topic configuration, or you cannot change the client code to include it.
  • Kafka Connect frameworks, especially CDC connectors, create and alter topics dynamically and might not let you add diskless settings to the workflow.
  • Some clients reject unknown topic configurations before the request reaches the broker.
  • MirrorMaker 2 replicates source topics with their existing configurations.

With this configuration, the service creates diskless topics based only on topic names.

Behavior and limitations​

The service applies configured regular expressions only when it creates a new topic. Existing topics do not change.

Topic matching​

For each new topic, the service checks the create-topic request and topic name in this order:

  • If the request sets diskless.enable=true, the service creates the topic as a diskless topic.
  • If the topic name matches any configured regular expression, the service creates the topic as a diskless topic.
  • Otherwise, the service creates the topic as a classic topic.

Excluded topics​

The service does not apply regular expressions to:

  • Internal Kafka topics, such as topics that start with __.
  • Compacted topics. If a topic is created with cleanup.policy=compact or cleanup.policy=compact,delete, the service creates it as a classic topic even if its name matches a regular expression.

Classic topic conflicts​

If a non-compacted topic name matches a configured regular expression, the service creates it as a diskless topic. If you set diskless.enable=false for the same topic, the request fails with a conflict error.

To create a classic topic, use a topic name that does not match any configured regular expression.

In the Aiven Console, the same conflict occurs if the topic name matches a configured regular expression.

Other limitations​

  • Invalid regular expressions are ignored and do not match any topics. Other valid regular expressions continue to be evaluated.
  • The regular expressions have no effect when kafka_diskless.enabled is false.
  • Topic type is set when the topic is created and cannot be changed later.
  • Do not set a replication factor. If you must set it when creating diskless topics, use 1 or -1.

Prerequisites​

To configure default diskless topic regular expressions, you need:

  • An Aiven for Apache Kafka® service with diskless topics enabled.
  • A project role or permission that lets you update service configuration. For details, see Project roles and permissions.
note

For existing services, automatic diskless topic creation by regular expression is available after scheduled maintenance updates complete.

Configure default diskless topic regular expressions​

Set auto_diskless_topic_regexes in the kafka_diskless section of the Kafka service configuration. Add each topic-name regular expression as a separate value.

Before you add regular expressions, review these requirements:

  • You can add up to 32 regular expressions.
  • Each regular expression can contain up to 128 characters.
  • Regular expressions use Java java.util.regex.Pattern syntax.
  • Supported characters are alphanumeric characters and ^ $ - _ . * + ? \ [ ] | { } ( ).
  • Unsupported characters, such as # or whitespace, are rejected.
  • Regex features such as look-ahead assertions are not supported.

Test each regular expression before you apply it. Use specific regular expressions where possible. Avoid broad regular expressions such as .* unless most new non-internal, non-compacted topics must be diskless.

note

The auto_diskless_topic_regexes option is not available in the Aiven Console for services with diskless topics enabled. Set it using the API or CLI.

Use the Aiven API to update the service configuration.

curl --request PUT \
  --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \
  --header "Authorization: Bearer TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "user_config": {
      "kafka_diskless": {
        "enabled": true,
        "auto_diskless_topic_regexes": ["events\\..*", "cdc\\..*"]
      }
    }
  }'

Replace:

  • PROJECT_NAME with the name of your Aiven project.
  • SERVICE_NAME with the name of your Aiven for Apache Kafka service.
  • TOKEN with your Aiven authentication token.
  • ["events\\..*", "cdc\\..*"] with the JSON-escaped regular expressions for topic names to create as diskless topics.

Changing the list of regular expressions might briefly interrupt topic creation requests. Kafka brokers remain online.

If you manage services or topics with infrastructure as code, ensure your topic naming conventions and regular expressions stay aligned. If you use the Aiven Terraform Provider, verify that your provider version supports auto_diskless_topic_regexes before adding this setting to your configuration.

After you save the configuration, Aiven applies the regular expressions to new topic creation requests. Existing topics keep their current type.

Regular expression examples​

GoalRegular expression
Create topics with the events. prefix as diskless topicsevents\..*
Create topics with the cdc. prefix as diskless topicscdc\..*
Create topics ending in .diskless as diskless topics.*\.diskless
Create topics that contain .events. as diskless topics.*\.events\..*

Regular expression matching uses full-match semantics, so leading ^ and trailing $ anchors are optional. Use .* to match variable text before or after a fixed value. For example, events matches only the topic name events. To match events.orders, use events\..*.

Verify the topic type​

After you create a topic that matches one of the regular expressions, verify that it was created as a diskless topic.

  1. In the Aiven Console, click your Aiven for Apache Kafka service.
  2. Click Manage stream > Topics.
  3. Check the Topic type column.

Topics that match a configured regular expression show Diskless.

Remove default diskless topic regular expressions​

To stop creating matching topics as diskless topics by default, set auto_diskless_topic_regexes to an empty array or null.

note

The auto_diskless_topic_regexes option is not available in the Aiven Console for services with diskless topics enabled. Update it using the API or CLI.

curl --request PUT \
  --url https://api.aiven.io/v1/project/PROJECT_NAME/service/SERVICE_NAME \
  --header "Authorization: Bearer TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "user_config": {
      "kafka_diskless": {
        "enabled": true,
        "auto_diskless_topic_regexes": []
      }
    }
  }'

To clear the value, set auto_diskless_topic_regexes to null.

Removing the regular expressions applies only to new topics. Existing topics do not change. The change does not disable diskless topics for the service. It only removes automatic matching.

Related pages