Skip to main content

Create Apache Kafka® topics automatically

Apache Kafka® can automatically create a topic when a message is sent to a topic that does not exist.

By default, Aiven for Apache Kafka enables automatic topic creation to stay compatible with standard Apache Kafka behavior. This is useful for testing and development. In production environments, it is recommended to disable automatic topic creation to prevent topics from being created accidentally due to typos or misconfigured clients.

If automatic topic creation is disabled and a message is sent to a non-existent topic, you might see the following error:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

In such cases, you have two options:

  1. Create topics in advance: Manually create the topics before use. This is recommended for production environments because it gives you control over topic settings such as partition count, replication factor, and retention time.
  2. Enable automatic topic creation: This option is simpler but has drawbacks. It can create unintended topics due to typos and applies default configuration values set at the service level.
note

If tiered storage is enabled for your Aiven for Apache Kafka service, all new topics have tiered storage enabled by default.

Enable automatic topic creation​

To enable automatic topic creation through the Aiven Console:

  1. Log in to the Aiven Console, select your project, and choose your Aiven for Apache Kafka® service.
  2. Click Service settings.
  3. Scroll to the Advanced configuration section, and click Configure.
  4. In the Advanced configuration dialog, click Add Advanced Configuration.
  5. Find the auto_create_topics_enable parameter and set it to true.
  6. Click Save configuration.
warning

Even if automatic topic creation is enabled, the user sending the message must have admin permissions. Aiven for Apache Kafka® checks the access control list (ACL) before creating the topic.

To update permissions, go to the Users tab in the service detail page in the Aiven Console.

Related pages