Skip to main content

Create Apache Kafka® topics automatically

Apache Kafka® can automatically create a topic if a message is produced to a topic that doesn’t already exist.

By default, Aiven for Apache Kafka enables automatic topic creation to remain compatible with standard Apache Kafka configurations, making it useful for testing and development. In production environments, disable automatic topic creation to avoid accidental topics created by typos, which can lead to a disorganized topic catalog.

If a message is produced to a non-existent topic, you see the following error message:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

In such cases, you have two options:

  1. Create topics in advance: Manually create the topics before use. This approach is recommended for production environments as it provides better control over settings such as partition count, replication factor, and retention time.
  2. Enable automatic topic creation: This option is simpler but has some drawbacks. It can lead to inadvertent topic creation due to typos and may create topics with default configuration values set at the service level.
note

If tiered storage is enabled for your Aiven for Apache Kafka service, all new topics have tiered storage enabled by default.

Enable automatic topic creation​

To enable automatic topic creation through the Aiven Console:

  1. Log in to the Aiven Console, select your project, and choose your Aiven for Apache Kafka® service.
  2. Click Service settings from the sidebar.
  3. Scroll to the Advanced configuration section, and click Configure.
  4. In the Advanced configuration dialog, click Add Advanced Configuration.
  5. Find the auto_create_topics_enable parameter and set it to true to enable automatic topic creation.
  6. Click Save configuration.
warning

Even with automatic topic creation enabled, the user account producing a message to a non-existent topic must have admin permissions. Aiven for Apache Kafka validates the access control list (ACL) before creating the topic. To change user permissions, go to the Users tab on your service detail page in the Aiven Console.