Skip to main content

Create Apache Kafka® topics automatically

Apache Kafka® provides the capability to automatically create topics when a message is produced to a topic that does not exist.

By default, Aiven for Apache Kafka disables this feature to prevent accidental topic creation. If a message is produced to a non-existent topic, you'll see the following error message:

KafkaTimeoutError: Failed to update metadata after 60.0 secs.

In such cases, you have two options:

  1. Create topics in advance: Manually create the topics before use. This approach is recommended for production environments as it provides better control over settings such as partition count, replication factor, and retention time.
  2. Enable automatic topic creation: While simpler, this option carries some drawbacks. It risks inadvertently creating new topics, especially due to typos, and may result in topics created with default configuration values defined at the service level.
note

When tiered storage is activated for your Aiven for Apache Kafka service, all new topics will have tiered storage enabled by default. Learn more about tiered storage.

Enable automatic topic creation​

To enable automatic topic creation through the Aiven Console:

  1. Log in to the Aiven Console, select your project, and choose your Aiven for Apache Kafka® service.
  2. Click Service settings from the sidebar.
  3. Scroll to the Advanced configuration section, and click Configure.
  4. In the Advanced configuration dialog, click Add Advanced Configuration.
  5. Find the auto_create_topics_enable parameter and set it to true to enable automatic topic creation.
  6. Click Save configuration.
warning

Even with automatic topic creation enabled, the user account producing a message to a non-existent topic must have admin permissions. Aiven for Apache Kafka validates the access control list (ACL) before creating the topic. To change user permissions, go to the Users tab on your service detail page in the Aiven Console.