Skip to main content

Create Apache Kafka® topics automatically

When you send a message to a topic that does not exist, Apache Kafka® can automatically create that topic.

Automatic topic creation is off by default on both Classic Kafka and Inkless Kafka services. Turn it on when you want topics to be created automatically (for example, in development). In production, leave it off to avoid creating topics by mistake (for example, due to typos or incorrectly configured clients).

If automatic topic creation is disabled and a producer sends to a non-existent topic, you may see errors such as:

UNKNOWN_TOPIC_OR_PARTITION

For example:

WARN [Producer clientId=console-producer] The metadata response from the cluster reported
a recoverable issue with correlation id 18: {example-topic=UNKNOWN_TOPIC_OR_PARTITION}
(org.apache.kafka.clients.NetworkClient)

You can:

  1. Create topics in advance: Create the topics before use. Prefer this in production so you control partition count, replication factor, and retention.
  2. Enable automatic topic creation: This is simpler, but you cannot set partitions, replication, or topic type. The topic uses default configuration values.
note

If tiered storage is enabled on your Classic Kafka service, new topics use tiered storage by default.

On Inkless Kafka services, automatically created topics are classic topics with remote storage. Diskless topics are not auto-created. Create them manually.

Enable automatic topic creation​

Enable automatic topic creation in the Aiven Console:

  1. In the Aiven Console, open your project and your Aiven for Apache Kafka® service.
  2. Click Service settings.
  3. Scroll to Advanced configuration and click Configure.
  4. In the Advanced configuration dialog, click Add Advanced Configuration.
  5. Find auto_create_topics_enable and set it to true.
  6. Click Save configuration.
warning

With automatic topic creation enabled, the user who sends the message must have admin permissions. To change permissions, open the Users tab on the service page in the Aiven Console.

Related pages