Introducing improved Aiven Kafka topic management

We're happy to announce Kafka topic management updates for Aiven Kafka that have been frequently requested from our customers.

These updates will improve the interoperability between Aiven Kafka service instances and the Kafka ecosystem applications and tooling. Let's jump into the details!

Topic management over the Kafka protocol

Aiven Kafka now supports creating, modifying and deleting topics over the Kafka protocol. The initial CreateTopic API call was introduced in Kafka 0.10.1.0, and support has been gradually extending ever since. The APIs have matured with Kafka 1.0 and the Kafka ecosystem tooling is increasingly starting to utilize these APIs.

With the API supported, utilizing Kafka Connect, Kafka Streams, KSQL and other tooling is now much easier than before.

We've also added a new ACL permission level, admin, allowing you to provide specific users permission to create, update, and delete topics through the Kafka protocol.

Automatic topic creation

In addition to the explicit API calls, we're adding an option to enable automatic topic creation. This feature can be enabled with the Aiven Command Line Client:

    avn service update \
        --project project-name \
        example-kafka-service \
        -c kafka.auto_create_topics_enable=true

Admin ACL permission is required for the topic creation to be successful.

Wrapping up

Both can be introduced to existing Kafka services by triggering a maintenance update from within the Aiven console. Additionally, all new Aiven for Apache Kafka deployments will automatically include these modifications.