Announcing Aiven for Apache Kafka^® cluster configuration backup

An outage or a planned shutdown could reset the state of your Apache Kafka cluster: Topics definition and configuration, User definitions and ACLs, as well as server configurations, need to be done from scratch. With the new cluster configuration backup feature, available in Aiven for Apache Kafka®, you can now restore all the above settings with a click, allowing you to recover from unexpected or planned shutdowns faster and safer.

What are Apache Kafka configuration backups?

Apache Kafka® is extremely resilient and provides flexible ways to distribute, replicate and copy data and configurations across multiple nodes in a cluster. All Aiven for Apache Kafka plans, even our startup ones, come with a 3-node distributed architecture. Data and configurations are automatically distributed across three different brokers for high availability. In the event of an error in one broker, a copy of the data and configurations already exist in another broker.

In the case of an accidental or planned cluster restart, restoring the Apache Kafka cluster requires restoring two elements: data and configuration parameters. While Apache Kafka data backup and restore is still an open topic in the community, having an option to quickly restore metadata means that you no longer need to worry about losing the setup of your cluster.

Losing your Kafka cluster configuration would automatically mean reconfiguring the cluster itself and recreating the topics, configurations, and permissions from scratch. The process is at best time-consuming when configuration details are stored in external scripts. In worse scenarios, it represents a configuration nightmare, with all the stakeholders needing to redo manual work in the platform. In any case, the turnaround time to get your Kafka cluster up and running again is longer than what you would have originally planned. And then, of course, manual work can introduce errors and misconfigurations.

This is where a feature that backs up and allows you to restore your Apache Kafka cluster configuration comes in really handy.

And guess what: this is what we’ve just added to all our Apache Kafka clusters.

How do the configuration backups work in Aiven for Apache Kafka?

Kafka cluster configuration backup in Aiven’s fully managed Apache Kafka service automatically restores the configuration of your Apache Kafka clusters to their previous state after a power off/on cycle or in the unlikely scenario an incident that has caused the cluster to fail. Aiven for Apache Kafka comes with a 99,99% uptime SLA, so the Kafka services are being taken care of by our support and SRE teams around the clock. But for more disastrous scenarios, like the crash of an entire cloud region, or an accidental cluster shutdown, the configuration backup and restore capability added to Aiven for Apache kafka can be helpful to quickly restore the previous status.

In more detail, configuration backups cover the following components from the Kafka ecosystem:

• Apache Kafka topics and related configuration parameters (such as retention time, and number of partitions)
• Apache Kafka users and ACLs
• Everything in the Schema Registry:
  • Schema definitions
  • Schema IDs
  • Configurations (such as compatibility level)
  • Subjects and subject versions
• Apache Kafka Connect configurations

Getting started with cluster configuration backups

Kafka cluster configuration backups are already available for new services and will be automatically enabled after the regular service maintenance cycle for existing ones. For more information on what’s included in the configuration backups, you can check the relevant documentation.

Get started today

After a cluster failure, the Apache Kafka configuration backup feature reduces the time required for bringing your Kafka cluster back online to a fraction. The result is happy Kafka users and a happier Kafka maintenance crew!

Not using Aiven services yet? Sign up now for your free trial today or get in touch with the Aiven team!

In the meantime, make sure you follow our changelog or our LinkedIn and Twitter accounts to stay up-to-date with product and feature-related news.