Announcing the availability of the high availability feature in Klaw, eliminating any single point of failure and ensuring minimal or no downtime.
Muralidhar Basani
|RSS FeedStaff Software Engineer at Aiven
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Many organizations use Klaw, one of Aiven’s open source projects, to manage and govern Apache Kafka® ACLs and permissions in their environments. It is important to ensure that Klaw is fault tolerant and able to serve requests seamlessly and efficiently.
In a few organizations, Klaw is managing from 10 to 100s of Kafka clusters, and it is clear that the application is quite business critical and we need to understand the severity of downtime. This implies that Klaw should be made available on multiple replicated instances and run in failover mode or active/active.
Before diving into what high availability is, let's understand why it's crucial for Klaw. As Klaw is essential in managing and monitoring Kafka clusters, ensuring its resilience against potential failures is paramount. Downtime can have dire consequences ranging from slight inconveniences to lost revenue and a damaged reputation. High availability for Klaw addresses these critical issues:
In version 2.4.0 and earlier, Klaw could not be deployed in a high availability setup due to architectural limitations in how Klaw managed its internal cache. This meant that Klaw could only be deployed as one instance in production or any environment.
Klaw stores most authorization-related data in the cache to avoid enormous database calls. This effectively reduces latency and gives users immediate response from the application.
To address this, the developer team behind Klaw introduced a feature that discovers all the available instances of Klaw which are deployed in high availability mode and resets the cache on those instances.
In the above example, a Klaw user adds (applies) a new team to instance 1. Klaw stores this information in its cache and triggers a new cache reset request on other instances which updates its local cache.
Klaw relies on its metadata which consists of users, teams, roles, permissions, clusters, environments and topics info. Klaw users can range from 50 to 500 or more users accessing the application requesting for topics, subscriptions, schemas and connectors daily. Handling all the requests with database queries is not efficient and database friendly. Hence, Klaw stores all information in its internal memory and resets it whenever there are changes requested. This allows Klaw to run seamlessly and serve the user’s requests without failure while querying real time data.
You can learn more about the specific implementation patterns for using Klaw in high availability mode with NGINX as the load balancing server on the Klaw blog.
High availability provides the following benefits to you:
You can learn more about Klaw, an Aiven open source project for external management of Apache Kafka ACLs and configurations on the Klaw project website or GitHub organization. Feel free to open a GitHub issue if you have any issues with Klaw.
The Klaw team is one of many open source maintainer teams that Aiven's Open Source Programs Office helps maintain and sponsor. Aiven also works on PGHoard, a PostgreSQL® backup service and Karapace, an Apache Kafka® REST proxy.
If you want to try out Klaw or any of Aiven's other open source projects, we recommend signing up for a free trial of Aiven for Apache Kafka®.
