Introducing Klaw for Apache Kafka^® governance

Running Apache Kafka® in an enterprise setting is no easy feat; governing Apache Kafka in an enterprise setting is a whole other headache. To help make it easier for everyone, Aiven has acquired Kafkawize, which provides a centralized governance layer on top of Apache Kafka, fully open source and freely available for download and use.

The project is now renamed Klaw. It will remain fully open source and free to use.

The old ways

There are two traditional ways in which enterprises handle tasks related to Apache Kafka configuration: freedom and gatekeeping.

Freedom and confusion
The infrastructure team freely creates new configurations and connections, based on requirements communicated to them by the service users. This typically involves lots of back-and-forth between teams, ambiguity about ownership and who should be kept in the loop. Changes might not be tracked systematically. The result is an organically-grown jungle gym of connections and elements, where it’s hard to find anything and know what it plugs into.

Gatekeeping and bottlenecks
A single gatekeeper controls what elements and connections are created. They negotiate with the stakeholders and carry the requirements to the infrastructure team, and nothing is deployed before the gatekeeper stamps their approval on it. The result is a slow process that depends on a handful of people (or even a single person!) to function, making it hard to update the configuration.

Klaw offers instead a process and a web-based data governance toolkit where teams using the service can submit their requests for new Topics, schemas, access authorizations and connectors. This democratizes access to the Apache Kafka configuration without sacrificing control over the changes.

What does Klaw do?

Klaw is a web based data governance toolkit for managing Apache Kafka Topics, ACLs, and schemas.

Klaw provides a self-service user interface where teams of Apache Kafka service users can request changes to the Apache Kafka configuration without the intervention of administrators. The changes you can request via Klaw are:

• Adding and defining roles for Kafka users
• Creating and promoting Topics from one environment to another
• Creating and updating schemas
• Authorizing users to produce to or consume from topics
• Adding connectors

The benefits of Klaw

When you make request using Klaw, they include all the information needed to implement them, eliminating the communication hassle between teams.

Also, with these well-structured requests, new Topics have a consistent configuration.

With an established workflow, there will always be a second set of eyes on any given request, ensuring that the quality of requests remains high and accountability is maintained.

As an added bonus, Klaw maintains a log of all events related to configuration changes. It’s easy to check later who requested what and when, and when the change went live.

In sum, Klaw can be used to implement a centralized governance and audit layer on top of Apache Kafka.

How does Klaw work?

The idea is that members of teams who use Apache Kafka can submit requests for changes. Their peers, from whichever team owns the relevant resources, can then approve it. In other words, all data is owned by the team that understands it.

When a request is made, the entire team who can approve it receives an alert. Any team member may approve the request.

Teams, roles and users

Your first step is to set up the teams you want and decide on their responsibilities. Two teams are provided by default: infrateam and stagingteam.

Next, you bring in users. The users are assigned to a team where they can request changes and approve requests. The easiest way to create users is often to use LDAP or integrate with an existing SSO provider. Only one user is provided by default, and that is the superadmin user.

To make it easier to manage users and their permissions, you can use roles and assign different permissions to each role. Two roles are provided by default: user and superadmin. You can create more as required.

Environments

Apache Kafka resources are set up into environments inside Klaw. You can manage clusters and environments separately and configure relationships between them to build a deployment pipeline.

For example, you can specify the following environment hierarchy:
Testing -> Staging -> Acceptance -> Production

You can then create a new Topic in your Testing environment, for example, and when the time comes to create it in your staging environment, you can request that the Topic be promoted. This process takes the entire Topic and its configuration in the Testing environment, and copies it over to the Staging environment—saving you the trouble of reiterating the creation process.

Synchronization

One of the coolest features of Klaw is the ability to synchronize Topics and ACLs between clusters. You can pull them from one cluster and then replicate them in another with a single click. This simplifies Klaw initialization, but also allows users to create a standard configuration for the entire Apache Kafka cluster.

Klaw and other applications

Klaw works with any Apache Kafka flavor, either pure self-managed Apache Kafka or a managed service such as Aiven for Apache Kafka.

For user authentication, Klaw works with Active Directory (AD), SSO (OAuth2) and LDAP.

Both Karapace, Aiven’s open source schema registry, and Confluent Schema Registry are supported, and users can submit a request for a schema that lives in them.

Supported protocols include PLAINTEXT, SSL, and SASL.

Klaw has its own Cluster API that plugs into the Apache Kafka AdminClient API for managing Apache Kafka resources.

Klaw and Aiven

Aiven is fully committed to keeping Klaw open source. Like Karapace, it is part of the company’s open source offering and is free for use by anyone. Aiven for Apache Kafka works seamlessly with Klaw, and as such it’s naturally our top recommendation, but not obligatory.

Apache Kafka deserves top-notch tooling, and Klaw provides an important link that is often missing in enterprise Apache Kafka environments.

Getting started with Klaw

To kick off your own Klaw deployment, visit the Klaw project page. Just download, read the docs, and start empowering your Apache Kafka users!

Further reading

• Kafka users and access control

• TypeScript, react-hook-form, zod: blissful forms

• SASL authentication with Aiven Kafka