Introducing Kafka Quotas in Aiven for Apache Kafka^®

Aiven for Apache Kafka® helps customers leverage the power and scalability of Apache Kafka® for their streaming data needs without worrying about the underlying infrastructure and the management overhead. In a production-grade Kafka environment, multiple teams often share the cluster resources by having dedicated topics where data is produced, consumed, enriched, and transformed.

While shared Kafka clusters come with many benefits to the organization, like greater performance stability and ease of operations for the Kafka environment, they can pose certain governance and standardization challenges for the platform teams running them. When multiple teams use a cluster, you can have the "noisy neighbor problem" where particular applications could overuse it, therefore negatively impacting the performance of other topics. To avoid such situations, data platform teams and engineering managers leverage Quotas in Apache Kafka as a way of controlling the used resources by client applications within a given cluster.

Today we are excited to announce that Kafka quotas are available in Aiven for Apache Kafka giving Aiven customers the ability to define, update, or verify the resources used by every application directly from the Aiven console or Aiven’s API and CLI. In the following paragraphs, we introduce what quotas in Apache Kafka are, their benefits for shared Kafka setups, and the supported quota types you can now leverage in the Aiven platform. Read on for more details!

A Short Explanation of Kafka Quotas

Quotas in Apache Kafka provide a mechanism for managing and limiting the number of broker resources Kafka clients use. By defining Kafka quotas, administrators of Apache Kafka can set defaults and supervise how the resources of a given cluster are being utilized by different producing or consuming applications. When a producing/consuming application is meeting its quota limits, the cluster will slow down the related ingestion/production of events keeping the stability performance in the cluster. Kafka quotas are ideal for shared Kafka clusters where different teams, applications, or use cases consume and produce events to the same Kafka cluster resources. In such scenarios, having control of the resources each application can use in the cluster will ensure that the performance of the cluster remains consistent and there are no potential degradations.

Why are Quotas in Kafka Necessary

For production-grade workloads, there are two main deployment options. Firstly, Kafka administrations can define application groupings, usage scenarios, and compliance requirements and based on those deploy dedicated, single-purpose Kafka clusters to satisfy their application-specific needs. Alternatively, Kafka users can deploy a shared, multi-purpose Kafka cluster for the entire organization and give access to streaming data from this main Kafka cluster to multiple applications and use cases for their internal application teams.

Whatever your deployment choice, it's rare to have an isolated Kafka cluster dedicated to a single application. In most cases, the cluster represents the backend of multiple data flows and use cases. It is therefore likely that different applications will have different usage patterns, with variable data traffic and fluctuating performance impacting not only the application in use but also all the other use clients sharing the same cluster resources.

Quotas in Apache Kafka prevent such situations by enforcing pre-defined usage limits of the cluster resources that can prevent performance degradation or a complete cluster outage. For example, an IoT provider that uses sensor data from traffic lights around the city of London might have heavier traffic volumes in the daytime and lower traffic volumes overnight (since most buses and the London Underground stop at around midnight). Instead of allowing all applications to read/write to the Kafka cluster throughout the day, less time-critical applications could be writing data to the topic at a lower volume during the day to prevent any cluster degradation or outage.

Quotas in Apache Kafka create resource boundaries between use cases and applications and provide a security method ensuring all those use cases have adequate resources to operate efficiently by allowing Kafka administrators to configure the producing/consuming traffic for each client application per Kafka broker in a cluster. According to the allocated resources, Kafka brokers will throttle the client applications’ requests avoiding potential resource saturation and outages in the cluster due to a small number of misbehaving clients that would otherwise be degrading the performance of the entire setup. Kafka quotas are thus an essential component in keeping sanity and governing Kafka operations, especially for shared, production-grade Kafka workloads where multiple teams are accessing, producing, or consuming streaming data from a centralized Kafka instance.

Key Benefits of Quotas in Apache Kafka

Using quotas in Apache Kafka comes with multiple benefits for the organization and enables more use cases for enterprise Kafka users. Some of these benefits include:

1. More accurate operations and governance for your Kafka environment

   The more the usage of Apache Kafka grows in an organization, the more likely operating the tens (or hundreds) of Kafka clusters becomes a bottleneck in further leveraging real-time data. This is where data governance for Apache Kafka using technologies like Klaw or Conduktor, helps with improving Apache Kafka governance across the organization. Additionally, efficiently managing the schema of your data becomes a must-have in your Kafka setup.

   Leveraging quotas in Apache Kafka allows data platform teams and operators of Apache Kafka to better control the users who use the Kafka cluster, what they have access to (in terms of Kafka topics and ACLs), and how much resource consumption they are allowed (quota limits).

2. Greater cost optimization of your Kafka resources

   Due to the varying throughput in the streaming world, specifically in multi-purpose scenarios, resource usage is complex to forecast. Therefore in the vast majority of cases, there's an overprovisioning of resources to give extra room for application teams to successfully deliver their end goals. Since Kafka quotas can be adjusted as and when needed, data platform teams can revise the priority of the workloads when necessary. So data platform teams can set higher limits during peak times and release them to other applications when not needed.

   Quotas in Kafka help further improve the performance stability of the Kafka cluster while at the same time optimizing fixed costs by sharing a single Kafka instance across use cases and by spreading the associated expenditure across teams in an organization. Especially in times when companies are trying to cut down on cloud infrastructure costs and use cost optimization strategies, having the ability to use your Kafka resources efficiently becomes a key not only for data platform teams but for FinOps and technology executives too.

3. Minimization of data and schema duplication

   One of the biggest obstacles to adopting multi-purpose Apache Kafka clusters in tech organizations these days is performance predictability. Stakeholders need more than data streaming from producers to consumers; they frequently have specific thresholds when it comes to the performance of the cluster itself. Quotas in Apache Kafka provide a mechanism to ensure customized resource limits per application that enhances the overall performance predictability of the Kafka cluster.

   Most of the use cases that were handled in dedicated single-purpose Kafka clusters before, can now be equally managed in a shared multi-purpose cluster, providing the additional benefit of data and schema consolidation.

   Configuring quotas in Apache Kafka minimizes time spent by data platform teams in figuring out new data schemas for the different Kafka topics. This significantly reduces time spent on duplicating schemas across clusters while speeding up teams with value creation and accelerating their time-to-market.

Supported quota types in Aiven for Apache Kafka

Aiven for Apache Kafka now comes with different quotas to help you manage resources effectively. These quotas offer greater flexibility in how you manage network bandwidth and CPU usage:

• Consumer throttle (Network bandwidth quota): This quota allows you to limit the amount of data a consumer can retrieve from the Kafka cluster per second. Setting a maximum network throughput prevents any single consumer from using excessive network bandwidth.

• Producer throttle (Network bandwidth quota): Similar to the consumer throttle, this quota limits the amount of data a producer can send to the Kafka cluster per second. It ensures that producers do not overload the system by sending excessive data, thereby maintaining system stability.

• CPU throttle: This quota is about managing CPU usage. You can manage CPU usage by setting a percentage of the total CPU time. Limiting the CPU resources for specific client IDs or users prevents any individual from monopolizing CPU resources, promoting fairness and efficient resource utilization.

For more information on Client IDs and users in Quotas as well as Quotas enforcement, you can check the relevant documentation.

Getting Started

You can easily add and manage quotas for your Aiven for Apache Kafka® service directly in the Aiven Console or by using the Aiven API. For a detailed description of how to add, set or modify quotas in your Aiven for Apache Kafka instance, please follow the steps in our documentation.

Not an Aiven customer yet? You can schedule a demo and explore Aiven for Apache Kafka firsthand or start your free 30-day here!

Further reading

• Kafka users and access control

• What is Apache Kafka®?