Aiven supports Apache Kafka^® 2.6

We’re excited to announce that Apache Kafka 2.6 is fully supported as a managed service on the Aiven platform. In typical Aiven fashion, we were able to make it available to our user base on August 13th — around one week from its release. Apologies for the latency of the blog post!

Apache Kafka 2.6 comes with a number of enhancements across the board, from its core to Kafka Connect and Streams; but, they all boil down to the same essence: making it easier to operate Kafka and its components at scale. Let’s dive into the more interesting highlights.

Apache Kafka 2.6 highlights

Kafka Core KIPs

KIP-568 — explicit rebalance triggering on the consumer

Generally consumer group rebalances are something that are typically abstracted away from the user and automatically handled as part of the group’s core functionality; but, there are cases when you may want your consumer app to be able to trigger a rebalance. Interestingly, this functionality is key to KIP-441 (discussed below) that makes scaling Streams apps friendlier.

Kafka Connect KIPs

KIP-610 — Error reporting in Sink connectors

Previously, Kafka Connect’s Dead Letter Queue (DLQ) functionality only supported error reporting within contexts of the transform operation, and key, value, and header converter operation. This KIP extends that functionality across the entire chain, thereby allowing sink connectors to report individual records as problematic and sending them to the DLQ.

KIP-585 — filter and conditional SMTs

In a nutshell, this KIP enhances the sophistication of Kafka Connect’s Single Message Transformation (SMT) functionality. Should the use case call for it, users can now selectively apply SMT to specific record types using predicates. Here are three default predicates:

• TopicNameMatches — used to apply an SMT based on topic names
• HasHeaderKey — used to apply an SMT when a record has a specific header
• RecordIsTombstone — used to apply an SMT based on whether a record is a tombstone

KIP-158 — kafka connect should allow source connectors to set topic-specific setting for new topics

This KIP allows users to configure Kafka Connect clusters so that they can handle topic creation for source connectors, instead of relying on users creating them manually, which presents scaling issues, or on brokers to create them, which presents a number of potential errors.

Kafka Streams key KIPs

KIP-447 — producer scalability with exactly once semantics (EOS)

Exactly once semantics (EOS) provides transactional message processing guarantees, but sometimes there is a semantic mismatch between consumers in a group and transactional producers primarily due to how each component handles partition assignment. Architecturally, the way in which the producers handled this mismatch didn’t scale well. This KIP rectifies this mismatch and reworks the design to handle a larger number of input partitions more gracefully.

KIP-441 — Smooth scale-out

This is a biggie when it comes to scaling out your Steams Application because it now allows the prior owner of a stateful task to maintain ownership until the new owner catches up. Ownership will then be transferred once complete, allowing for processing during the scale-out process.

Wrapping up

The aforementioned are only the tip of the iceberg. For this release, there were dozens enhancements, whether new features, improvements, or bug fixes. You can check out the Apache Kafka 2.6 release notes to dive even deeper into what this release offers.

For those of you who want to try managed Kafka 2.6, you can either launch a brand new cluster or conduct a no-downtime upgrade to the latest version. In the meantime, make sure you follow our changelog and blog RSS feeds or our Twitter account to stay up-to-date.