Announcing Inkless clusters: Cloud Kafka done right

TL;DR

Inkless clusters are now available on Aiven.

Inkless is our open-source Apache Kafka fork which introduces diskless topics and brings major improvements over classic Kafka topics.

With this upgrade, users can expect to improve their Kafka, low-latency workloads with:

• Scale and rebalance up to 10x faster vs. Apache Kafka with Tiered Storage
• Slash node and disk failure recovery time by up to 90%
• Fully auto-balancing and self-healing built on Apache Kafka 4.0
• Use unlimited storage without preprovisioning, eliminating disk-related bottlenecks

Users also have access to diskless topics - our innovative, replication-less, cloud-native architecture that offers even greater benefits.

Today’s “Kafka-compatible” market forces you to split truth across multiple clusters. Fast here, cheap there - adding copies, latency, and migrations. Inkless removes that fork with its composable design that runs classic and diskless topics inside the same cluster, powered by Open Source innovations like KRaft, diskless topics (KIP-1150), and Tiered Storage.
And since Inkless is Apache Kafka (as it's a fork of the Apache Kafka project that's 100% compatible with the Kafka API, which we plan to sunset once KIP-1150 is merged) it works with Aiven’s Open Source managed ecosystem of connectors, Mirror Maker, schema registry and the other Kafka tools you love.

Inkless has been maturing in our BYOC offering and today we are releasing it on Aiven Cloud.
Diskless topics come ready out of the box - so you have KIP-1150’s benefits without waiting for 1+ years for the KIP to be merged upstream.

But Inkless clusters are about much more than just diskless topics. For the first release of Inkless clusters, we intentionally directed our focus to improve the baseline Apache Kafka experience of classic topics.

In other words, we are exclusively talking about how we improved low-latency, classic topics.

1. Real performance benefits

The new Kafka cluster type is designed to deliver dramatically better baseline performance than standard Apache Kafka with Tiered Storage. It scales and rebalances up to 12× faster, cuts node or disk failure recovery by up to 90%, and keeps the low-latency Kafka experience you expect, so your existing clients keep working without changes.

Just as importantly, Inkless improves the cluster elasticity: compute and storage scale independently, with Diskless and Tiered Storage built in. Under the hood, Inkless is the result of years of work and more than 100k lines of changes on top of classic Kafka to deliver auto-balancing, self-healing operations and the performance gains above.

We have extensively benchmarked Inkless on how it eliminates the operational overhead vs. Apache Kafka, let’s see it in action.

10x faster scaling

Adding brokers and rebalancing happens in seconds, or minutes at high scale. You can scale just-in-time instead of overprovisioning days in advance for traffic spikes.

For this blog post, we benchmarked a 144-partition topic at a continuous compressed 128 MB/s data IN/OUT with 1TB of data per broker.

In this test, we requested a cluster scale-up of 3 brokers (6 to 9) on both the new Inkless, and the old Apache Kafka cluster types in parallel.
In classic Kafka this took 90 minutes.

In Inkless, the same low-latency workload caught up in less than ten minutes (10x faster).

>90% faster failure recovery

Brokers recover significantly faster from failure, without consuming higher cluster resources. This means that remaining capacity stays available for traffic.

In our scenario, we killed one of the nine nodes. This gave us a spike in under replicated partitions (URP) with messages to be caught up, as expected.

This known problem used to take us about 100 minutes to recover from.

In contrast, Inkless now recovers in just 9 minutes (~11x faster).

Up to 3x higher throughput with diskless workloads

KIP-1150’s diskless topics allows the broker’s compute to be more efficiently used to accept and serve traffic, as it no longer needs to be used for replication. We have seen at least a 70% increase in throughput for the same machines. This efficiency upgrade increases your cluster’s capacity for free - up to 3x throughput in best cases. In parallel, we are cooking part 2 of our high-scale benchmarks with more demanding mixed workloads and new machine types.

2. Real Operational Benefits

Mixed workloads, in one cluster

Inkless is the only cloud Kafka offering that gives you the ability to tune the balance of latency versus cost for each individual topic inside the same cluster.

The ability to run everything behind a single pane of glass is very valuable - it reduces the operational surface area, simplifies everything behind a single networking topology, and lets you configure your cluster in a unified way (one set of ACLs). Perhaps most critically, you no longer need migrations.

In other words, Inkless lets you go from managing Kafkas (and all the complexity that comes with that) to managing a Kafka.

Our customers find great value in flexibility, so we built Inkless to be composable. Here is what our design will enable in future iterations:

• Replicated, 3-AZ for low latency and enterprise-grade reliability ≈99.99%.
• Replicated, single-AZ (3-node): ≈99.9% SLA - a pragmatic default when a rare zonal blip is acceptable.
• Diskless Standard with ≈99.99% SLA and maximum savings when seconds of E2E latency are fine (≈1.5–2s).
• Diskless Express: object-store durability with sub-second E2E latency and ≈99.99% SLA.
• Global Diskless: built-in multi-region diskless replication, 99.999% SLA.
• Lakehouse via tiered storage - open-table analytics on the very same streams, with zero-copy or dual-copy depending on economics/latency.

With all topic types switchable on the fly.

Infinite, effortless storage

Inkless gives you storage that scales automatically without pre-provisioning. You can now size your clusters solely by throughput and retention. That’s it. You no longer have to think about what disk capacity to size your cluster by, nor deal with out of disk alerts.

3. Real Cost Benefits

Inkless is priced lower than our traditional Aiven for Apache Kafka clusters. Here is a representative comparison of how much a workload will cost on Inkless vs Aiven for Apache Kafka today.

Out Now

Inkless SaaS clusters are available on AWS, with GCP launching shortly. Inkless BYOC is available for AWS & GCP today.

To celebrate the launch, we are temporarily discounting networking fees to zero to all users on AWS.