KIP-1150 Accepted, and the Road Ahead

During the last Get Kafka-Nated podcast of 2025 I was asked what the most important thing that happened to the Apache Kafka community was. Confluent’s acquisition by IBM was fresh in our minds, but I still mentioned that, for me, the Diskless proposal was the biggest event. That sentence contained some hope, the hope that it would be approved. Well, that day finally has come. On March 2, 2026, the vote to accept Diskless topics into Apache Kafka passed with overwhelming support of 9 binding votes and 5 non-binding ones. On this day, we, the Kafka community, decided to bring Apache Kafka to the cloud era through the front door.

How did we get here?

On April 16, 2025, we published the first version of the KIP-1150: Diskless Topics.
As with many other existing KIPs that exist, this idea came from a business need. In this case, a need from Aiven to have faster cluster upgrades, better recovery, as well as the need to lower the infrastructure costs (reduced load, reduced networking, …). We first tried to see if there was a way to quickly tweak Apache Kafka to natively use object storage in the cloud, but we quickly realized that this was a bigger endeavor that would need a proper rewrite of critical parts of Apache Kafka. Some months later, in October 2024, we created the Inkless repository in order to start experimenting and see if it was possible to add Diskless topics to Apache Kafka.

Typically, Kafka Improvement Proposals are self-contained and describe a feature in its entirety. An example of this is KIP-405: Tiered Storage. This KIP has likely been the most important one in Apache Kafka up until KIP-1150. But that KIP is also well known because how long it took to go from a proposal, to accepted, to implemented. From the first draft to being accepted, it took us two years and three months (from Dec 14, 2018, to February 18, 2021). We took this as a reference and devised a plan on how to split the proposal so we could reach an agreement quickly on the what, while leaving the how open for discussion. This is why KIP-1150 has different sub-proposals (one for the core changes and another one for the Diskless Coordinator). This enabled us to better isolate the discussion of “do we want to have this feature in Apache Kafka?” from the technical details on how to achieve this. We usually referred to KIP-1150 as the “meta KIP” or the “motivational KIP”.
But we didn’t invent this strategy, this pattern was already used in KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum which would end up introducing KRaft as a metadata quorum mechanism in subsequent proposals.

If you are a regular member of the Kafka community, you are surely aware of the numerous forks that have appeared over the years that brought Kafka-compatible software closer to the cloud, making them cloud native: WarpStream, Buffstream, Freight Clusters, Red Panda Cloud topics… Notice a pattern there? Stan, better known as “the Kafka guy”, compiled a nice recollection of the events in a reddit post. Of all these cloud-native streaming solutions, none of them are Open Source. They’re either built on an independent, proprietary fork or on completely different software that’s Kafka API compatible.
We asked the community if it was time Apache Kafka got the cloud native treatment.
We started by asking this shyly, just a general question, “do we want this at all?” with KIP-1150. The more interesting details on how we would do this were specified in other KIPs linked in our main one. Soon after, we saw something I personally had never seen in the Kafka community, not one, not two but three proposals were created in that same space!

• KIP-1176: Tiered Storage for the Active Log Segment
• KIP-1183: Unified Shared Storage
• …and of course KIP-1150

In my eight years of Kafka experience, I’ve seen KIPs becoming dormant and new ones replacing them, I’ve seen KIPs abandoned and picked up by a different author. But what I haven’t seen yet was having three very distinct proposals for a massive feature. It is as if with KIP-1150, we opened the flood gates of cloud nativeness and the community came all at once with different proposals!
But don’t take this as a bad sign for the Kafka community, au contraire! This is a clear indicator that the Apache Kafka community is stronger than ever, regardless of the numerous proprietary innovations out there, our community is still willing to contribute back and help move Kafka forward.
This forced us to narrow down what we meant by Diskless topics, porting some details of the implementation specific KIPs to the main one.
During these ten months, we have talked to many people in the community, listened to their feedback, and incorporated details of the competing KIPs into our own. Along the way, the authors of KIP-1183 (Slack HQ) dropped their KIP and officially supported KIP-1150 and we started to collaborate by splitting the work to be done on dependent future work (for example, enabling compacted topics for Tiered Storage, which is crucial for diskless topics to also support them).
Nowadays, KIP-1150 reflects a proposal to detach compute from storage, while at the same time making disks completely optional and eliminating inter-AZ traffic for replication purposes, as well as, at the produce and consume level.
If you have attended any Kafka conference during this last year, you might have seen us talking about it on the stage or in our booth.

What did we do?

But the work we did was not limited to writing design docs. We also came up with an implementation for it, in the open, right there for everyone to see and play with. This work can be found on github.com/aiven/inkless.
This was a brave decision from Aiven’s leadership. Our tagline is “The open-source data platform for streaming”, so we put our money where our mouth is. However, we didn’t want to split from Apache Kafka by creating yet another new fork. We didn’t want to create the seventh Kafka compatible project that offered cloud native characteristics. For this reason, we started working in the open by treating that fork as a feature branch for upstream, and in a sense, it still is. All the development we do on diskless topics is done in the open for everyone to see, there is no hidden fork, no open-core model, just pure open source. The whole purpose of our Inkless fork is to be deleted once Diskless topics are merged upstream. But in the meantime, this fork allows people to experiment with it, test it, and provide feedback. Our customers value open source highly, to the extent that some only run on open source infrastructure!

If you are still paying attention to my rambling, you might be asking, why implement something only to re-implement it so that it can be accepted upstream? The answer is simple, we wanted to put this idea to the test, and what better way to do it than to put it in front of our customers?
Our Inkless fork became our blueprint environment for the very idea of Diskless Kafka. We have learned a lot from running Inkless with some of our most innovative customers who were eager to help push Kafka to the next level. But we didn’t just test in production, we also used the renowned Antithesis software to validate our implementation, and Kafka’s along the way. We just released a piece about it recently.
Countless hours of operation, performance optimizations, and usability pitfalls taught us a great deal and showed us some of the limitations of our initial ideas.
Inkless was created to implement Diskless Topics, but at the same time, it looped back and improved the design by trying it out through practical application, not just by theoretical analysis.
Inkless is the spearhead, Diskless is the standard.

One might be tempted to think the work is done and we are at the end of the journey, but in reality, this has just started. If I’m allowed to use a hiking analogy, by getting KIP-1150 approved, we mapped the route and reached the base camp; now comes the climb.
What does this climb consist of? KIP-1150 focused on the high level design of the diskless topics solution, while the implementation details and the fine-grained design were delegated to KIP-1163: Diskless Core and KIP-1164: Diskless Coordinator.

Now that KIP-1150 is approved, our full attention is focused on working on these KIPs and getting them over the line. We know this will be a long journey and full of meaningful discussions which I hope you, the reader, will be an active participant in! So please, head over to the DISCUSS threads in the mailing list and share your thoughts!

The elephant in the room

Earlier in the blog post, I mentioned that after seeing different forks and Kafka-compatible projects taking advantage of the cloud native primitives, we decided to propose adding this feature to Apache Kafka. But why did we decide to make this project open source and do a KIP?
In the end, Aiven is a for-profit company, and as with any for-profit company, its objective is to make money. So why does Aiven care about contributing this to upstream?
Any sensible company dedicates efforts to keep their infrastructure relevant, we don’t see why companies relying on open source projects should behave differently.

Aiven offers managed open source services to customers. The value we sell, in the end, is reliability and problem solving, and we care about these open source projects as if they were ours.
We are firm supporters of Apache Kafka and we know it’s one of the best technologies out there. For this reason, we dedicate a significant amount of our time to keep the project and its ecosystem up to date. These are not just words, we currently maintain one of the biggest sets of open source connectors and we have the de-facto standard plugin for Tiered Storage that targets Object Storage backends.
Strategically speaking, it doesn’t make sense to split the Apache Kafka community even further by adding yet another Kafka compatible product, this would contribute to weakening the position of Apache Kafka as the reference standard in the Streaming ecosystem, and we would risk leaving Apache Kafka as the reference API with a myriad of different proprietary solutions out there.

Aiven’s success is directly tied to the success of the open source projects we rely on. The better Apache Kafka is adapted for the future, the better Aiven is able to provide added value to its customers. It is our incentive to drive Apache Kafka’s usage up, and we do this by making it faster, more robust, accessible, connectable, and cloud-native-ready.

We didn't bring Kafka to the cloud era through the front door just to keep the house to ourselves. We did it so everyone else could move in, too.