Skills as Guardrails: Contributing to Apache Kafka® with AI, Without Knowing Every Module

Let me start with something most Kafka contributors think but rarely say out loud: nobody understands all of Kafka.

I'm not a core committer and have only contributed a few times, but those contributions I have made have been in part thanks to using coding assistants.

There are some issues with this approach though, the Apache Kafka project is huge. It's split into many parts: the core, the server, the client libraries, the streams engine, the storage layer, the consensus code, and more. The codebase spans multiple Java versions and some legacy Scala. Even the network protocol has its own code-generation step.

As a result of all this a developer who spends all day in one part may have never opened another. That's normal, and it's not a gap in anyone's skill; it's just how big and how old the project is.

So when I started using an AI coding assistant to work on Kafka, my first worry was simple: if I don't fully understand a part of the codebase myself, how can I judge whether an AI-generated change is safe?

Here's the answer I found, and it's the whole point of this post. What makes AI-assisted work on Kafka safer isn't the AI knowing every module. It's giving the AI a set of clear rules to follow. Write down the rules that keep a change correct and "allowed," hand them to the assistant, and suddenly you can do careful, useful work almost anywhere in the codebase, without lowering your standards and without pretending you know things you don't.

These rules for AI assistants are commonly called guardrails. This post is about what they are, why they matter, and how you can build your own.

One thing up front, so there's no confusion: this is not "learn to use AI and you can change anything in Kafka." You still need the basics: topics and partitions, producers and consumers, offsets and replication, and several kafka concepts. Guardrails might let you skip memorizing every module; but they don't let you skip understanding Kafka. I'll come back to that at the end, because it matters.

Kafka Contribution Is A Challenge For AI Assistants

One of the hardest parts of contributing to big open-source projects like Kafka are rules and customs that aren’t always obvious and AI assistants often skip over this nuance in favour of looking at the code in isolation. For example:

• Some changes need a KIP first. A KIP (Kafka Improvement Proposal) is a design document you write and get the community to discuss and vote on before you change certain things: anything users depend on, like a public method or a config setting.
• Different modules use different Java versions. A feature that compiles fine in one module may not build in another.
• Some code is on its way out. A class might be deprecated or scheduled for removal. Deprecation doesn't always guarantee deletion, but improving a class that's already marked for removal is usually wasted work.
• Reviewers expect proof, not guesses. In Kafka, "I think this class does X" isn't enough. You're expected to open the file and show it.

Now picture an eager AI assistant that doesn't know any of this. Ask it to "fix" something, and it will happily rename a public method or add a config option. The code compiles, the tests might even pass, and the pull request is still dead on arrival, because it needed a KIP nobody filed, or it broke a rule the AI never knew existed.

This is where skills come in. A skill gives the AI a reusable way to approach a task: the steps it should follow, the checks it should perform, and the situations where it should stop rather than guess. In Claude Code, I define these workflows in SKILL.md files and support them with project-wide instructions in CLAUDE.md along with lessons captured from earlier corrections. These lessons include workflows committed to writing, project rules, and knowledge accumulated over time. Together, this context forms the guardrails.

These instructions are not hard enforcement mechanisms, and the AI still needs supervision. But they give it guidance and working habits that a general-purpose model would otherwise lack.

Guardrail #1: The KIP Gate

This is the single most valuable rule I've written, so it goes first.

The rule is blunt: before writing or even suggesting a code change, stop and ask, "Does this need a KIP?" If the answer is yes, do not make the change (unless you are working on a KIP). Instead, say so, explain why, and suggest something that doesn't need a KIP. Check the rules first, write code second.

Written down, it looks something like this (simplified; you'd write your own to fit your project):

Loading code...

The phrase that trips up newcomers: what counts as "public"? In Kafka, it doesn't mean "any method marked public" or "any method with a Javadoc comment." It means specifically the classes and methods that appear on Kafka's published Javadoc website, decided by an explicit allowlist of packages. Packages named internal or internals are generally excluded from that published API, which makes them a useful first signal, even if the class inside is technically public in the Java sense. But package location alone doesn't determine whether a change needs a KIP. The honest check is whether it appears on the published Javadoc site, not whether the code uses the public keyword.

A real example: I once asked the assistant to "just fix" an error message that came back wrong from the server. Sounds like a one-line change. But the meaning of that error is something client applications rely on; change it, and you might silently break someone's production software. With the KIP Gate in place, the assistant refused: "This changes behavior users depend on. It needs a KIP and a mailing-list discussion first. Here's a documentation fix that doesn't, if you want a quick win instead."

That refusal is worth more than a hundred correct edits. A bad change here doesn't just waste my time, it wastes a reviewer's time too. The gate gives the AI the same instinct a good contributor has: wait, is this even mine to change?

Guardrail #2: Which language goes where

The second guardrail is just a map of which modules use which languages and Java versions. The AI can't see that map unless you give it one. Without it, the assistant might confidently suggest a modern Java feature in a module that can't use it, and the build fails. Written down, it looks like this:

Loading code...

Now when the AI reaches for a Java 17 feature in Streams code, it catches itself: this module is on Java 11, use the older way. No failed build, no confused back-and-forth. It doesn't make the AI an expert in Streams; it just prevents a whole category of "worked in my head, failed on the build server" mistakes.

Guardrail #3: Don't quietly break things while "cleaning up"

The third guardrail is about how changes get made, especially when the AI is cleaning up existing code. I use two rules to keep cleanup changes from altering existing behavior.

The first: keep every side effect. Before replacing old code with a "cleaner" version, list everything the original code does and confirm that the new code preserves it. Here's a classic Java trap:

Loading code...

The "cleaner" rewrite looks nicer, and it's wrong: try-with-resources only calls close(), so it quietly skips cleanUp(). I've seen this exact mistake cause the state to leak from one test into the next, producing bugs that are miserable to track down. Compare what the code does, not what it looks like.

The second: check what a class really does by finding who uses it, not by trusting its name. A class called DefaultThing sounds like the main one everyone uses, but the name is only a clue. Search the codebase for where the class is created and called before making claims about its role. Sounding correct while being wrong is exactly how you lose a reviewer's trust.

Guardrails add up over time

The guardrails so far are written down as reusable skills and project instructions. Another source is the corrections I give the AI as we work. Every time it gets something wrong, I turn that correction into a rule.

A few examples:

• Check whether code is already deprecated before proposing work on it.
• When I point to a specific file, read that exact file rather than a similar one nearby. My pushback isn't proof; the file is.
• Flag tricky judgment calls instead of silently deciding. For example, when picking up someone else's unfinished pull request, raise the question of how to credit their work.

Each rule costs one moment of annoyance. After that, the AI is far more likely to handle the same situation correctly. Over time, that starts to feel like training a teammate who never forgets what you've already explained, instead of starting from a blank slate every morning.

You can take this one step further and mine your chat history from previous sessions by providing it as context to your AI assistant and getting it to create it’s own guardrails automatically. Two honest lessons about the harvesting itself: show the AI your existing rules first, or it will cheerfully "rediscover" ones you already wrote, and note which rules came from reviewers rather than from your own judgment. Several of the code rules above came from a reviewer's comments on my pull request, not from my own conclusions, and I made sure to record that instead of quietly claiming the insight as mine.

What the AI can't bring, and you have to

Some tasks the AI can carry almost end to end: sweeping for stale comments, scripting a flaky-test rerun, drafting a design doc's scaffolding. But anything touching a KIP isn't one of them. To review or write a KIP well, you have to actually understand the design, often by pulling the branch and running it locally, not just reading about it. The AI can draft, summarize, and check consistency, but you're the one who has to judge whether the design is sound.

Underneath all of it, you need two fundamentals. First, at least one JVM language (Java is the safest bet; most new code is Java). If you can't follow the code the AI produces, you can't review it, and reviewing it is your job. Second, the Kafka concepts themselves: topics and partitions, producers and consumer groups, offsets and replication, KRaft and several kafka concepts. None of this is something the AI can hand you in the moment. It's the mental model that lets you tell a good AI suggestion from a plausible-but-wrong one. The guardrails assume a human with real understanding on the other end; they reduce how often the AI breaks the project's rules, but they can't supply your judgment about the domain.

So this isn't "learn to use AI and you can change anything in Kafka." It's the opposite: learn the core concepts and required tooling first, and AI becomes a genuine multiplier, stretching your reach across modules you don't know in detail because you understand the fundamentals they share.

Teaching the AI to respect Kafka's contributing process: the KIPs, the deprecated code, reviewers' healthy skepticism etc actually helped me learn it myself. That's the whole point. On a project too big for anyone to fully understand, an assistant willing to stop and say "wait, this needs a KIP, this isn't ours to just change" is worth more than any amount of code it could have written.

You don't need to understand every module. You need to write down the guardrails. Do that, and you can go almost anywhere.

If you want to try this yourself, start with one rule: your version of the KIP Gate. Everything else can grow from there.