Deploying Apache Kafka® Streams next to your data

Intro to Kafka Streams

If you’re using Apache Kafka for data streaming, then you’re likely to want to take data from one topic and write an amended version of that data to another topic, for instance: for filtering (give me the items that have been delivered), enrichment (add some more information to messages, from another data source) or anomaly detection (show me potential signs of trouble).

The premier solution in this space is the Apache Kafka Streams library. It’s part of the Kafka project, in continuous development alongside Kafka itself. It uses the Java streams capability (part of Java version 8, which was released in 2014) to give a concise but readable way of describing the transformations to perform on the data:

Loading code...

It handles all the work needed to make a solution robust in real world situations: scalability, fault tolerance, and handling how producers and consumers actually behave. This is the sort of effort that’s easy to forget when writing a consume/amend/produce program from scratch.

When we were looking at applications that would fit well into Aiven Apps, we knew we wanted to include Kafka Streams in our catalog. Unfortunately, when I looked around at the end of 2025, I couldn’t find a complete, documented and working example. So I wrote one.

https://github.com/Aiven-Labs/kafka-streams-example provides working code that

• Uses Java 25, with Gradle and Groovy for configuration and building.
• Handles Apache Avro messages that use the Confluent Wire Format.
• Provides examples for GenericAvroSerde and SpecificAvroSerde (a Serde is something that provides a serializer and a deserializer to convert between values in a program and bytes in a message).

The repository README has detailed instructions on how to build the examples and run them in a variety of ways, including at the command line, using the Dockerfile and using compose files. The code is vendor neutral, but it does work well with Aiven for Apache Kafka® and the Karapace schema registry that comes with every Aiven Kafka service.

What is provided

The repository comes with four examples that work out of the box, but are suitable as the basis for real applications. The intent is to allow the user to concentrate on what they want the streams mechanism to do, not on how to construct and build the actual program.

• Log messages
  The first application, GenericLogApp.java, uses a GenericAvroSerde and the peek() transformation to read each message value and write its content to the logs. It does not write to an output topic.
  It will work on any topic, as it does not require any understanding of the content of each message. Since it doesn’t write to a topic, it can be useful for testing the “read” logic.

• Copy messages
  The second application, GenericCopyApp.java, adds the .to() operation to the end of the stream. This copies each message value, unaltered, to a different topic using the same Serde.
  Again, it will work on any input topic, as it does not require understanding of the content of each message.

• Filter messages
  The third and fourth applications, GenericFilterApp.java and SpecificFilterApp.java, add the filter() and mapValues() transformations to the stream.

  • filter() selects only messages with the state value set to Delivered
  • mapValues() is then used to create a new (output) message with only some of the input message values, and using a different schema for the output (changing some of the value names).

  The effect is to write messages to the output topic summarising only the delivered items.

  Both use a SpecificAvroSerde to write the output messages, using the same (compiled) output schema. As you’ll guess from the names, GenericFilterApp.java uses a generic Serde for input. It looks up the input schema from the input message at runtime. SpecificFilterApp.java uses a specific Serde for input, which is compiled into the application.

The applications don’t require use of an Aiven for Apache Kafka service, but they are set up to work with messages produced by the Aiven for Kafka sample data generator for Logistics, which writes to the default input topic logistics_data_gen. If you pair that with Aiven’s free tier Kafka service, then you can start experimenting with working Kafka Streams examples for no cost - and the developer tier gets you a more performant Kafka at $35/month if you move on to more realistic data.

The code comes with:

• A container file (Dockerfile) for build and deployment.
• Compose files for building and linking to an existing Kafka service - one for generic Kafka and one for use with Aiven Apps.
• Tests.

Deploying with Aiven Apps

I mentioned at the start of the article that we are interested in Kafka Streams for use in Aiven Apps, so let’s look at how to deploy the examples there.

The compose file for Aiven Apps

The compose.aiven.yaml file specifies how to deploy the Kafka Streams application as an Aiven App, using an Aiven for Apache Kafka service.

• The Aiven Apps deployment process sees images: apache/kafka and realise that the kafka-streams-app requires a Kafka service.
• A Karapace schema registry is always started at the same time as the corresponding Aiven for Kafka service. At the moment, App deployment doesn’t automatically set up the environment variables for connecting to Karapace, so you’ll need to specify them as part of the deployment. This also means you will need to start the Kafka service before deploying the application.

Loading code...

1. Create a fork that does what you want

Fork the repository and create a branch to work on. It’s necessary to create a fork of the repository for three reasons:

1. To deploy an Aiven App, you need to connect the GitHub account owning the application repository to your Aiven organization.
2. Unless you want to run the GenericLogApp, you need to edit the Dockerfile to set the APP_NAME.
3. If you’re working on a real use case, and not just one of the demos, you’ll want to edit the source code and change how it handles messages.

Edit the Dockerfile to set which Kafka Streams app you want to run, GenericLogApp, GenericCopyApp, GenericFilterApp or SpecificFilterApp, by changing the line

Loading code...

Make a note of the value specified for APP_NAME - you will need it when deploying the Aiven App, below.

If you want to alter what the Java code does, this is the time to do that as well.

If you're working on a local clone of the repository, remember to push to the upstream, so that Aiven Apps will be able to see your changes.

2. Create an Aiven for Apache Kafka service

Create an Aiven for Apache Kafka service and a topic for the output (filtered) messages.

You can do this using the Aiven console. The documentation explains how to create a free tier service, or a developer tier service - either will work well for experimentation. I’m going to assume that this is called kafka-streams-demo. Once that is running, create a topic called logistics_data_delivered.

3. Start the sample data flow

We’re going to use the Aiven for Kafka sample data generator to send messages to the Kafka service. To get the sample data streaming:

1. Go to the Aiven console and select the kafka-streams-demo service you just created.
2. Click on the Overview page, and in the Start stream section, click Generate sample data.
3. Choose the Logistics data scenario.
4. Click Enable & Continue, click Confirm to accept the schema, and again to accept the topic (logistic_data_gen) and settings. The topic will be created for you.
5. Click Start data stream.

4. Deploy using the console

The following is a summary - check the documentation for the most up-to-date information on Aiven App deployment, and the README for the details for the Kafka Streams examples.

1. In the Aiven Console go to your project and click Applications.
2. Click Deploy app.
3. If you haven’t already done so, use Connect another account to connect your GitHub account to your Aiven organization.
4. Select your Account, your forked Repository, and the appropriate Branch.
5. Click Next.
6. Select the manifest file compose.aiven.yaml and click Scan.
7. On the card for the Kafka service, click the paired arrows icon , and choose the Kafka service you created earlier.
8. On the card for the Kafka Streams service, click the pen icon to edit its configuration.
   • Check that the application Name makes sense for you, and edit it if necessary. The aim is to have a name that you’ll recognise and be able to find later.
   • In the Environment variables section:
     • Set the value for APP_NAME to the same value you set in the Dockerfile above (the appropriate one of GenericLogApp, GenericCopyApp, GenericFilterApp or SpecificFilterApp).
     • Set the INPUT_TOPIC name to logistics_data_gen
     • Set the OUTPUT_TOPIC name to "" (for GenericLogApp), logistics_data_copied (for GenericCopyApp) or logistics_data_delivered (for either FilterApp).
     • Copy the SCHEMA_ values from the Schema registry tab on the Kafka Service Overview page (in the Aiven web console). For the password, use the toggle switch to make it a secret.
     • Do not change the values for FAT_JAR_NAME or JAVA_HOME.
9. To deploy the app services, click Deploy.

The Kafka Streams app will start to build. See its progress in the Build logs tab. It will connect to the Kafka service when it runs, and you can then see its progress in the Runtime logs tab.

Loading code...

6. Check it’s working

Go to the Runtime applications page in the Aiven console, and check the Build logs tab for the application. You should be able to see log messages as streaming data is processed.

Unless you’re running GenericLogApp, which doesn’t send messages to another topic, go to the Topics tab for your Kafka service, click on the output topic name, click on Review messages, change the Format to avro and click on Fetch messages. You should see messages that have been processed.

Finally, the repository README has instructions on how to run a Python application to see the messages in both the input and output topics.

Try it for yourself

I’ve described above how to run the Kafka Streams examples using Aiven Apps, but the repository README has instructions on how to run them in a variety of ways, whether using an Aiven for Kafka service or not.

Some other things to try:

• Run any of the examples locally, using the Dockerfile. You can still use a (free or development tier) Aiven for Kafka service to generate the logistics data.
• Edit the code to work with one of the other Aiven for Kafka sample data streams.
• If you have an existing Kafka data source, edit the code to work with that.
• If you want to dive deeper, have a look at official Apache Kafka Streams documentation, and in particular the Streams DSL documentation, which describes all the operations Kafka Streams can use.