Apache Kafka®

Have you ever been confused by all this talk about kafkas and streaming? Get the basics in this article full of information and resources.

This is a collection of Terraform Cookbook for Apache Kafka

Getting started with Aiven for Apache Kafka®? Check out this list of our most used Kafka connectors!

Learn how to create a ClickHouse sink connector for Apache Kafka® with Aiven platform

Generating artificial datasets that closely resemble real data without compromising sensitive information

Secure Your Apache Kafka Infrastructure: Best Practices for Planning, Monitoring, and Testing to Protect Against Security Threats and Data Integrity Issues.

Make sense of social media data with Apache Kafka® and Apache Flink®

Ever wondered how to enrich your real-time streaming data with machine learning-driven analysis?

A guide to what tiered storage is, and how you can start using it with Aiven for Apache Kafka® and Terraform. We’ll set up a cluster, load the data and observe the metrics.

Learn how to gather individual events into sessions, collections of events from a single user over a given time span, using Apache Flink®

Find out how Apache Kafka® and Apache Flink® allow you to move from batch processing to streaming, but keep using SQL in the data pipeline.

How can you test an empty data pipeline? Read on to discover how to create pretend streaming data using Python and Faker.

Use Karapace, an open source Apache Kafka® schema registry, to prevent data errors by managing the data model across databases

Implement a real-time change data capture workflow from an Azure SQL database using Aiven for Apache Kafka® and Debezium

Build serverless Event Driven Architectures (EDA) by combining Apache Kafka® with AWS Lambda functions. Learn how to trigger Lambda functions based on events flowing in an Apache Kafka topic

Follow along to implement the outbox pattern in PostgreSQL® and create a change data capture workflow that doesn't create duplicate data!

Implement a real-time change data capture workflow from an Amazon Relational Database Service database using Aiven for Apache Kafka®

Implement a change data capture workflow in Apache Kafka®, a key component of any organization with high data integrity requirements.

Use an HTTP sink connector to feed data into Apache Kafka® and deploy it using Terraform. A part of Aiven's Terraform Cookbook.

Deploy an Aiven for Apache Kafka® node and integrate it as a data source with Aiven for ClickHouse® using Terraform. A part of Aiven's Terraform Cookbook.

Use Debezium as a source connector to integrate PostgreSQL® and Apache Kafka® and use Terraform to deploy. A part of Aiven's Terraform Cookbook.

Use MongoDB as a source for Apache Kafka® topics and deploy the integration with both using Terraform. A part of Aiven's Terraform Cookbook.

Replicate data from one Apache Kafka® cluster to another using MirrorMaker 2. Useful when migrating Apache Kafka® services. A part of Aiven's Terraform Cookbook.

Manage Apache Kafka® with the open source Karapace schema registry, and deploy both using Terraform. A part of Aiven's Terraform Cookbook.

Deploy an Apache Kafka® node to Aiven for Apache Kafka® with custom configurations defined in Terraform. A part of Aiven's Terraform Cookbook.

Learn how to use Apache Kafka® as a source and sink to process streaming data, and how to deploy that with Terraform. A part of Aiven's Terraform Cookbook.

Use Terraform to connect Apache Kafka® to OpenSearch®. A part of Aiven's Terraform Cookbook series.

Franz is a desktop client for managing Apache Kafka® deployments. Learn how to connect to an Aiven for Apache Kafka® service using Franz

Learn how to migrate an existing Apache Kafka® cluster to a new service provider using Apache Kafka® MirrorMaker 2 - for instance, when moving from an on-premises system to a managed or self-hosted Kafka cluster.

Configure a connection to an Apache Kafka® cluster and use Apache Avro™ to send messages to it.

Scaling up? Learn how to retroactively manage your data platforms with Terraform

Mastodon → Apache Kafka® → OpenSearch® → knowledge

Meet your compliance goals using Aiven for OpenSearch®, S3 storage and Aiven for Apache Kafka®.

Learn how to produce and consume records with Apache Kafka Java client library

Let your data flow with the Apache Kafka®️ HTTP sink connector

If you want to analyze Mastodon posts, getting them into Apache Kafka® is a sensible first step. Read on to find out how to do this with Typescript and NodeJS.

Terraform can orchestrate your infrastructure the same way every time, giving you confidence in your consistent platform. Idempotency for the win.

When it comes to making a performant Apache Kafka® cluster, partitioning is crucial. Read this article to learn which tools you can use to partition your data.

We all know we shouldn't use naughty words. Learn how to remove them from your streaming data using DataCater.

Learn how to perform analytics with fast response time over huge volumes of data without the need to downsample data.

Find out how to use Apache Kafka® to migrate across database technologies while keeping the target continually in sync with the source.

Learn the basics of Apache Kafka® Producers and Consumers through building an interactive notebook in Python.

Find out how to use Apache Kafka® Connect to update an old app-to-db design to use up-to-date tech tools without disrupting the original solution.