Streaming

Have you ever been confused by all this talk about kafkas and streaming? Get the basics in this article full of information and resources.

This is a collection of Terraform Cookbook for Apache Kafka

Getting started with Aiven for Apache Kafka®? Check out this list of our most used Kafka connectors!

Learn how to create a ClickHouse sink connector for Apache Kafka® with Aiven platform

Generating artificial datasets that closely resemble real data without compromising sensitive information

Adding long term memory to agent communication with OpenSearch®, Amazon Bedrock, RAG and LangChain

Integrating Valkey® pub/sub with Amazon Bedrock for AI agent communication

Make sense of social media data with Apache Kafka® and Apache Flink®

Ever wondered how to enrich your real-time streaming data with machine learning-driven analysis?

Find out how Apache Kafka® and Apache Flink® allow you to move from batch processing to streaming, but keep using SQL in the data pipeline.

How can you test an empty data pipeline? Read on to discover how to create pretend streaming data using Python and Faker.

Build serverless Event Driven Architectures (EDA) by combining Apache Kafka® with AWS Lambda functions. Learn how to trigger Lambda functions based on events flowing in an Apache Kafka topic

Use an HTTP sink connector to feed data into Apache Kafka® and deploy it using Terraform. A part of Aiven's Terraform Cookbook.

Deploy an Aiven for Apache Kafka® node and integrate it as a data source with Aiven for ClickHouse® using Terraform. A part of Aiven's Terraform Cookbook.

Use Debezium as a source connector to integrate PostgreSQL® and Apache Kafka® and use Terraform to deploy. A part of Aiven's Terraform Cookbook.

Use MongoDB as a source for Apache Kafka® topics and deploy the integration with both using Terraform. A part of Aiven's Terraform Cookbook.

Replicate data from one Apache Kafka® cluster to another using MirrorMaker 2. Useful when migrating Apache Kafka® services. A part of Aiven's Terraform Cookbook.

Manage Apache Kafka® with the open source Karapace schema registry, and deploy both using Terraform. A part of Aiven's Terraform Cookbook.

Deploy an Apache Kafka® node to Aiven for Apache Kafka® with custom configurations defined in Terraform. A part of Aiven's Terraform Cookbook.

Learn how to use Apache Kafka® as a source and sink to process streaming data, and how to deploy that with Terraform. A part of Aiven's Terraform Cookbook.

Use Terraform to connect Apache Kafka® to OpenSearch®. A part of Aiven's Terraform Cookbook series.

Franz is a desktop client for managing Apache Kafka® deployments. Learn how to connect to an Aiven for Apache Kafka® service using Franz

Mastodon → Apache Kafka® → OpenSearch® → knowledge

Learn how to produce and consume records with Apache Kafka Java client library

Let your data flow with the Apache Kafka®️ HTTP sink connector

If you want to analyze Mastodon posts, getting them into Apache Kafka® is a sensible first step. Read on to find out how to do this with Typescript and NodeJS.

Terraform can orchestrate your infrastructure the same way every time, giving you confidence in your consistent platform. Idempotency for the win.

When it comes to making a performant Apache Kafka® cluster, partitioning is crucial. Read this article to learn which tools you can use to partition your data.

We all know we shouldn't use naughty words. Learn how to remove them from your streaming data using DataCater.

Learn how to perform analytics with fast response time over huge volumes of data without the need to downsample data.

Find out how to use Apache Kafka® to migrate across database technologies while keeping the target continually in sync with the source.

Learn the basics of Apache Kafka® Producers and Consumers through building an interactive notebook in Python.

Pure SQL pipelines using Apache Flink®? Learn how to set them up with a local Docker-based platform for Flink, including an SQL client!

Find out how to use Apache Kafka® Connect to update an old app-to-db design to use up-to-date tech tools without disrupting the original solution.