Centralizing logs: Apache Kafka® + OpenSearch®

Many log setups start with each service shipping logs straight to a search cluster, which works for a while. Then volume increases, a new team wants the same logs in a different tool, a retention rule changes, and every one of those changes means redeploying log forwarders or touching application config.

In Observability is a data problem, we argued that logs, traces and metrics are just data, and they deserve the same architecture you give any other important data. This post applies that idea to the most common starting point: centralizing logs.

Put Apache Kafka® between whatever produces your logs and whatever searches them, and run your queries in OpenSearch®. Your applications write logs once, and the pipeline decides how they are filtered and where they go.

This pattern is for architects and platform teams that collect logs from many services and serve multiple downstream consumers. It lets you do three things: change where logs go without redeploying applications, absorb indexing spikes without dropping logs, and replay logs after a mapping or parsing change. If your logs go to one place and volume is steady, you can start with OpenSearch on its own and add Kafka when you need it, and this post covers that path too.

Buffer-and-route: Kafka in front of OpenSearch

Buffer-and-route splits log handling into three jobs, each owned by a different layer.

1. Produce. Applications, log forwarders and services write logs to Kafka topics. They know one endpoint and nothing about where logs end up.
2. Buffer. Kafka holds the stream for a retention window you set. If OpenSearch slows down, is upgraded or is resized, producers keep writing and nothing is dropped while the backlog fits in that window.
3. Route. Consumers read the topics and deliver logs to destinations. OpenSearch gets the logs people search. Object storage gets the full archive. Aiven for ClickHouse® gets a copy for long-range analysis.

With the buffer in place, indexing spikes stop being outages, because OpenSearch catches up at its own pace instead of rejecting writes. After a mapping or parsing change, you replay the original logs from Kafka into a new index. Without Kafka, you either reindex from the cluster you're searching on, which adds load and recovers only what was indexed correctly, or accept that logs OpenSearch rejected are lost.

The routing layer is where every pipeline change happens. Want to drop debug logs from search but keep them in the archive? Add a second index for one noisy service? Send security events to a separate cluster with longer retention? Each of those is a change to a connector or a stream processor, and none of them touches application code.

The same routing layer handles multi-tenancy. Add the ExtractTopic transform from Aiven's open source transforms for Kafka Connect to the OpenSearch sink, and each record lands in its tenant’s index based on a tenant field. Per-tenant indexes keep tenants cleanly separated, but every index adds shards, so with many small tenants a shared index with a tenant field and document-level security scales better.

On Aiven, Kafka is a managed service like OpenSearch, so you don’t have to run or patch it yourself. With diskless topics in Aiven for Apache Kafka, you don't size storage either. Unlike classic Kafka topics, topic data goes to cloud object storage instead of broker disks, so there are no disks to provision or rebalance as log volume grows. Logs still reach search within seconds of being written, about 1.5 seconds at P99, fast enough for dashboards and incident investigations.

What Aiven gives you out of the box

On Aiven, each layer of the pattern is a managed service or a built-in integration, so the pipeline is configuration, not custom code. Four pieces do most of the work.

The OpenSearch logs integration sends Aiven service logs straight to OpenSearch. Enable it on any Aiven service and its logs land in daily indexes in your Aiven for OpenSearch service. You set two values: an index prefix (default logs) and an index retention limit in days (default 3, up to 10,000). OpenSearch deletes indexes older than the limit for you.

The Kafka logs integration sends the same service logs to a Kafka topic instead. Point it at a topic on Aiven for Apache Kafka and your platform logs join the same stream as your application logs, so they enter the buffer-and-route pattern too.

The OpenSearch sink connector moves logs from Kafka topics into OpenSearch. Aiven builds and maintains it as an open source project, so every piece of the pipeline, from Kafka to the connector to OpenSearch, is open source. Setup requires only three steps: enable Kafka Connect on your Kafka service (or use a dedicated Connect cluster), choose OpenSearch sink in the Aiven Console, and give it the topics to read and your OpenSearch connection details. For JSON logs, the integration is just configuration, not code:

Loading code...

By default, each topic lands in an index of the same name. For index rotation, the connector also writes to an existing index alias or to date-named indexes.

You can set up the OpenSearch logs integration, the Kafka logs integration and the OpenSearch sink connector through the Aiven Console, CLI, API, Terraform or the Kubernetes operator. Managing them in Terraform means a pipeline change becomes a reviewed pull request, not a redeploy of every service.

OpenSearch Dashboards is where people search the logs. Every Aiven for OpenSearch service comes with OpenSearch Dashboards, so there is a place to explore logs from the first day without running a separate UI. With Piped Processing Language in the Observability logs explorer, a question like "which services are producing the most errors right now?" becomes a short query:

Loading code...

Field names depend on how your logs are structured, so adjust them to match your index mappings.

For long-term retention, add an object storage sink connector. Configure the Amazon S3, Google Cloud Storage or Azure Blob Storage sink to read the same Kafka topics, the same way you set up the OpenSearch sink, and every log also lands in object storage. OpenSearch then keeps the window people search, and the archive keeps everything else, ready to load back when an investigation needs older logs. A later post will cover tiering and retention in depth.

When OpenSearch is enough on its own

If your situation fits the OpenSearch alone column, start there and add the buffer later. If any row fits the Add Kafka column, add it now:

Starting with OpenSearch alone and adding Kafka as your needs change is a sensible approach. If you already run Aiven for Apache Kafka for application data, adding the buffer costs little. Because Aiven can send service logs to either OpenSearch or Kafka with an integration setting, moving to buffer-and-route later is a configuration change for platform logs. For application logs, the change happens in your log shipper, not your applications. Log forwarders such as Fluent Bit already support both outputs, so you add a Kafka output, run both while you verify, then remove the direct OpenSearch output. After that, every future change happens in the pipeline.

Building your log pipeline on Aiven

A log pipeline you can change without touching your apps starts with having them write to a stream. From there, everything else is routing. Kafka provides the buffer and the replay, OpenSearch provides the search, and Aiven's integrations and connectors turn the wiring between them into configuration you manage in Terraform.

Next in the series: Tracing with OpenTelemetry on OpenSearch®. Start from the first post: Observability is a data problem

Try it: Create an Aiven for OpenSearch service, load the sample web logs from the built-in demo datasets in OpenSearch Dashboards, and explore them in the Observability logs explorer. Create an Aiven for OpenSearch service, load the sample web logs from the built-in demo datasets in OpenSearch Dashboards, and explore them in the Observability logs explorer. If you already run other Aiven services, enable the OpenSearch logs integration on one of them to search real logs from that service. Start free on Aiven