Skip to main content

Apache Kafka® concepts

A comprehensive glossary of essential Apache Kafka® terms and their meaning.

Broker​

A server that operates Apache Kafka, responsible for message storage, processing, and delivery. Typically part of a cluster for enhanced scalability and reliability, each broker functions independently but is integral to Kafka's overall operations, separate from tools like Apache Kafka Connect.

Consumer​

An application that reads data from Apache Kafka, often processing or acting upon it. Various tools used with Apache Kafka ultimately function as either a producer or a consumer when communicating with Apache Kafka.

Consumer groups​

Groups of consumers in Apache Kafka are used to scale beyond a single application instance. Multiple instances of an application coordinate to handle messages, with each group allocated to different partitions for even workload distribution.

Event-driven architecture​

Application architecture centered around responding to and processing events.

Event​

A single discrete data unit in Apache Kafka, consisting of a value (the message body) and often a key (for quick identification) and headers (metadata about the message).

KRaft​

Kafka Raft (KRaft) is the metadata and consensus management system in Apache Kafka, replacing ZooKeeper. Aiven for Apache Kafka uses KRaft starting with Apache Kafka 3.9 to manage metadata, controllers, and consensus, removing the need for a separate ZooKeeper cluster.

Kafka node​

See Broker

Kafka server​

See Broker

Message​

See Event

Partitioning​

A method used by Apache Kafka to distribute a topic across multiple servers. Each server acts as the leader for a partition, ensuring data sharding and message order within each partition.

Producer​

An application that writes data into Apache Kafka without concern for the data consumers. The data can range from well-structured to simple text, often accompanied by metadata.

Pub/sub​

A publish-subscribe messaging architecture where messages are broadcast by publishers and received by any listening subscribers, unlike point-to-point systems.

Queueing​

A messaging system where messages are sent and received in the order they are produced. Apache Kafka maintains a watermark for each consumer to track the most recent message read.

Record​

See Event

Replication​

Apache Kafka's feature for data replication across multiple servers, ensuring data preservation even if a server fails. This is configurable per topic.

Topic​

Logical channels in Apache Kafka through which messages are organized. Topics are named in a human-readable manner, like sensor-readings or kubernetes-logs.

ZooKeeper​

A distributed system that managed metadata, controller election, and configuration storage in Apache Kafka. Starting in Apache Kafka version 3.9, Aiven for Apache Kafka uses KRaft instead.