Connect to Aiven for Apache Kafka® with command line tools
These examples show how to send messages to and receive messages from an Aiven for Apache Kafka® service using command line tools.
Prerequisites
kafka-console-producer.sh
and kafka-console-consumer.sh
are part of
the Apache Kafka® toolbox included with the open source Apache Kafka®
code. They may be found in the
bin
directory of the unpacked archive.
For kafka-avro-console-producer
follow the installation instructions
in its GitHub
repository, or get the
Confluent platform using the Tar archive option at Quick Start for
Confluent
Platform.
The kafka-avro-console-producer
tool is in the bin
directory of the
unpacked Confluent archive.
You do not need to install the Confluent platform to use the
kafka-avro-console-producer
tool with an Aiven for Apache Kafka
service that is using Karapace.
To setup the configuration file needed by all three tools, follow the guide to set up properties to use the Apache Kafka toolbox.
Variables
Variable | Description |
---|---|
HOST | Host name for the connection |
PORT | Port number to use for the Kafka service |
CONFIGURATION_PATH | Path to your configuration file for Apache Kafka toolbox |
SCHEMA_REGISTRY_HOST | Host name for your schema registry |
SCHEMA_REGISTRY_PORT | Port number for your schema registry |
SCHEMA_REGISTRY_USER | User name for your schema registry |
SCHEMA_REGISTRY_PWD | Password for your schema registry |
TARGET_TOPIC | The name of the Kafka topic to be written to/read from |
In the command lines below, values in {
and }
are to be replaced -
so {PORT}
would be replaced by the appropriate port number, for
instance 12345
.
Produce messages
With kafka-console-producer
you can send multiple messages into your
topic.
kafka-console-producer.sh --broker-list {HOST}:{PORT} \
--topic {TARGET_TOPIC} \
--producer.config {CONFIGURATION_PATH}
Once the connection is successfully established you can send messages one after another by typing them in the terminal. For example:
message1
message2
Produce messages with schema
With kafka-avro-console-producer
you can include the schema by
connecting to your schema registry
- The
schema.registry.url
value must be a full URL, typically starting withhttps://
- Aiven's Karapace is an acceptable schema
registry for this purpose. See Use Karapace with Aiven for Apache
Kafka®
for how to enable it for your Aiven for Kafka service. The
SCHEMA_REGISTRY_
values for the command line can be found on the service Overview page, on the Schema registry tab.
kafka-avro-console-producer --broker-list {HOST}:{PORT} \
--producer.config {CONFIGURATION_PATH} \
--topic {TARGET_TOPIC} \
--property value.schema='{"type":"record","name":"Test","fields":[{"name":"id","type":"string"}]}' \
--property schema.registry.url={SCHEMA_REGISTRY_HOST}:{SCHEMA_REGISTRY_PORT} \
--property basic.auth.credentials.source=USER_INFO \
--property basic.auth.user.info={SCHEMA_REGISTRY_USER}:{SCHEMA_REGISTRY_PASSWORD}
After the connection is established you can send messages according to selected schema. For example:
{"id": "1"}
For more information on how to use kafka-avro-console-producer
see the
Confluent developer
documentation.
Consume messages
With kafka-console-consumer
you can read messages from your topic. For
example, run the command below to start reading from the beginning of
the topic.
kafka-console-consumer.sh --bootstrap-server {HOST}:{PORT} \
--topic {TARGET_TOPIC} \
--consumer.config {CONFIGURATION_PATH} \
--from-beginning