Aug 18, 2016

Getting started with Aiven for Apache Kafka®

Getting started with Aiven for Apache Kafka®

heikki-nousiainen

Heikki Nousiainen

|RSS Feed

Chief Technology Officer at Aiven

Apache Kafka is a high-throughput publish-subscribe message broker, perfect for providing messaging for microservices. Aiven offers fully managed Apache Kafka as a service in Amazon Web Services, Google Compute Engine, DigitalOcean and UpCloud (Microsoft Azure support is coming during 2016 Q3!).

Apache Kafka

Apache Kafka is a popular open-source publish-subscribe message broker. Kafka is distributed by design and offers scalability, high-throughput and fault-tolerance. Kafka excels in streaming data workloads, ingesting and serving hundreds of megabytes per second from thousands of clients.

Apache Kafka deployment example

Apache Kafka was originally developed by LinkedIn and open sourced in 2011.

Kafka is often used as a central piece in analytics, telemetry and log aggregation workloads, where it is used to capture and distribute event data at very high data rates. It can act as a communications hub for microservices for distributing work over a large cluster of nodes.

With Aiven, we use Kafka as a message bus between our cluster nodes as well as delivering telemetry, statistics and logging data. Kafka's guarantees for message delivery and fault-tolerance allows us to simplify and de-couple service components.

What is Aiven for Apache Kafka

Aiven for Apache Kafka is our fully managed Kafka service. We take care of the deployment and operational burden of running your own Kafka service, and make sure your cluster stays available, healthy and always up-to-date. We ensure your data remains safe by encrypting it both in transit and at rest on disk.

We offer multiple different plan types with different cluster sizing and capacity, and charge only based on your actual use on an hourly basis. Custom plans for deployments that are larger or have specific needs can also be requested. Aiven also makes it possible to migrate between the plans with no downtime to address changes in your requirements.

Below, I'll walk you through setting up and running with your first Aiven for Apache Kafka service.

Getting started with Aiven for Apache Kafka

Creating Aiven for Apache Kafka service is easy: just select the correct service type from the drop down menu on the new service creation dialog. You'll have the option of selecting three or five node cluster plans with the storage sizing of your choice. The larger node count allows for larger throughput or larger replica factors for mission critical data. If unsure, pick a three node cluster; you can always change the selected plan at a later time.

All Aiven services are offered over SSL encrypted connections for your protection. With Kafka, you're also required to perform client authentication with service certificates we provide. You can find and download these keys and certificates on the connection parameters section on the service details page: access key and certifications plus CA certificate you can use to verify the Aiven endpoint. Store these locally, we'll be referring back to them in code examples below (ca.crt, client.crt, client.key).


Finally, you can create the topics you'd like to use under the topics tab on the service details page. In Kafka terms, topics are logical channels that your send messages to and read them from. Topics themselves are divided into one or more partitions. Partitions can be used to handle larger read/write rates, but do note that Kafka's ordering guarantees are only valid within one partition.

When creating a topic, you can select number of partitions, number of replicas and how many hours the messages are retained in the Kafka logs before deletion. You also can increase the number of partitions at a later time.

]

That's it! The service is up and running and ready to capture and distribute your messages. Aiven team will take care of the operational burden of your cluster, and ensure it remains available and in use at all times. To utilize the service, we've included code examples in Python and Node.js below. Just make sure to replace the value of bootstrap_servers below with the service URL from the service details page. Also, verify that the SSL settings below point to the actual key and certificate files downloaded earlier.

Accessing Aiven for Apache Kafka in Python

Producing messages - Kafka term for sending them:

from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers="getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705", security_protocol="SSL", ssl_cafile="ca.crt", ssl_certfile="client.crt", ssl_keyfile="client.key", ) for i in range(1, 4): message = "message number {}".format(i) print("Sending: {}".format(message)) producer.send("demo-topic", message.encode("utf-8")) # Wait for all messages to be sent producer.flush()

Consuming or receiving the same:

from kafka import KafkaConsumer consumer = KafkaConsumer( "demo-topic", bootstrap_servers="getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705", client_id="demo-client-1", group_id="demo-group", security_protocol="SSL", ssl_cafile="ca.crt", ssl_certfile="client.crt", ssl_keyfile="client.key", ) for msg in consumer: print("Received: {}".format(msg.value))

Output from the producer above:

$ python kafka-producer.py Sending: message number 1 Sending: message number 2 Sending: message number 3

And the consuming side:

$ python kafka-consumer.py Received: message number 1 Received: message number 2 Received: message number 3

Accessing Aiven for Apache Kafka in Node.js

Here's a Node.js example utilizing node-rdkafka module:

var Kafka = require('node-rdkafka'); var producer = new Kafka.Producer({ 'metadata.broker.list': 'getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705', 'security.protocol': 'ssl', 'ssl.key.location': 'client.key', 'ssl.certificate.location': 'client.crt', 'ssl.ca.location': 'ca.crt', 'dr_cb': true }); producer.connect(); producer.on('ready', function() { var topic = producer.Topic('demo-topic', {'request.required.acks': 1}); producer.produce({ message: new Buffer('Hello world!'), topic: topic, }, function(err) { if (err) { console.log('Failed to send message', err); } else { console.log('Message sent successfully'); } }); });

And the consuming side:

var Kafka = require('node-rdkafka'); var consumer = new Kafka.KafkaConsumer({ 'metadata.broker.list': 'getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705', 'group.id': 'demo-group', 'security.protocol': 'ssl', 'ssl.key.location': 'client.key', 'ssl.certificate.location': 'client.crt', 'ssl.ca.location': 'ca.crt', }); var stream = consumer.getReadStream('demo-topic'); stream.on('data', function(data) { console.log('Got message:', data.message.toString()); });

Next steps

If you're not using Aiven services yet, go ahead and sign up now for your free trial at https://console.aiven.io/signup!


Related blogs

  • What is Apache Kafka®? illustration

    Jun 9, 2022

    Have you ever been confused by all this talk about kafkas and streaming? Get the basics in this post full of information and resources.

  • Look at me, I'm flying!

    Nov 14, 2022

    Aiven contributes back to the Apache Kafka® community with a dedicated full-time team. Find out more about our work!

    Look at me, I'm flying!

    Josep

  • Introduction to event-driven architecture illustration

    Aug 18, 2021

    Goodbye request-response, welcome producers and consumers! Read on to discover event-driven architecture, the best way to build microservice applications.

Subscribe to the Aiven newsletter

All things open source, plus our product updates and news in a monthly newsletter.