Aiven Blog

Apr 21, 2019

Aiven open sources Google Cloud Storage Kafka connector

Aiven's latest open source connector allows you stream Kafka data to GCS, read this post to learn more about it.

kyle-buzzell

Kyle Buzzell

|RSS Feed

Head of Growth Marketing at Aiven

We’ve recently created and released the Aiven GCS Kafka Sink Connector. This connector will allow you to move data from your Kafka clusters to Google Cloud Storage. Even better, it is open source.

By making it open source, we ensure that you can use our connector regardless of whether you host and manage Kafka yourself or choose a service provider to do so.

You can learn how to configure and use it here. If you would like to make contributions to the project and request features, visit our project here. But before you go, let’s look at Kafka connectors in a little more detail.

Why Kafka connectors are important

Simply put, Kafka needs connectors to live up to its full potential. These connectors allow you to couple Kafka with other systems so that you can easily do things such as streaming Change Data Capture.

They come in two forms, source and sink. The former connects a source of data to Kafka while the latter does the opposite. For example, the GCS sink connector for sending Kafka data to Google Cloud Storage.

But there’s an issue: many of these connectors have either been released as proprietary or undergone license changes that have restricted their use.

That is part of the reason why we’ve begun to create our own connectors, to support our customers’ needs. However, there’s an additional philosophical reason—to continue supporting the open source community.

Not the first connector nor the last

Our first two open source Kafka connectors are sinks to Amazon S3 and GCS. We’ll continue to improve on our existing connectors and extend the list. Remember, we consider your feedback when deciding what to offer.

So make sure you let us know what Kafka connectors you’re interested in and if there is enough demand, we’ll make sure to include it.

In the meantime, sign up to our changelog and blog RSS feeds to stay up-to-date with the latest Aiven news, and don’t forget to follow us on Twitter or LinkedIn. If you’re not an Aiven customer, test our platform out with a 30 day free trial.

Related resources