We’ve recently created and released the Aiven GCS Kafka Sink Connector. This connector will allow you to move data from your Kafka clusters to Google Cloud Storage. Even better, it is open source.
By making it open source, we ensure that you can use our connector regardless of whether you host and manage Kafka yourself or choose a service provider to do so.
You can learn how to configure and use it here. If you would like to make contributions to the project and request features, visit our project here. But before you go, let’s look at Kafka connectors in a little more detail.
Why Kafka connectors are important
Simply put, Kafka needs connectors to live up to its full potential. These connectors allow you to couple Kafka with other systems so that you can easily do things such as streaming Change Data Capture.
They come in two forms, source and sink. The former connects a source of data to Kafka while the latter does the opposite. For example, the GCS sink connector for sending Kafka data to Google Cloud Storage.
But there’s an issue: many of these connectors have either been released as proprietary or undergone license changes that have restricted their use.
That is part of the reason why we’ve begun to create our own connectors, to support our customers’ needs. However, there’s an additional philosophical reason—to continue supporting the open source community.
Not the first connector nor the last
Our first two open source Kafka connectors are sinks to Amazon S3 and GCS. We’ll continue to improve on our existing connectors and extend the list. Remember, we consider your feedback when deciding what to offer.
So make sure you let us know what Kafka connectors you’re interested in and if there is enough demand, we’ll make sure to include it.
In the meantime, sign up to our changelog and blog RSS feeds to stay up-to-date with the latest Aiven news, and don’t forget to follow us on Twitter or LinkedIn. If you’re not an Aiven customer, test our platform out with a 30 day free trial.
Dec 17, 2020
5 benefits of an Apache Kafka®-centric microservice architecture
Microservices are the most flexible way to provide an online service today. Find out why Apache Kafka is a good choice when microservices are on the line.
Jun 9, 2022
What’s coming for Uptime 2022?
Check out this post for the program highlights for Uptime 2022 - the conference on open source data technology in the cloud, organized by Aiven.
Floor & Ana
Feb 15, 2022
Move from batch to streaming with Apache Kafka® and Apache Flink®
Find out how Apache Kafka® and Apache Flink® allow you to move from batch processing to streaming, but keep using SQL in the data pipeline.
Subscribe to the Aiven newsletter
All things open source, plus our product updates and news in a monthly newsletter.