Aiven now provides Kafka Connect as separate service

Aiven now supports the ability to run Kafka Connect as a separate service from your primary Kafka cluster. This eliminates resource competition, improves scalability, and saves costs.

Kafka Connect is a framework and a runtime for integrating Kafka with other systems. Kafka Connectors can either be source (for pulling data from other systems into Kafka) or sink connectors (for pushing data into other systems from Kafka).

Why Kafka Connect? Kafka Connect enables you to more easily source your data from any number of popular data sources and send it forward to any number of common data destinations.

In fact, the more connectors your service provider has, the more you’ll be able to do with your managed Kafka service. Here’s the list of currently supported Aiven Kafka Connect source and sink connectors.

Why Kafka Connect as a Service

The ability to run a Kafka Connect service separately from your primary Apache Kafka cluster eliminates resource competition, as Kafka Connect Services and Aiven for Apache Kafka services no longer cannibalize resources from each other. Also, you will be able to scale your Kafka Connect service independently of your Aiven for Kafka service. This provides a couple of benefits:

• You don’t need a more expensive tier to accommodate both Aiven for Apache Kafka and Kafka Connect on a single cluster. Previously, you needed at least Business-level service plan to provide the necessary resources. Now you can run Aiven Kafka Connect on a Startup plan.

• You’ll get more consistent and higher performance on the Kafka Service itself.

Aiven is the only cloud provider that allows Kafka Connect to run independently on a cluster.

How to create a Kafka Connect service

You can easily start a Kafka Connect service from within your existing Aiven for Apache Kafka service. To do this, open your service and go to the Connectors tab.

Click the Create New Kafka Connect Integration button and follow the prompts. This will allow you to specify an existing or new Kafka Connect service on which to run your Kafka connectors.

Wrapping up

With Kafka Connect as a Service, you have yet another option for configuring your managed Kafka service within your data pipeline; go here to see pricing. In the meantime, follow our blog or changelog RSS feeds; or, follow us on Twitter or LinkedIn to stay up to date with Aiven.