Say goodbye to Kafka Connect upgrade hassles with the ability to pin specific connector versions.
Michael Tansini
|RSS FeedProduct Manager, Data Streaming
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Connecting the data across your business and getting it where it needs to be can often be challenging and place undue operational stress across your application, infrastructure, and platform teams. Apache Kafka®, and in particular the Apache Kafka® Connect framework simplifies these pain points by allowing you to use Kafka to transport data from where it is produced, to where it needs to be stored, analyzed, or transformed.
Aiven has many years of supporting and contributing to the Kafka open source community, and in particular the Kafka Connect framework. Aiven’s contributions to items like Tiered Storage enable customers to stream data from sources without worrying about storage becoming a bottleneck in your platform operations. Aiven now offers over 30+ connectors, allowing you to retrieve data from where it resides and sync it uninterruptedly to your target locations 24/7. However, when Aiven has released the latest support version of a connector, there are often two reactions: excitement over what has been fixed or optimized, and concern about having to upgrade to a newer version, with any edge cases or unknown bugs that may emerge. Fixing that can demand developers' time - a resource few and far between, with cascading impacts on downstream teams’ workflows who may rely on the timely delivery of this data.
Aiven’s goal is to simplify the running of your software platform, and speaking to customers confirmed our beliefs that we needed to find a better way to allow you to test, update, and, if necessary, revert connector updates. Now Aiven is introducing multi-version support for new connectors for increased flexibility and control when using and updating Apache Kafka Connect. Users can now pin selected connector versions to their Kafka Connect instance, ensuring version stability independent of other maintenance updates. It allows users to test and revert connector changes easily, simply by changing the pinned version, minimising the number of changes that are made at any one time to incrementally change business-critical data flows. Where connector versions are deprecated, users will not lose connectivity but will receive alerts so they can take appropriate action.
This functionality is available for all Aiven customers on our Debezium, JDBC, and Snowflake connectors, and can be applied via the Aiven API and Terraform for dedicated Kafka Connect services. These connectors are some of our most used integrations and can be quickly set up to support industry-standard patterns like Change Data Capture (CDC). Users on our older Debezium connector version 1.9 will no longer need to contact support to upgrade to the newest version - they can simply pin the new Debezium version to upgrade. In the future, We'll be rolling out more connector versions in the coming months, but if there's a particular connector and version you're interested in seeing support for, feel free to use Aiven Ideas and let us know!
What does the future hold for the Kafka connect framework? One key change Aiven is looking out for is KIP-891, which allows customers to have multiple versions of the same connector on a single Kafka Connect instance. If you have a distributed engineering function, with individual teams responsible for updating the connector versions, this greatly simplifies the operational burden of updating these connector versions and not causing downstream data issues.
If you want to get started with Aiven for Apache Kafka® why not have a look at:
How to get started with Aiven’s Kafka, including our documentation on launching a dedicated Kafka Connect cluster today!
The documentation on Multi-Version Connector support and how to pin connector versions to ensure continuity of behaviour
Use the power Change Data Capture pattern with Kafka Connect and Debezium to capture and act on incremental updates as soon as possible
