Get started with Aiven for Apache Kafka® Connect

Get started with Aiven for Apache Kafka® Connect and integrate it with an Aiven for Apache Kafka® service.

Prerequisites

Your project must include at least one Aiven for Apache Kafka® service. If it does not, create one.

Console

• Access to the Aiven Console

Terraform

• Terraform installed
• A personal token

Create a dedicated Aiven for Apache Kafka® Connect service

Create a Kafka Connect service and integrate it with an Aiven for Apache Kafka® service.

note

For Inkless Kafka services, Kafka Connect must run as a dedicated service.

If no Kafka Connect service exists, create a Kafka Connect service and enable the integration from the Kafka service.

Console

1. Log in to the Aiven Console and open the Aiven for Apache Kafka service.
2. In the left sidebar, click Connectors.
   • For Classic Kafka services, click Integrate standalone service.
   • For Inkless Kafka services, click Create Kafka Connect.
3. In the Kafka Connect integration dialog, do one of the following:
   • If a compatible Kafka Connect service exists, select Existing service and choose the service.
   • If no compatible service exists, select New service to create one.
4. If you create a new Kafka Connect service, configure the service:
   1. Select a cloud provider and region.
   2. Select a service plan.
   3. In Service basics, enter a service name. You cannot change the service name after creation.
5. Click Create service.
6. After the service is created, return to the integration dialog and click Enable to enable the Kafka Connect integration.

After the integration is enabled, Kafka Connect appears as an integration on the Kafka service. Wait until the service status changes from Rebuilding to Running before using it.

Terraform

The following example files are also available in the Aiven Terraform Provider repository on GitHub.

1. Create a file named provider.tf and add the following:

   Loading...

2. Create a file named service.tf and add the following:

   Loading...

3. Create a file named variables.tf and add the following:

   Loading...

4. Create the terraform.tfvars file and add the values for your token and project name.

To apply your Terraform configuration:

1. Initialize Terraform by running:

   terraform init

   The output is similar to the following:

   
   Initializing the backend...
   
   Initializing provider plugins...
   - Finding aiven/aiven versions matching ">= 4.0.0, < 5.0.0"...
   - Installing aiven/aiven v4.9.2...
   - Installed aiven/aiven v4.9.2
   ...
   Terraform has been successfully initialized!
   ...

2. To create an execution plan and preview the changes, run:

   terraform plan

3. To deploy your changes, run:

   terraform apply --auto-approve

Next steps

• Review the Aiven examples repository for Kafka Connect configuration examples.
• Generate test data using the sample data generator.
• Find available connectors in the list of supported Kafka Connect plugins.