Configure HashiCorp Vault
Configure and use HashiCorp Vault as a secret provider in Aiven for Apache Kafka® Connect services.
Prerequisites
- Access to the Aiven Console.
- Aiven for Apache Kafka service with Apache Kafka Connect set up and running.
- Aiven CLI.
- Aiven Terraform Provider installed.
- HashiCorp Vault address and token.
The integration with HashiCorp Vault is not yet available on the Aiven Console.
Configure secret providers
Set up HashiCorp Vault in your Aiven for Apache Kafka Connect service to manage and access sensitive information.
- API
- Terraform
- CLI
Use the ServiceUpdate
API to update your service configuration. Add the HashiCorp Vault configuration to
the user_config
using the following API request:
curl --request PUT \
--url https://api.aiven.io/v1/project/{PROJECT_NAME}/service/{SERVICE_NAME} \
--header 'Authorization: Bearer YOUR_BEARER_TOKEN' \
--header 'Content-Type: application/json' \
--data '{
"user_config": {
"secret_providers": [
{
"name": "vault",
"vault": {
"auth_method": "token",
"address": "https://vault.aiven.fi:8200/
"token": "YOUR_VAULT_TOKEN"
}
}
]
}
}'
Parameters:
PROJECT_NAME
: Name of your Aiven project.SERVICE_NAME
: Name of your Aiven Kafka service.url
: API endpoint for updating service configuration. Replace{project_name}
and{service_name}
with your project and service names.Authorization
: Token used for authentication. ReplaceYOUR_BEARER_TOKEN
with your Aiven API token.Content-Type
: Specifies that the request body is in JSON format.auth_method
: Authentication method used by HashiCorp Vault. In this case, it is token.address
: Address of the HashiCorp Vault server.token
: Your HashiCorp Vault token.
Configure HashiCorp Vault using the Aiven Terraform Provider. Add this configuration to your main.tf
file,
or optionally create a dedicated vault.tf
file for managing secret providers:
-
Create the
main.tf
file for your resources:terraform {
required_providers {
aiven = {
source = "aiven/aiven"
version = ">=4.0.0, < 5.0.0"
}
}
}
provider "aiven" {
api_token = var.aiven_api_token
}
resource "aiven_kafka_connect" "kafka_connect" {
project = var.project_name
cloud_name = "your-cloud-region"
plan = "startup-4"
service_name = "kafka-connect"
kafka_connect_user_config {
secret_providers {
name = "vault"
vault {
auth_method = "token"
address = "https://vault.aiven.fi:8200/"
token = var.vault_token
}
}
}
} -
Declare variables in a
variables.tf
file:variable "aiven_api_token" {
description = "Aiven API token"
type = string
}
variable "project_name" {
description = "Aiven project name"
type = string
}
variable "vault_token" {
description = "HashiCorp Vault token"
type = string
sensitive = true
} -
Provide variable values in a
terraform.tfvars
file:aiven_api_token = "YOUR_AIVEN_API_TOKEN"
project_name = "YOUR_PROJECT_NAME"
vault_token = "YOUR_VAULT_TOKEN"
Parameters:
project_name
: Name of your Aiven project.cloud_name
: Cloud provider and region for hosting the Aiven for Apache Kafka service. Replace with the appropriate region. For example,google-europe-west3
.plan
: Service plan for the Aiven for Apache Kafka Connect service.service_name
: Name of your Aiven for Apache Kafka Connect service.auth_method
: Authentication method used by HashiCorp Vault. Set totoken
.address
: URL address of the HashiCorp Vault server.token
: Token used for authenticating with HashiCorp Vault.aiven_api_token
: API token for authentication. Replace with your Aiven API token to manage resources.
Configure HashiCorp Vault as a secret provider using Aiven CLI:
avn service update SERVICE_NAME \
-c secret_providers='[
{
"vault": {
"auth_method": "token",
"address": "https://vault.aiven.fi:8200/",
"token": "YOUR_VAULT_TOKEN"
},
"name": "vault"
}
]'
Parameters:
SERVICE_NAME
: Name of your Aiven Kafka service..name
: Name of the secret provider. In this case,vault
.auth_method
: Authentication method used by HashiCorp Vault. In this case, it istoken
.address
: Address of the HashiCorp Vault server.token
: Your HashiCorp Vault token.
Reference secrets in connector configurations
You can use secrets stored in HashiCorp Vault with any connector. The examples below show how to configure secrets for JDBC connectors, but you can follow the same steps for other connectors.
JDBC sink connector
- API
- Terraform
- CLI
Configure a JDBC sink connector using the API with secrets referenced from HashiCorp Vault:
curl -X POST https://api.aiven.io/v1/project/{PROJECT_NAME}/service/{SERVICE_NAME}/connectors \
-H "Authorization: Bearer YOUR_BEARER_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "your-connector-name",
"connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:{DATABASE_TYPE}://{HOST}:{PORT}/{DATABASE_NAME}?user=${vault:PATH/TO/SECRET:USERNAME}&password=${vault:PATH/TO/SECRET:PASSWORD}&ssl=require",
"topics": "your-topic",
"auto.create": true
}'
Parameters:
PROJECT_NAME
: Name of your Aiven project.SERVICE_NAME
: Name of your Aiven Kafka service.name
: Name of the connector.connector.class
: Specifies the connector class to use, in this case,io.aiven.connect.jdbc.JdbcSinkConnector
.connection.url
: JDBC connection URL with placeholders forDATABASE_TYPE
,HOST
,PORT
,DATABASE_NAME
, and the username and password retrieved from HashiCorp Vault.topics
: Apache Kafka topic where the data can be sent.auto.create
: Iftrue
, the connector automatically creates the table in the target database if it does not exist.
Configure a JDBC sink connector using the Aiven Terraform Provider with secrets referenced
from HashiCorp Vault. Add this configuration to your main.tf
file, or optionally create
a dedicated vault.tf
file for managing secret providers:
resource "aiven_kafka_connector" "jdbc_sink_connector" {
project = var.project_name
service_name = aiven_kafka_connect.kafka_connect.service_name
connector_name = "jdbc-sink-connector"
config = {
"connector.class" = "io.aiven.connect.jdbc.JdbcSinkConnector"
"connection.url" = "jdbc:postgresql://{HOST}:{PORT}/{DATABASE_NAME}?user=${vault:PATH/TO/SECRET:USERNAME}&password=${vault:PATH/TO/SECRET:PASSWORD}&ssl=require"
"topics" = "your-topic"
"auto.create" = "true"
}
}
Parameters:
project
: Name of your Aiven project where the Kafka Connect service is located.service_name
: Name of the Aiven for Kafka Connect service where the connector is to be created.connector_name
: Name for your JDBC sink connector.connector.class
: The Java class that implements the connector. For JDBC sink connectors, use"io.aiven.connect.jdbc.JdbcSinkConnector"
.connection.url
: The JDBC URL for your database. Replace{HOST}
,{PORT}
, and{DATABASE_NAME}
with your actual database details. The username and password are retrieved from HashiCorp Vault using the${vault:PATH/TO/SECRET:USERNAME}
and${vault:PATH/TO/SECRET:PASSWORD}
syntax.topics
: Apache Kafka topics that the connector consumes data from.auto.create
: Iftrue
, the connector automatically creates the table in the target database if it does not exist.
Configure a JDBC sink connector using the Aiven CLI with secrets referenced from HashiCorp Vault.
avn service connector create SERVICE_NAME '{
"name": "jdbc-sink-connector",
"connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:{DATABASE_TYPE}://{HOST}:{PORT}/{DATABASE_NAME}?user=${vault:PATH/TO/SECRET:USERNAME}&password=${vault:PATH/TO/SECRET:PASSWORD}&ssl=require",
"topics": "your-topic",
"auto.create": true
Parameters:
SERVICE_NAME
: Name of your Aiven Kafka service.name
: Name of the connector.connector.class
: Specifies the connector class to use, in this case,io.aiven.connect.jdbc.JdbcSinkConnector
.connection.url
: JDBC connection URL with placeholders forDATABASE_TYPE
,HOST
,PORT
,DATABASE_NAME
, and the username and password retrieved from HashiCorp Vault.topics
: Apache Kafka topic where the data can be sent.auto.create
: Iftrue
, the connector automatically creates the table in the target database if it does not exist.
JDBC source connector
- API
- Terraform
- CLI
Configure a JDBC source connector using the API with secrets referenced from HashiCorp Vault:
curl -X POST https://api.aiven.io/v1/project/{PROJECT_NAME}/service/{SERVICE_NAME}/connectors \
-H "Authorization: Bearer YOUR_BEARER_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"name": "your-connector-name",
"connector.class": "io.aiven.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:{DATABASE_TYPE}://{HOST}:{PORT}/{DATABASE_NAME}?ssl=require",
"connection.user": "${vault:PATH/TO/SECRET:USERNAME}",
"connection.password": "${vault:PATH/TO/SECRET:PASSWORD}",
"incrementing.column.name": "id",
"mode": "incrementing",
"table.whitelist": "your-table",
"topic.prefix": "your-prefix_",
"auto.create": true
}'
Parameters:
PROJECT_NAME
: Name of your Aiven project.SERVICE_NAME
: Name of your Aiven Kafka service.name
: Name of the connector.connector.class
: Specifies the connector class to use, in this case,io.aiven.connect.jdbc.JdbcSinkConnector
.connection.url
: JDBC connection URL with placeholders forDATABASE_TYPE
,HOST
,PORT
,DATABASE_NAME
, and the username and password retrieved from HashiCorp Vault.connection.user
: Database username retrieved from HashiCorp Vault.connection.password
: Database password retrieved from HashiCorp Vault.incrementing.column.name
: Column used for incrementing mode.mode
: Mode of operation, in this case,incrementing
.table.whitelist
: Tables to include.topic.prefix
: Prefix for Apache Kafka topics.auto.create
: Iftrue
, the connector automatically creates the table in the target database if it does not exist.
Configure a JDBC source connector using the Aiven Terraform Provider with secrets referenced from
HashiCorp Vault. Add this configuration to your main.tf
file, or optionally
create a dedicated connectors.tf
file for managing Apache Kafka connectors:
resource "aiven_kafka_connector" "jdbc_source_connector" {
project = var.project_name
service_name = aiven_kafka_connect.kafka_connect.service_name
connector_name = "jdbc-source-connector"
config = {
"connector.class" = "io.aiven.connect.jdbc.JdbcSourceConnector"
"connection.url" = "jdbc:postgresql://{HOST}:{PORT}/{DATABASE_NAME}?ssl=require"
"connection.user" = "${vault:PATH/TO/SECRET:USERNAME}"
"connection.password" = "${vault:PATH/TO/SECRET:PASSWORD}"
"incrementing.column.name" = "id"
"mode" = "incrementing"
"table.whitelist" = "your-table"
"topic.prefix" = "your-prefix_"
"auto.create" = "true"
}
}
Parameters:
project
: Name of your Aiven project where the Kafka Connect service is located.service_name
: Name of the Kafka Connect service where the connector is to be created.connector_name
: Name for your JDBC source connector.connector.class
: The Java class that implements the connector. For JDBC source connectors, use"io.aiven.connect.jdbc.JdbcSourceConnector"
.connection.url
: The JDBC URL for your database. Replace{HOST}
,{PORT}
, and{DATABASE_NAME}
with your actual database details. The username and password are retrieved from HashiCorp Vault using the${vault:PATH/TO/SECRET:USERNAME}
and${vault:PATH/TO/SECRET:PASSWORD}
syntax.connection.user
: The username for connecting to the database, retrieved from HashiCorp Vault.connection.password
: The password for connecting to the database, retrieved from HashiCorp Vault.incrementing.column.name
: The name of the column that the connector will use for incrementing mode, typically a primary key column.mode
: The mode of operation for the connector.incrementing
mode reads data incrementally based on the specified column.table.whitelist
: A list of tables that the connector includes when reading data from the database.topic.prefix
: Prefix for Apache Kafka topics.auto.create
: Iftrue
, the connector automatically creates the table in the target database if it does not exist.
Configure a JDBC source connector using the Aiven CLI with secrets referenced from HashiCorp Vault:
avn service connector create SERVICE_NAME '{
"name": "jdbc-source-connector",
"connector.class": "io.aiven.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:{DATABASE_TYPE}://{HOST}:{PORT}/{DATABASE_NAME}?ssl=require",
"connection.user": "${vault:PATH/TO/SECRET:USERNAME}",
"connection.password": "${vault:PATH/TO/SECRET:PASSWORD}",
"incrementing.column.name": "id",
"mode": "incrementing",
"table.whitelist": "your-table",
"topic.prefix": "your-prefix_",
"auto.create": true
}'
Parameters:
SERVICE_NAME
: Name of your Aiven Kafka service.name
: Name of the connector.connector.class
: Specifies the connector class to use, in this case,io.aiven.connect.jdbc.JdbcSinkConnector
.connection.url
: JDBC connection URL with placeholders forDATABASE_TYPE
,HOST
,PORT
,DATABASE_NAME
, and the username and password retrieved from HashiCorp Vault.connection.user
: Database username retrieved from HashiCorp Vault.connection.password
: Database password retrieved from HashiCorp Vault.incrementing.column.name
: Column used for incrementing mode.mode
: Mode of operation, in this case,incrementing
.table.whitelist
: Tables to include.topic.prefix
: Prefix for Apache Kafka topics.auto.create
: Iftrue
, the connector automatically creates the table in the target database if it does not exist.