Apache Kafka® with custom configurations

Deploy an Apache Kafka® node to Aiven for Apache Kafka® with custom configurations defined in Terraform. A part of Aiven's Terraform Cookbook.

Subscribe to RSS

This example deploys an Aiven for Apache Kafka® service with some custom configurations, as well as a Kafka topic, a Kafka user, and an access control list (ACL) to allow fine-grained permissions about which topic that user can access, using the Aiven Terraform Provider

Before looking at the Terraform script, let's visualize the resources:

flowchart LR subgraph Aiven-for-Apache-Kafka id4[[Aiven Kafka User]] id5[[Aiven Kafka User ACL]] end; Producer --> Aiven-for-Apache-Kafka --> Consumer

Be sure to check out the getting started guide to learn about the common files required to execute the following recipe. For example, you'll need to declare the variables for project_name, api_token, and kafka_user_name.

Common files

Navigate to a new folder and add the following files.

Add the following to a new provider.tf file:

terraform {
  required_providers {
    aiven = {
      source  = "aiven/aiven"
      version = ">=4.0.0, < 5.0.0"
    }
  }
}

provider "aiven" {
  api_token = var.aiven_api_token
}

You can also set the environment variable TF_VAR_aiven_api_token for the api_token property. With this, you don't need to pass the -var-file flag when executing Terraform commands.

To avoid including sensitive information in source control, the variables are defined here in the variables.tf file. You can then use a *.tfvars file with the actual values so that Terraform receives the values during runtime, and exclude it.

The variables.tf file defines the API token, the project name to use, and the prefix for the service name:

variable "aiven_api_token" {
  description = "Aiven console API token"
  type        = string
}

variable "project_name" {
  description = "Aiven console project name"
  type        = string
}

variable "kafka_user_name" {
  description = "Username for Aiven for Apache Kafka user"
  type        = string
}

The var-values.tfvars file holds the actual values and is passed to Terraform using the -var-file= flag.

var-values.tfvars file:

aiven_api_token     = "<YOUR-AIVEN-AUTHENTICATION-TOKEN-GOES-HERE>"
project_name        = "<YOUR-AIVEN-CONSOLE-PROJECT-NAME-GOES-HERE>"
kafka_user_name     = "<A-SAMPLE-USERNAME>"

Services.tf file

Here is the sample Terraform script to stand-up Aiven for Apache Kafka and related resources. The script also performs some custom configurations on these resources.

services.tf file:

resource "aiven_kafka" "demo-kafka" {
  project                 = var.project_name
  cloud_name              = "google-europe-west1"
  plan                    = "business-4"
  service_name            = "demo-kafka"
  maintenance_window_dow  = "sunday"
  maintenance_window_time = "01:00:00"
  default_acl             = false

  kafka_user_config {
    kafka_rest      = true
    kafka_connect   = false
    schema_registry = true
    kafka_version   = "3.4"

    kafka {
      auto_create_topics_enable  = true
      num_partitions             = 3
      default_replication_factor = 2
      min_insync_replicas        = 2
    }

    kafka_authentication_methods {
      certificate = true
    }

    public_access {
      kafka_rest = true
    }
  }
}

resource "aiven_kafka_topic" "demo-kafka-topic" {
  project      = var.project_name
  service_name = aiven_kafka.demo-kafka.service_name
  topic_name   = "demo-kafka-topic"
  partitions   = 5
  replication  = 3
}

resource "aiven_kafka_user" "demo-kafka-user" {
  project      = var.project_name
  service_name = aiven_kafka.demo-kafka.service_name
  username     = var.kafka_user_name
}

resource "aiven_kafka_acl" "demo-kafka-user-acl" {
  project      = var.project_name
  service_name = aiven_kafka.demo-kafka.service_name
  username     = var.kafka_user_name
  permission   = "read"
  topic        = aiven_kafka_topic.demo-kafka-topic.topic_name
}

Execute the files

The init command performs several different initialization steps in order to prepare the current working directory for use with Terraform. In our case, this command automatically finds, downloads, and installs the necessary Aiven Terraform provider plugins.

terraform init

The plan command creates an execution plan and shows you the resources that will be created (or modified) for you. This command does not actually create any resource; this is more like a preview.

terraform plan -var-file=var-values.tfvars

If you're satisfied with the output of terraform plan, go ahead and run the terraform apply command which actually does the task or creating (or modifying) your infrastructure resources.

terraform apply -var-file=var-values.tfvars

Let's go over a few of these configurations and understand their functions:

aiven_kafka resource configurations:

  • default_acl parameter, when set to true, creates default wildcard Kafka ACL. This example sets this parameter to false and prevents the default wildcard ACL for resources.

  • For kafka_user_config, schema_registry is set to true, which enables the Karapace Schema Registry <https://aiven.io/blog/what-is-karapace>_ and kafka_rest allows you to view the messages in the topics from the Aiven web console when set to true.

  • auto_create_topics_enable under kafka nested configurations enables the auto creation of topics when set to true. This means that a topic doesn't need to exist before sending a message.

  • num_partitions will set the number of partitions for the automatically created topics.

  • By default, the replication factor is 2. This example sets default_replication_factor to 2 and thus requires a minimum of two brokers. For production environments, a replication factor of 3 is recommended.

  • min_insync_replicas indicates that at least 2 replicas (brokers) should respond back if all replicas(brokers) are not functioning properly. When all replicas are functioning properly, this setting has no effect.

  • The certificate parameter under the kafka_authentication_methods nested configurations, when set to true, enables certificate/SSL authentication.

aiven_kafka_topic resource configurations:

  • partitions denotes the number of partitions to create in the topic, and replication sets the replication factor for the topic.

aiven_kafka_user resource configurations:

  • We are passing a preset username using var.kafka_user_name

aiven_kafka_acl resource configurations:

  • This ACL allows read access to the demo-kafka-topic topic for the var.kafka_user_name user.

Warning

By default, Aiven adds an avnadmin account to every new service and adds admin permission for all topics to that user. When you create your own ACLs to restrict access, you probably want to remove this ACL entry.

Note

When using the Aiven Terraform Provider, you can add the default_acl key to your resource and set it to false if you do not want to create the admin user with wildcard permissions.

More resources

Keep in mind that some parameters and configurations will vary for your case. Some related resources are provided below: