Create an Aiven for Apache Kafka® service

You can create an Aiven for Apache Kafka® service using the Aiven Console, CLI, or Terraform. During creation, you can enable diskless topics for Bring Your Own Cloud (BYOC) deployments. If you do not enable diskless topics, the service stores topic data on local disks by default.

Decide whether to enable diskless topics

Choose the configuration that fits your workload:

• Standard Kafka service: Uses local disk storage for lower latency and all-region availability.
• Kafka service with diskless topics: Stores data in cloud object storage for cost-optimized scaling in Bring Your Own Cloud (BYOC) environments.

Diskless topics are currently supported only for BYOC deployments on AWS and Google Cloud.

note

You cannot enable diskless topics on an existing Kafka service that was created with local storage only. To use diskless topics, create a Kafka service with diskless support enabled. Once enabled, you can create both diskless and classic topics within that service.

For details on the differences between topic types, see Classic vs. diskless topics.

Prerequisites

Make sure you have the following:

Console

• Access to the Aiven Console
• An Aiven project to create the service in

CLI

• Aiven CLI installed
• A personal token

Terraform

• Terraform installed
• A personal token

Additional requirements for diskless topics

To create a Kafka service with diskless topics, make sure that:

• You have a BYOC environment set up in your cloud account on AWS or Google Cloud
• Diskless topics are enabled for your organization by Aiven. If the option does not appear in the Aiven Console, contact Aiven support

Create a Kafka service

Create a Kafka service that stores topic data on local disks by default.

Console

1. In your project, click Services.

2. Click Create service.

3. Select Aiven for Apache Kafka®.

4. Under Optimize cost, keep diskless topics turned off to create a standard Kafka service.

   tip

   To create a Kafka service with diskless topics instead, see Create a Kafka service with diskless topics (BYOC).

5. Under Add service metadata, set the following:

   • Version: Select the Kafka version. The latest supported version appears by default.
   • Service name: Enter a name for the service.
     important

     You cannot change the name after creation.

   • Tags: Optional. Add resource tags to organize your services.

6. Select the cloud provider, region, and plan.

   note

   Available plans and pricing vary between cloud providers and regions.

7. Optional: Add disk storage. You can also enable Tiered storage to offload older data automatically to object storage.

8. Review the Service summary. Confirm the version, region, plan, and estimated price.

9. Click Create service.

The service status changes to Rebuilding during creation. When it changes to Running, your Kafka service is ready.

CLI

Create a Kafka service using the Aiven CLI.

avn service create kafka-demo \
  --service-type kafka \
  --cloud google-europe-west3 \
  --plan business-4

Parameters:

• kafka-demo: The name of your Kafka service.
• google-europe-west3: The cloud and region.
• business-4: The plan name.

Wait until the service status changes to RUNNING.

Terraform

Use Terraform to create a Kafka service in your Aiven project.

1. Create a file named provider.tf and add the following:

   Loading...

2. Create a file named service.tf and add the following:

   Loading...

3. Create a file named variables.tf and add the following:

   Loading...

4. Create the terraform.tfvars file and add the values for your token and project name.

5. Optional: To output connection details, create a file named output.tf and add the following:

   Loading...

To apply your Terraform configuration:

1. Initialize Terraform by running:

   terraform init

   The output is similar to the following:

   
   Initializing the backend...
   
   Initializing provider plugins...
   - Finding aiven/aiven versions matching ">= 4.0.0, < 5.0.0"...
   - Installing aiven/aiven v4.9.2...
   - Installed aiven/aiven v4.9.2
   ...
   Terraform has been successfully initialized!
   ...

2. To create an execution plan and preview the changes, run:

   terraform plan

3. To deploy your changes, run:

   terraform apply --auto-approve

Create a Kafka service with diskless topics (BYOC)

Use diskless topics to store Kafka data in cloud object storage instead of local disks. You can use both diskless and classic topics in the same Kafka cluster.

For instructions on setting up a BYOC environment, see Create a custom cloud (BYOC).

Console

1. In your project, click Services.
2. Click Create service.
3. Select Aiven for Apache Kafka®.
4. Under Optimize cost, turn on Enable diskless topics.
5. Under Add service metadata, set the following:
   • Version: Select the Kafka version. The latest supported version appears by default.
     note

     Diskless topics require Apache Kafka® version 4.0 or later.

   • Service name: Enter a name for your service.
     important

     You cannot change the name after creation.

   • Tags: Optional. Add resource tags to organize your services.
6. Select the cloud provider, BYOC region, and plan.
7. Under Select plan, choose one of the plans available for diskless topics.
8. Review the Service summary on the right. Confirm the version, region, plan, and estimated price.
9. Click Create service.

CLI

You can create a Kafka service with diskless topics enabled using the Aiven CLI.

avn service create SERVICE_NAME \
  --project PROJECT_NAME \
  --service-type kafka \
  --cloud CLOUD_NAME \
  --plan PLAN_NAME \
  -c kafka_version=4.0 \
  -c kafka_diskless.enabled=true

Parameters:

• SERVICE_NAME: Name of your Kafka service.
• PROJECT_NAME: Your Aiven project name.
• CLOUD_NAME: Custom BYOC cloud region, for example custom-aws-eu-central-1.
• PLAN_NAME: Diskless-compatible plan, such as business-8-inkless. Plans that support diskless topics include -inkless in the plan name.
• kafka_diskless.enabled: Enables diskless topics. Must be set to true.

Terraform

You can create a Kafka service with diskless topics enabled using Terraform.

1. Create a file named main.tf and add the following:

   terraform {
     required_providers {
       aiven = {
         source  = "aiven/aiven"
         version = ">=4.0.0, <5.0.0"
       }
     }
   }
   
   provider "aiven" {
     api_token = var.aiven_token
   }
   
   resource "aiven_kafka" "diskless_kafka" {
     project      = var.aiven_project_name
     service_name = "kafka-diskless"
     cloud_name   = "custom-aws-eu-central-1"
     plan         = "business-8-inkless"
   
     kafka_user_config = {
       kafka_version         = "4.0"
       kafka_diskless = {
         enabled = true
       }
     }
   }

2. Create a variables.tf file:

   variable "aiven_token" {
     description = "Aiven API token"
     type        = string
   }
   
   variable "aiven_project_name" {
     description = "Your Aiven project name"
     type        = string
   }

3. Initialize and apply your configuration:

   terraform init
   terraform apply --auto-approve

After service creation

When you create a Kafka service with diskless topics, Aiven deploys it directly in your BYOC environment using your connected cloud account. The service runs entirely within your AWS or Google Cloud account.

Aiven configures the following:

• Access to object storage for storing Kafka topic data, either through an Aiven-managed or a customer-provided bucket, depending on your BYOC configuration.
• A PostgreSQL-based coordinator managed as a service integration with Kafka. This coordinator maintains message ordering and metadata consistency for diskless topics. It is required for the current implementation of diskless topics. For details about how the coordinator is upgraded, see PostgreSQL service upgrades.

After creation, the Kafka Diskless PostgreSQL integration appears on the Integrations page in the Aiven Console. This integration is managed by Aiven and cannot be modified or deleted.

To learn more about how diskless topics work, see Diskless topics overview.

Related pages

• Diskless topics overview
• Diskless topics architecture
• Batching and delivery in diskless topics
• Create a Kafka topic