Create an Aiven for Apache Kafka® service

You can create an Aiven for Apache Kafka® service using the Aiven Console, CLI, or Terraform. During creation, you can enable diskless topics for Bring Your Own Cloud (BYOC) deployments. If you do not enable diskless topics, the service stores topic data on local disks by default.

Decide whether to enable diskless topics

Choose the configuration that fits your workload:

Standard Kafka service: Uses local disk storage for lower latency and all-region availability.
Kafka service with diskless topics: Stores data in cloud object storage for cost-optimized scaling in Bring Your Own Cloud (BYOC) environments.

Diskless topics are currently supported only for BYOC deployments on AWS.

note

You cannot enable diskless topics on an existing Kafka service that was created with local storage only. To use diskless topics, create a Kafka service with diskless support enabled. Once enabled, you can create both diskless and classic topics within that service.

For details on the differences between topic types, see Classic vs. diskless topics.

Prerequisites

Make sure you have the following:

Console
CLI
Terraform

Access to the Aiven Console
An Aiven project to create the service in

Additional requirements for diskless topics

To create a Kafka service with diskless topics, make sure that:

You have a BYOC environment set up in your cloud account on AWS.
Diskless topics are enabled for your organization by Aiven. If the option does not appear in the Aiven Console, contact Aiven support.

Create a Kafka service

Create a Kafka service that stores topic data on local disks by default.

Console
CLI
Terraform

In your project, click Services.
Click Create service.
Select Aiven for Apache Kafka®.
In the Optimize cost section, keep diskless topics turned off to create a standard Kafka service.

tip
To create a Kafka service with diskless topics instead, see Create a Kafka service with diskless topics (BYOC).
Select a Cloud.

note
Available plans and pricing vary between cloud providers and regions.
Select a Plan.
Optional: Add disk storage. You can also enable Tiered storage to offload older data automatically to object storage.
In the Service basics section, set the following:
- Service name: Enter a name for the service.
  important
  You cannot change the name after creation.
- Version: Select the Kafka version. The latest supported version appears by default.
- Tags: Optional. Add resource tags to organize your services.
Review the Service summary. Confirm the version, region, plan, and estimated price.
Click Create service.

The service status changes to Rebuilding during creation. When it changes to Running, your Kafka service is ready.

Create a Kafka service using the Aiven CLI.

avn service create SERVICE_NAME \
  --service-type kafka \
  --cloud CLOUD_REGION \
  --plan PLAN_NAME

Parameters:

SERVICE_NAME: The name of the Kafka service
CLOUD_REGION: The cloud and region
PLAN_NAME: The plan name

Wait until the service status changes to RUNNING.

Use Terraform to create a Kafka service in your Aiven project.

Create a file named provider.tf and add the following:
```
Loading...
```
Create a file named service.tf and add the following:
```
Loading...
```
Create a file named variables.tf and add the following:
```
Loading...
```
Create the terraform.tfvars file and add the values for your token and project name.
Optional: To output connection details, create a file named output.tf and add the following:
```
Loading...
```

To apply your Terraform configuration:

Initialize Terraform by running:

terraform init

The output is similar to the following:

Initializing the backend...

Initializing provider plugins...
- Finding aiven/aiven versions matching ">= 4.0.0, < 5.0.0"...
- Installing aiven/aiven v4.9.2...
- Installed aiven/aiven v4.9.2
...
Terraform has been successfully initialized!
...

To create an execution plan and preview the changes, run:
```
terraform plan
```
To deploy your changes, run:
```
terraform apply --auto-approve
```

Create a Kafka service with diskless topics (BYOC)

Use diskless topics to store Kafka data in cloud object storage instead of local disks. You can use both diskless and classic topics in the same Kafka cluster.

For instructions on setting up a BYOC environment, see Create a custom cloud (BYOC).

Console
CLI
Terraform

In your project, click Services.
Click Create service.
Select Aiven for Apache Kafka®.
Under Optimize cost, turn on Enable diskless topics.
Under Add service metadata, set the following:
- Version: Select the Kafka version. The latest supported version appears by default.
  note
  Diskless topics require Apache Kafka® version 4.0 or later.
- Service name: Enter a name for your service.
  important
  You cannot change the name after creation.
- Tags: Optional. Add resource tags to organize your services.
Select the cloud provider, BYOC region, and plan.
Under Select plan, choose one of the plans available for diskless topics.
Review the Service summary on the right. Confirm the version, region, plan, and estimated price.
Click Create service.

You can create a Kafka service with diskless topics enabled using the Aiven CLI.

avn service create SERVICE_NAME \
  --project PROJECT_NAME \
  --service-type kafka \
  --cloud CLOUD_NAME \
  --plan PLAN_NAME \
  -c kafka_version=4.0 \
  -c kafka_diskless.enabled=true

Parameters:

SERVICE_NAME: Name of your Kafka service.
PROJECT_NAME: Your Aiven project name.
CLOUD_NAME: Custom BYOC cloud region, for example custom-aws-eu-central-1.
PLAN_NAME: Diskless-compatible plan, such as business-8-inkless. Plans that support diskless topics include -inkless in the plan name.
kafka_diskless.enabled: Enables diskless topics. Must be set to true.

You can create a Kafka service with diskless topics enabled using Terraform.

Create a file named main.tf and add the following:

terraform {
  required_providers {
    aiven = {
      source  = "aiven/aiven"
      version = ">=4.0.0, <5.0.0"
    }
  }
}

provider "aiven" {
  api_token = var.aiven_token
}

resource "aiven_kafka" "diskless_kafka" {
  project      = var.aiven_project_name
  service_name = "kafka-diskless"
  cloud_name   = "custom-aws-eu-central-1"
  plan         = "business-8-inkless"

  kafka_user_config = {
    kafka_version         = "4.0"
    kafka_diskless = {
      enabled = true
    }
  }
}

Create a variables.tf file:

variable "aiven_token" {
  description = "Aiven API token"
  type        = string
}

variable "aiven_project_name" {
  description = "Your Aiven project name"
  type        = string
}

Initialize and apply your configuration:

terraform init
terraform apply --auto-approve

After service creation

When you create a Kafka service with diskless topics, Aiven deploys it directly in your BYOC environment using your connected cloud account. The service runs entirely within your cloud account.

Aiven configures the following:

Access to object storage for storing Kafka topic data, either through an Aiven-managed or a customer-provided bucket, depending on your BYOC configuration.
A PostgreSQL-based coordinator managed as a service integration with Kafka. This coordinator maintains message ordering and metadata consistency for diskless topics. It is required for the current implementation of diskless topics. For details about how the coordinator is upgraded, see PostgreSQL service upgrades.

After creation, the Kafka Diskless PostgreSQL integration appears on the Integrations page in the Aiven Console. This integration is managed by Aiven and cannot be modified or deleted.

To learn more about how diskless topics work, see Diskless topics overview.

Decide whether to enable diskless topics​

Prerequisites​

Additional requirements for diskless topics​

Create a Kafka service​

Create a Kafka service with diskless topics (BYOC)​

After service creation​