Skip to main content

Switch a classic topic to a diskless topic Early availability

Switch an existing classic topic in Aiven for Apache Kafka® to a diskless topic without copying data or renaming the topic. The topic remains available during the switch.

Records written before the switch remain readable from the classic topic log. Aiven writes new records to the diskless topic.

note

This feature is in early availability and is not enabled by default. To request access, contact your account team or Aiven support.

Prerequisites​

  • The diskless topics feature is enabled for the service. To request access, contact your account team or Aiven support.

  • The service runs Apache Kafka® 4.1 or later.

  • You have access to one of the following:

Considerations​

Before switching a topic, review the following:

  • The switch is one-way. You can't switch a diskless topic back to a classic topic. This might change in a future release.
  • Diskless topic limitations apply after the switch. Review Limitations of diskless topics to confirm that diskless topics support your workload.
  • Unclean leader election must be turned off for the topic. If unclean leader election is enabled, the switch does not start.
  • If tiered storage is not enabled for the topic, Aiven enables it automatically as part of the switch. Topics that cannot have tiered storage enabled, such as topics with compaction configured, are not eligible for the diskless switch.
  • The switch runs in the background. A successful update request doesn't mean that every partition has finished switching. You can't view per-partition switch status or progress in the Aiven CLI, Aiven API, topic configuration, or customer-facing metrics integrations.

Switch a topic to a diskless topic​

To switch a classic topic to a diskless topic, enable diskless for the topic using the Aiven CLI or Aiven API.

Run the following command:

avn service topic-update SERVICE_NAME TOPIC_NAME \
  --project PROJECT_NAME \
  --diskless-enable

Replace the following values:

  • PROJECT_NAME: Name of your Aiven project.
  • SERVICE_NAME: Name of your Aiven for Apache Kafka service.
  • TOPIC_NAME: Name of the topic to switch.

After the update request succeeds, Aiven starts switching the topic to a diskless topic.

Verify the switch request​

Verify the topic configuration to confirm that Aiven accepted the diskless switch request.

Run the following command:

avn service topic-get SERVICE_NAME TOPIC_NAME \
  --project PROJECT_NAME

Replace the following values:

  • PROJECT_NAME: Name of your Aiven project.
  • SERVICE_NAME: Name of your Aiven for Apache Kafka service.
  • TOPIC_NAME: Name of the topic.

In the command output, find diskless_enable and remote_storage_enable and verify that their VALUE is true. This confirms that Aiven accepted the diskless switch request. It does not confirm that every partition has finished switching. Per-partition switch status or progress is not exposed in the topic configuration.

What to expect during the switch​

During the switch:

  • The topic remains available.
  • Producers might briefly receive errors that clients can retry. Most Kafka clients retry these errors by default. If you have not changed the default producer retry settings, no special tuning is usually required.
  • Consumer applications do not need changes to read records written before or after the switch.
  • The topic name and retention settings do not change.
  • Aiven manages the partition-level switch process. You do not need to take action after the update request succeeds.

Producer settings​

If you changed any of these Kafka producer settings, verify that the values allow producers to retry records for at least as long as the defaults:

Producer settingDefaultWhy it matters during the switch
delivery.timeout.ms120000Allows up to 2 minutes for retrying a record.
retries2147483647Effectively unlimited and bounded by delivery.timeout.ms.
enable.idempotencetrueAvoids duplicate or reordered records.
acksallSupports enable.idempotence and durable failover.

How the switch works​

When you switch a topic to a diskless topic, Aiven does the following for each partition:

  1. Stops writing new records to the classic topic log.
  2. Waits until records written to the classic topic partitions are safely replicated.
  3. Records the offsets where the diskless topic partitions start.
  4. Initializes the diskless topic partitions from those offsets.
  5. Routes reads and writes for those partitions through the diskless storage system.

Records written before the switch, including records already moved to tiered storage, remain readable until they expire based on the topic retention settings.

Related pages