Enable tiered storage for Aiven for Apache Kafka®

Tiered storage significantly improves the storage efficiency of your Aiven for Apache Kafka® service. You can enable tiered storage for topics in a Classic Kafka service.

Prerequisites

• Access to an Aiven organization and at least one project.
• Aiven for Apache Kafka® service with Apache Kafka version 3.6 or later. Upgrade to the latest default version and apply maintenance updates for the latest fixes and improvements when using tiered storage.
• Aiven CLI.

note

Review the trade-offs and limitations of tiered storage before enabling it.

Steps to enable tiered storage

Console

1. In the Aiven Console, select your project.
2. Create an Aiven for Apache Kafka service or select an existing one.

   • For a new service:

     1. On the Create Apache Kafka® service page, scroll down to the Tiered storage section.
     2. Click Enable tiered storage.
     3. View the pricing for tiered storage in the Service summary.

   • For an existing service:

     1. Go to the service's Overview page and click Service settings from the sidebar.
     2. In the Service plan section, click Enable tiered storage to activate it.
     3. Click Activate tiered storage in the confirmation window.

When tiered storage is activated and configured for your topics, monitor usage and costs in Storage usage and settings in Aiven Console section.

note

Alternatively, if tiered storage is not yet active for your service, you can enable it by selecting Tiered storage from the sidebar.

CLI

Enable tiered storage for your Aiven for Apache Kafka service using the Aiven CLI:

1. Retrieve the project information with:

   avn project details

   For a specific project, use:

   avn project details --project <PROJECT_NAME>

2. Find the name of the Aiven for Apache Kafka service for enabling tiered storage with:

   avn service list

   Make a note of the SERVICE_NAME corresponding to your Aiven for Apache Kafka service.

3. Enable tiered storage with:

   avn service update                  \
      --project demo-kafka-project     \
      demo-kafka-service               \
      -c tiered_storage.enabled=true

In this command:

• --project demo-kafka-project: Replace demo-kafka-project with your project name.
• demo-kafka-service: Specify the Aiven for Apache Kafka service you intend to update.
• -c tiered_storage.enabled=true: Activates tiered storage.

Configure default retention policies at service-level

To manage data retention, set default policies for tiered storage at the service level.

1. In the Aiven Console, select your project and your Aiven for Apache Kafka service.
2. Click Service settings from the sidebar.
3. Scroll to the Advanced configuration section, and click Configure.
4. In the Advanced configuration dialog, click Add Advanced Configuration.
5. Define the retention policy:
   • Find kafka.log_local_retention_ms and set the value to define the retention period in milliseconds for time-based retention.
   • Find kafka.log_local_retention_bytes and set the value to define the retention limit in bytes for size-based retention.
6. Click Save configuration to apply your changes.

You can configure retention policies from the Storage usage and settings in Aiven Console page.

Optional: Configure client-side parameter

For optimal performance and reduced risk of broker interruptions when using tiered storage, update the client-side parameter fetch.max.wait.ms from its default value of 500 ms to 5000 ms.

This consumer configuration is no longer necessary starting from Apache Kafka version 3.6.2. Consider upgrading to Apache Kafka version 3.6.2 or later before enabling tiered storage.

Related pages

• Tiered storage in Aiven for Apache Kafka® overview
• How tiered storage works in Aiven for Apache Kafka®
• Enable and configure tiered storage for topics