Tiered Storage reduces storage costs, increases operational flexibility, and provides unlimited data retention scalability
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Aiven announces the general availability of Tiered Storage in Aiven for Apache Kafka®. Tiered storage enables Apache Kafka® to seamlessly move older, less frequently accessed data to a lower cost storage tier within Kafka. This enables infinite data retention while substantially reducing storage costs and improving performance and operational efficiency. Tiered storage in Aiven for Apache Kafka is production-ready for AWS, Google Cloud, Azure, and soon to be available for Bring Your Own Cloud (BYOC) deployments for these cloud providers.
The popularity of Apache Kafka for building real-time data pipelines and event-driven architectures continues to accelerate. Larger datasets are being stored for longer durations, bringing growing challenges of scalability, performance, operational management, and especially cost-efficiency.
Traditionally, Kafka stored all data on fast local disks and defaulted to a short 3- to 7-day retention period to mitigate the high cost of storage. As data accumulated, customers either purchased more storage, or moved data out of Kafka to less expensive datastores like Hadoop Distributed File System (HDFS) or other databases, data lakes, or data warehouses for long-term retention and analysis. But this created a fragmented data access experience for consumers that needed to access both fresh and historical data-–maintaining queries to both Kafka and other datastores is complex and cumbersome.
Tiered storage addresses this challenge where storage is no longer the bottleneck or budget buster in scaling Kafka. Log segments on the Kafka broker node remain on local disk for fast access, and cold data is automatically moved within Kafka to lower cost cloud object storage like Amazon S3, Google Cloud Storage, or Azure Blob Storage. Using tiered storage reduces the size of broker node storage, which eliminates the need for large node-to-node data replication operations and makes upgrades and migrations smoother and more performant.
Tiered storage optimizes costs, simplifies scaling, boosts performance, and improves operational efficiency. These benefits make Apache Kafka a more versatile and cost-effective solution for both real-time and long-term data storage. For use cases involving large volumes of historical data, the cost savings can be very substantial.
Understanding when and why to use tiered storage in Aiven for Apache Kafka helps you maximize its benefits, particularly around cost savings and system performance.
Top scenarios for using tiered storage include:
Tiered storage operates seamlessly for both Apache Kafka producers and consumers. When tiered storage is enabled, data produced to a topic is initially stored on the local disk of the Kafka broker. Data is transferred to remote storage asynchronously, as configured by the Kafka admin, and does not interfere with the producer activity. When consumers fetch records stored in remote storage, there is a slight delay introduced when the Apache Kafka broker gets the records from remote storage. However, the broker downloads and caches these records locally, enabling faster access in subsequent retrievals.
Tiered storage pricing starts from $0.09 per GB/month, depending on the precise object storage system used. Costs are determined by the amount of remote storage used, measured in GB/hour, and will be confirmed when enabling tiered storage in the Aiven console. For a detailed explanation of tiered storage pricing, please check the documentation. Tiered storage supports cloud storage services from all three hyperscalers – Amazon S3, Google Cloud Storage, and Azure Blob Storage.
Please contact Aiven Support to request that tiered storage be made available to you at the project level.
Read more about Tiered Storage in our blog article Introducing Tiered Storage for Aiven for Apache Kafka®: Unlocking Improved Cost-Efficiency and Scalability and see how to start using it in A guide to Apache Kafka® tiered storage with Aiven and Terraform.
