Aiven Inkless

Inkless is the name for Aiven's innovative Apache Kafka service, purpose-built for the cloud. Inkless modernises Kafka's design by incorporating diskless topics to slash running costs.

Why are we naming it Inkless? Apache Kafka draws its name from Franz Kafka, the novelist whose ink-on-paper craft mirrors traditional data systems’ reliance on I/O operations. "Inkless" Kafka reimagines this paradigm, replacing I/O with cloud-native storage - enabling data persistence through scalable, decentralized architectures rather than conventional disk-bound writes.

The Kafka cost estimator reflects a real-world deployment across three availability zones (AZs). It includes key features like Tiered Storage and Fetch-from-Follower, SSD-backed brokers with built-in capacity headroom, and realistic cloud pricing for compute, storage, and cross-AZ traffic.

When diskless topics are selected, the model also accounts for the lower operational effort required to run these clusters.

1. High availability and replication

All Kafka clusters modeled in the calculator are spread across three AZs. Each topic uses a replication factor of 3, ensuring durability and availability.

This replication level is fixed in the estimator and matches Apache Kafka’s default recommendation for production environments.

2. Kafka features enabled by default

The estimator assumes two key features are always on:

• Tiered Storage (KIP-405)Recent data (up to 12 hours) is stored on local SSDs. Older data is offloaded to object storage such as Amazon S3 or Google Cloud Storage.
• Fetch-from-Follower (KIP-392)Consumers read from in-zone brokers, keeping 100% of traffic local to each AZ and reducing cross-AZ costs.

3. Disk and capacity guardrails

To reflect realistic operational behavior, the estimator includes resource usage limits for each broker:

If a workload can tolerate HDD latency, the calculator may favor Diskless Topics. Offloading data to remote storage removes the local disk I/O bottleneck.

4. Cluster sizing and partition limits

To avoid excessive partition reassignments and reduce mean time to recovery (MTTR), the calculator applies Kafka community sizing guidance:

• Up to 60 brokers per cluster
• Up to 4,000 partitions per broker
• Up to 200,000 partitions per cluster
• Up to 10,000 client connections per broker

When these thresholds are exceeded, the model assumes a second cluster is created.

5. Cross-AZ network pricing

The estimator uses actual cloud provider pricing for cross-AZ data transfer:

Cross-region traffic is not included by default. To model inter-region mirroring, enable the corresponding option in the calculator.

6. Operational effort

The estimator includes operational staffing assumptions based on Aiven’s internal telemetry:

Clusters using only Diskless Topics require less manual intervention. By offloading data to object storage, Tiered Storage removes the need for local disk management. This simplification also reduces the impact of incidents. The staffing estimates used in the calculator are intentionally conservative compared to public total cost of ownership (TCO) models.

BYOC (Bring Your Own Cloud) is Aiven’s deployment model that runs Diskless Topics - and optionally other Aiven services - directly inside your own cloud environment. This model gives you full control over your environment while Aiven manages operations through its control plane.

• Custom cloud setup: You register your cloud account with Aiven. A lightweight agent is deployed along with the required IAM roles and network rules in your specified subnets. Learn more
• Service deployment: When you launch a Diskless Topics service, all components, including agents, caches, and object store connectors, run inside your cloud account. Kafka data is written directly to your own S3 (or equivalent) buckets.
• Management plane connection: A secure connection links your cloud environment to the Aiven Management Plane for monitoring, scaling, and upgrades. No Kafka topic data leaves your VPC. Learn more.
• CLI and automation: You can automate deployments using the avn byoc CLI or with Terraform modules generated by the Aiven Console. Learn more.

• Cost control: Compute, storage, and network usage stay within your own cloud account, allowing you to take advantage of reserved instance pricing and long-term discounts.
• Compliance and residency: Data remains entirely within your environment. Logs, snapshots, and backups do not leave your cloud account, helping you meet data residency and audit requirements.
• Simplify procurement: There are no additional vendor VPCs or private-link charges. All billing appears in your existing cloud invoice.
• Operational freedom: Aiven automates scaling, patching, and upgrades, freeing your team from the overhead of self-hosting and manual operations.

BYOC tiers are flat-fee subscription levels based on the sustained compressed ingress throughput of your Diskless Topics BYOC deployment.

Each tier includes:

• Aiven’s self-healing control plane
• 99.99% uptime SLA
• 24×7 site reliability engineering (SRE) coverage
• Security patches and automated version upgrades

Pricing scales with the volume of data (in MB/s) streamed through Kafka. Higher throughput maps to higher tiers, but every tier provides the same operational benefits.

Running Kafka at scale requires engineering expertise and operational maturity. With Diskless Topics in BYOC, data is stored in your own object storage, and compute runs in your own cloud account—enabling you to benefit from your provider’s cost optimizations. Aiven handles deployment, monitoring, upgrades, and recovery.

The pricing model is designed to keep the cost per MB/s low while meeting Aiven’s reliability and automation standards. It simplifies operations and offers a cost-effective alternative to managing Kafka infrastructure in-house.

Sample BYOC tiers

The following tiers represent 95th percentile sustained throughput ranges. They are not hard limits. Occasional short bursts above the defined range are allowed. If sustained traffic increases, you can upgrade tiers without downtime.

Flexibility and cloud consistency

These tiers are designed to be flexible starting points, not rigid constraints. Aiven’s team can:

• Recommend the best instance type based on your workload (for example, compute-heavy compression, memory-heavy fan-outs)
• Create a custom tier if your traffic pattern includes irregular spikes, long-tail ETL loads, or regulatory requirements.

The pricing remains consistent across AWS, Google Cloud, and Azure. You retain any savings from reserved instances, committed use discounts, or storage tiering available in your own account.

Diskless topics are now an integrated feature within the Aiven for Kafka service, specifically for Bring Your Own Cloud(BYOC) deployments. To use them, enable the feature as part of your service's creation. You can find this option in the Console UI or when using other creation methods like the CLI, API, or Terraform.

Diskless topics are fully compatible with traditional Kafka topics. They use the same producer and consumer APIs, preserve message ordering and offsets, support exactly-once semantics, and can run alongside classic topics in the same cluster—no application changes needed.

The difference is in how data is stored. Instead of writing to broker disks, diskless topics stream data directly to cloud object storage.

This approach offers three key benefits:

• Lower storage costs by using scalable object storage instead of premium disks
• Independent scaling of compute and storage
• Longer retention without adding broker capacity

Diskless topics are not ideal for ultra-low-latency workloads that require sub–500 ms end-to-end delivery. For those scenarios, traditional disk-based Kafka is a better fit.

However, for use cases that can tolerate one to two seconds of latency between producing and consuming data, diskless topics offer significant advantages. They reduce storage costs and allow compute and retention to scale independently.

Where diskless topics shine