Processing Thousands of Data Events with Sub-15ms Latency: How Kamino Retail is Disrupting Advertising with Aiven

Fast-growing AdTech startup Kamino Retail processes thousands of data events per second with sub-15 ms latency to power hyper-targeted retail media campaigns. Leveraging Aiven for Apache Kafka® and Aiven for ClickHouse® on Google Cloud, Kamino Retail seamlessly handles 3x traffic surges during peak events. Despite handling and storing a huge volume of data, Kamino Retail has significantly cut costs by using ClickHouse’s tiered storage. With Kamino Retail anticipating up to 50x data growth, these savings will grow exponentially. The Aiven platform empowers Kamino Retail to scale globally, disrupt traditional advertising and drive significant revenue growth for Kamino Retail and its customers.

Ambitions for global growth

Founded in early 2023 in France, Kamino Retail is a rapidly expanding AdTech startup specializing in retail media. By leveraging targeted customer advertising at the online point of sale, Kamino Retail enables online retailers to monetize their inventory and is disrupting traditional digital advertising.

The company boasts a strong client base, including prominent French retailers Auchan and Fnac Darty. A recent acquisition by Equativ, an independent AdTech company spanning North and South America, Europe and Asia, has significantly accelerated Kamino Retail’s growth trajectory.

Together, Kamino Retail & Equativ offer a flexible, innovative, and agnostic all-in-one platform to maximize the value of retailers’ advertising space and 1st party data. The platform supports on-site and off-site activation for both endemic and non-endemic brands, offering support for direct sales, and capturing programmatic demand to significantly boost retailers’ revenue.

With this momentum, Kamino Retail now aims to scale its operations across Europe and ultimately enter the US market.

Critical role of real-time data

Real-time data is the cornerstone of Kamino Retail’s business model. The company must process a huge volume of events with incredible speed, capturing crucial information like ad requests, clicks and user interactions on its customer retailer websites. For example, it has a sub-50-ms response time SLA for ad requests. This rapid data flow is critical for Kamino Retail to deliver highly targeted and effective product promotions to customers, while preserving their shopping experience.

Kamino Retail’s revenue model is based on revenue sharing with its customers, making campaign performance directly linked to mutual success. This necessitates accurate quoting, forecasting and performance metrics for both Kamino Retail and its customers.

“To optimize our ad campaigns, we meticulously track a high volume of events to understand ad spending effectiveness and gauge audience engagement,” says Emmanuel Valette, one of Kamino Retail’s co-founders and current CTO. “This is business critical. If we don’t track events, we don’t generate revenue. The more data we track, the more effectively we can predict future outcomes and optimize campaign performance.”

Choosing the right platform for startup growth

Based on his successful experience at a previous company, Emmanuel Valette chose the Aiven platform to establish a highly scalable, real-time data infrastructure for Kamino Retail when creating the business. The use of open-source market-standard solutions was important to avoid vendor lock-in.

“Aiven’s managed services model reduced setup time from one month to just one week, crucial for a resource-constrained startup like Kamino Retail in its earliest days. The model also allows us to serve retailers with millions of daily website visitors without requiring any infrastructure oversizing,” says Emmanuel Valette. “Moreover, Aiven’s cloud-agnostic approach aligns perfectly with our customer needs and enhances our attractiveness to potential prospects, both critical factors for our early-stage growth.”

ClickHouse and Kafka: a powerful combination

Kamino Retail uses Aiven for Apache Kafka for high-throughput data handling, and Aiven for ClickHouse for efficient data storage and analytics, both running on Google Cloud Platform.

“The built-in integration between Aiven for Apache Kafka and Aiven for ClickHouse is key,” says Emmanuel Valette. “Both services were super easy and fast to get up-and-running and it’s great having a single control panel.”

The powerful combination gives Kamino Retail tremendous event tracking capabilities, enabling it to capture and process thousands of events per second.

Event tracking: sub-15 ms request to Kafka latency

“The speed and scalability of the system is impressive. The process of receiving a request and sending an event to the Kafka cluster takes less than 15 ms in 99% of cases, despite the huge volume of data (>5 TB) we tracked in Clickhouse for our retail clients in our first 18 months,” says Emmanuel Valette.

With strict high-performance, low-latency service levels, particularly for its ad server, Kamino Retail developed a custom connector layer for the Apache Kafka cluster using Rust. This seamlessly integrates with Kafka to stream data into ClickHouse, enabling Kamino Retail to efficiently gather data from diverse marketing campaign sources.

The Clickhouse cluster only requires two engineers to manage. The company uses the tiered storage feature in Aiven for ClickHouse, which allows it to organize and store its data in two tiers.

Tiered storage reduces costs while 50x growth in data anticipated

Data older than three months is automatically transferred from the faster, more expensive SSD storage to the less expensive, slower Object Storage. This is well-suited for infrequently accessed historical data, making it a cost-effective solution for storing valuable historical data for future analysis. This allows Kamino Retail to significantly optimize storage costs while ensuring historical data remains readily available for future analysis.

“It feels like the ClickHouse tiered storage feature was designed for us,” says Gaetan Bervet, Data Engineer at Kamino Retail. “In 18 months, we’ve tracked 5 TB of data for our retail clients in object storage, saving us a significant amount in storage costs as well as engineering time. We anticipate our object storage will grow by a factor of 10 to 50 over the next few years as our business expands.”

Analytics and visualization tools

To perform analytics, both for itself and its customers, Kamino Retail uses ClickHouse’s Materialized View feature. Gaëtan Bervet and his team set up the materialized view to pre-process events that come in before query time, which enables the raw events to be filtered out and leaves only the relevant data to be queried. It is a much less complex and time-consuming process than filtering at query time which leads to lower query latency for customers.

Kamino Retail’s various internal users query these views with Superset, an open-source dashboard visualization layer, facilitating rapid iteration and streamlined business operations. Its customers access performance data, and a near real-time view of the value and revenue generated from Kamino Retail, through the dedicated Kamino Retail platform. In both cases, no user experiences delays because they are powered by Aiven for ClickHouse, which provides best-in-class analytics power and blazing fast query execution.

Seamlessly scaling for peak events

Retail traffic exhibits significant seasonality, with peak periods experiencing traffic volumes up to three times higher than average, necessitating substantial infrastructure flexibility. With the Aiven platform in place, Kamino Retail can deal with such surges, free from concerns about data infrastructure scaling limitations.

“For this year’s Black Friday, we knew Aiven would handle the traffic we were expecting and, if required, we could make a few tweaks to the Kafka configuration. For example, adding storage with Aiven for Kafka's Dynamic Disk Space (DDS) feature. In the end, we scaled our operations over the Black Friday shopping period to support three times the normal volume of events,” says Gaëtan Bervet.

Kamino Retail’s campaign monitoring continues to evolve. Its ad-programming team gets near-instantaneous feedback on campaign performance metrics, which allows them to continually optimize campaigns. Kamino Retail is also able to generate comprehensive daily reports on campaign performance, which inform its own future strategies, revenue-maximizing tactics and even the shape of its product roadmap.

The right partner for business expansion

As Kamino Retail continues to expand its global reach leveraging Equativ’s network, its reliance on ClickHouse and Kafka will burgeon. “Aiven has provided us with a great service since we started. As we scale our business, we are confident that our Kafka and ClickHouse clusters and the support we get from Aiven will grow with us,” says Emmanuel Valette.