This online marketplace provider moved to the cloud, using Apache Kafka to improve scalability and reduce maintenance effort.
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
Fast growing online marketplace platform provider Mirakl was spending too much time and effort creating and maintaining a monolith system for each customer. Aiven for Apache Kafka enabled their transformation to cloud- native services.
Mirakl, founded in 2012, powers more than 300 online marketplaces. They enable brands, retailers and distributors (both B2C and B2B businesses) to build platforms that sell both their own products but also third-party inventory.
Mirakl’s original solution called for an independent monolith system to be created for each customer. As their business expanded, this approach began to require too much effort. For one thing, a monolith setup is fragile: if one part of the application must be updated or just crashes, all features are impacted and the customer is left without services, and the provider’s margins suffer. Add up all that downtime, and that’s a lot of wasted capacity.
Mirakl also ran into scalability issues. As their customers’ business grew they wanted to buy more capacity, but adding it was a huge investment in terms of effort, and customers were reluctant to scale up with no subsequent way to scale back down at need. This hampered not only the customers, but also Mirakl’s own growth and bottom line.
In 2016, Mirakl had had enough of the monolith. They decided to take a leap into a cloud-native microservice architecture that would reduce their downtime and be far easier to scale. Mirakl also wanted a truly open source solution in order to avoid vendor lock.
Right from the outset, Mirakl knew that Apache Kafka was the only sensible solution for their business.
Initially they ran an Apache Kafka cluster themselves, and soon discovered how painful it could be. “We upgraded it twice ourselves, and after the second time I vowed never to do it again,” says Matthieu Mouminoux, Lead Architect at Mirakl.
So they started to look for a SaaS provider for their full Kafka system.
“We were looking for a partner with a high degree of operational excellence,” says Nagi Letaifa, VP of Engineering at Mirakl. “They would also need to offer extensive support and be widely compliant with security standards.”
They discovered a newly-founded company, called Aiven, offering a fully managed Apache Kafka solution, with transparent pricing and easy setup.
“I liked how you just create an account and then it already works. No calling up the vendor or setting up implementation meetings,” Matthieu says. This made it easy to test and evaluate the service and get a better understanding of what they were getting into.
Mirakl’s due diligence soon revealed that Aiven maintained compliance with all major data security regulations. So after a period of light testing, Mirakl decided to take Aiven for Apache Kafka out for a more serious spin.
"I liked how you just create an account and then it already works."
Matthieu Mouminoux
Lead Architect at Mirakl
So how did it go, betting on Aiven? After three years, Mirakl are more than happy. They are seeing multiple benefits from using Aiven for Apache Kafka, as opposed to setting up their own Kafka cluster. One big advantage is of course the fact that Aiven handles all the management and upgrade tasks.
“It’s critical for us to have our developers focused on delivering value, instead of struggling with operations,” Nagi says. “To put it bluntly, we’re an e-marketplace company, not a Kafka maintainer.”
Mirakl are also discovering that their time-to-market has been greatly reduced. The reason lies very much at the grass root level. A monolithic application has a single codebase, which limits the number of developers or teams who can work on it concurrently. With microservices, teams can work on different parts of the system at the same time. This cuts down on development time and increases the agility of the business.
At first glance, using a managed service looks more expensive than a self-hosted cluster. But after you add up the maintenance, support and connector development costs, and add the constant concern of keeping your critical resources onboard, the total cost of ownership is actually much lower.
Mirakl are currently running about 100 servers and tons of microservices; Kafka processes some 9000 messages every second. This means upgrades are a headache, but they’re no longer Mirakl’s headache.
“Thanks to Aiven, the time our teams used to spend on running Apache Kafka can instead be poured into our core business. We’ve found exactly the partner we were looking for. They take care of the infrastructure securely and reliably, but we still hold the keys to our own data.”
"Developers can stay focused on their job, instead of managing Kafka."
Matthieu Mouminoux
Lead Architect at Mirakl
