Apache Kafka® infrastructure moved to AWS bringing new levels of scalability and reliability to support growth
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
As one of Europe's leading online comparison platforms for products and travel and flight, idealo is a data-intensive business. Part of Axel Springer SE, the company has ambitions to further expand its presence across Europe. However, with an on-premises data infrastructure that was struggling to scale, idealo decided that moving to AWS was an essential part of its growth strategy. idealo chose Aiven to manage its Apache Kafka® infrastructure in the cloud, removing the management burden from internal teams and allowing them to focus on innovation – and cementing its position in a highly competitive market.
idealo started in 2000 with a mission to help users make the best possible purchasing decisions thanks to transparent pricing. It has been part of media and tech giant Axel Springer SE since 2007. Every day, idealo parses billions of offers from shops of all sizes and assigns each offer to a specific product that users can find easily in the idealo app and website. The company now has an average of 76 million visits per month in Germany alone and displays more than 500 million offers from around 50,000 shops, making it one of the most popular German e-commerce websites and a leading platform in the highly competitive price-comparison space. idealo has also expanded into France, the UK, Italy, Austria and Spain, and is continuing to innovate and diversify its offering to stay ahead of the competition and achieve its goal of becoming No.1 in Europe.
Over the past two decades, idealo’s e-commerce and price comparison sites have grown significantly. The company has added merchants and product lines, expanded into different European markets and responded to changing online retail trends like Black Friday. As the business scaled up, its IT infrastructure was coming under more and more strain – notably around high-volume trading periods like Christmas.
“We made the strategic decision to move our IT infrastructure to AWS for better scalability. Our data center couldn’t provide the capacity we needed to grow the business and support our teams’ plans,” says Wanis Fahmy, Senior Cloud Engineer, idealo.
Everyday idealo collaborates with thousands of merchants who share information about their listed products – including pricing, descriptions and images. idealo then processes, analyzes, ranks and shares that data on its website. “With 50,000 merchants and 500 million product offers, we handle massive amounts of data so having a fast, reliable data infrastructure is a critical part of our business model. People are interested in the best price now, not five minutes ago. Delays, inaccurate data or intermittent service could impact trust in the idealo brand,” says Fahmy. Consequently, idealo has been a long time and heavy user of Apache Kafka, the open source data streaming platform.
Before migrating to AWS, idealo had 30 on-premises Kafka clusters transferring data between the systems and apps underpinning its price comparison service. The company set itself an ambitious cloud migration plan to be completed in just 18 months. Moving the data infrastructure, including Kafka, was an essential part of the plan.
Although the team had acquired solid experience of managing Kafka clusters on-premises, there were concerns it wasn’t sufficient for a cloud-based operation. Any loss of performance or, even worse, any downtime would result in a noticeable and immediate loss in trading. idealo needed a reliable partner to help with the migration and subsequent management of Kafka.
“Making sure our Kafka deployment is always up to date and always in sync with best practice is a huge task for us. Moving to the public cloud gave us the opportunity to find ways to hand this stress over to the experts, so our team could focus on our business,” says Fahmy.
idealo evaluated different vendors and the team chose Aiven. “Aiven was the only vendor that ticked all the boxes,” says Fahmy. “The other SaaS providers couldn’t match the technical capabilities of the Aiven data platform and we also wanted predictability, especially with pricing.”
Aiven’s straightforward, easy-to-understand pricing model was pivotal in the evaluation. “Moving to the public cloud was new for us, with a lot of different cost and pricing models. We weren’t sure how that would turn out – but we did know that soaring costs from things like data egress charges would really undermine our plans for change,” says Fahmy.
The migration to AWS would be staged, with some teams moving out of the data centers earlier than others. “One of the key technical requirements was that the Kafka clusters would work in dual mode – using Virtual Private Cloud (VPC) peering to ensure connection to the on-premises clusters, as well as allowing access to the ones already migrated to AWS. Aiven was the only vendor able to provide the flexibility to work both modes. It was very easy to switch back and forth, for example, to disable public access and just rely on peering and vice versa,” explains Fahmy.
idealo embarked on a successful proof of concept with Aiven for Apache Kafka® which provided all the technical reassurance required. Soon after it started moving the Kafka clusters to Aiven on AWS.
Honey Feelisch, Project Manager, idealo, was the lead project manager for the AWS migration. Even when some teams were still running their applications on-premises, Feelisch encouraged everyone to start using Aiven for Apache Kafka as a central data hub by mirroring or replicating the messages. “This strategy allowed us to untangle the dependencies between the teams. It also gave the freedom and the capability for everyone to listen to the Kafka messages and to think about how they could benefit,” says Feelisch.
Three months after the proof of concept was completed, the Kafka infrastructure had successfully moved into production in the cloud. Today, around 60 Kafka clusters are managed by Aiven, supporting a number of different teams with an increasingly wide range of use cases.
“We use Kafka a lot for event-driven asynchronous communication between teams and their various microservices,” says Fahmy. “The second biggest use case takes advantage of Aiven for Apache Kafka® Connect which some of our teams use for the ingestion of data into our big data lake in AWS.”
Since moving to Aiven, the popularity and usage of Kafka has increased. “With Aiven, our Kafka clusters are much more reliable than they were before. That means our teams can focus on the core business and not care about maintenance. It's also very easy to start up a new cluster now. That makes it a lot more attractive and accessible to other teams,” says Fahmy.
From a business perspective, idealo has the reassurance that the Kafka components of its critical data infrastructure are highly available at all times and that data will continue to be moved quickly and successfully even during peak periods like Black Friday.
“Aiven helps us accommodate increases in workloads and data volumes. This means the systems underpinning our price comparison site can scale and adjust to our business needs as we continue to grow in popularity, expand into new markets and meet our ambitious growth targets,” says Andreas Hankel, CTO, idealo.
The focus of the last year has been on migrating workloads to AWS and Aiven for Apache Kafka. But as Fahmy notes, working with Aiven is creating new possibilities and new use cases that hadn’t previously been possible. “In the past some teams avoided using Kafka because it’s complicated, but now, thanks to Aiven, they have an opportunity to create new clusters and create new customer services based on that. Moving forward, we’re going to learn a lot and I’m excited to see the innovation it will spark.”
