Aiven helps Schibsted Media Group maintain a stable Apache Kafka® cluster essential to making timely data-driven business decisions and creating business value out of data
Markos Sfikas
|RSS FeedProduct Marketing Manager Streaming
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
How do established media companies survive in the digital age? That’s the question challenging newsrooms and board meetings around the world. For Schibsted Media Group, the largest media group in Scandinavia, part of the answer has been expansion into the digital space and seeking to be data-driven.
Schibsted is a world-class media house, and today offers financial services as well as online marketplaces that connect buyers and sellers throughout Europe’s Nordic region. It’s a highly successful model and advertising sales from the new marketplaces has become a major revenue stream for the business. In Q4 2022, Nordic Marketplaces delivered underlying revenue growth of 6% driven by double-digit growth in classifieds revenues. It’s also a model that relies on high-performance IT systems and rapid data throughput, explains Joanna Eriksson, Staff Data Engineer, Schibsted.
Eriksson’s team enables collection and distribution of behavioral data which allows to create value in multiple ways: by enabling news sites to cover most relevant news fast, to deliver personalized experience for marketplaces users and enable advertisers to deliver targeted messaging. It’s a complex system that handles 1.2 billion ‘events’ every day. On normal days, peak traffic is about 90,000 events per second while on the occasion of major breaking news it can increase by 50%.
About seven years ago, Schibsted set-up its data processing pipeline for tracking events and to transfer event data between various applications at speed. It used an Apache Kafka® cluster hosted in Amazon Web Services — the cloud made it easier and more cost effective to manage the regular spikes in traffic volume. “The cloud enables us to focus on our core business. It’s good to be able to abstract away a lot of the operations that come with having our own physical hardware,” explains Eriksson.
“We chose Apache Kafka as our data-streaming platform because it’s ideal for event-driven applications and near real-time data transfer,” Eriksson says. “It’s well known for its excellent performance, low latency, fault tolerance and high throughput. It's capable of handling hundreds of thousands of events per second. It was just what we needed and unlocked new use cases.”
At the beginning, the open-source technology delivered what Schibsted needed but as demands increased, its performance was put under increasing pressure. The Kafka platform is critical because it directly impacts the EUR 230 million Schibsted generates each year through ad revenue. As Eriksson explains, “If the Kafka pipeline goes down, it impacts our ability to measure and understand user activity which also directly affects revenue from advertising. The link is very clear.”
Schibsted’s digital media subscriptions also generate significant and growing revenues for the company. “Subscription customers rightly expect to see relevant content when they log in,” says Eriksson. “And that content is based on the data that we collect and provide. The paid experience needs to be superior to the free experience.”
The Kafka cluster therefore needs to be highly stable, and Eriksson’s team was becoming increasingly concerned that the stability it needed was lacking. “We were pushing our Kafka cluster to the limit with heavy workloads. Our team needed to become Kafka experts to keep the system optimized. It just wasn’t possible,” says Eriksson.
That’s when Aiven came on to the scene and Schibsted switched to Aiven for Apache Kafka®, running on AWS. With Aiven managing the system on its behalf, Schibsted is now benefiting from 99.99% uptime providing Eriksson’s team with the reassurance it needed. The service has resulted in significant improvements in stability, even in periods of very high traffic when it comfortably handles 135,000 events per second, much more than was possible previously.
“We have millions of Euros in ad revenue depending on this system so the potential downstream impact of interruption to our Kafka service is huge,” says Eriksson. “With Aiven for Apache Kafka, at last we have a high performance, low latency and super resilient platform managed by someone else.”
On the occasion when issues do arise, Schibsted has access to the Aiven support team 24/7 to figure out what is going on and fix it. “The Kafka experts at Aiven have gone above and beyond what we expected, even advising on elements that fall outside what is managed by Aiven. They’ve helped us with debugging issues in our own applications and advised on parameters we can tweak on our side to optimize the performance of the system,” Eriksson says.
“You can’t predict when issues will happen—they usually end up being at really inconvenient times like Christmas eve rather than 10am on a Monday,” says Eriksson. “Now we have Aiven experts who can help us figure things out in those stressful situations so no more festive panics for us!”
As Schibsted continues to deliver more user and customer value through its marketplaces, and strengthens its classifieds offerings, the work of Eriksson’s team will become even more critical to the business. With Aiven for Apache Kafka, the team has a resilient Kafka cluster that can keep pace with the business and scale as required.
To get the latest news about Aiven and our services, plus a bit of extra around all things open source, subscribe to our monthly newsletter! Daily news about Aiven is available on our LinkedIn and Twitter feeds.
If you just want to find out about our service updates, follow our changelog.
