Cybersecurity leader secures competitive advantage while saving 30%-40% on Apache Kafka costs while streaming 50TB of data daily, per region, with no data loss
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
Sophos is a global leader of advanced security solutions for defeating cyberattacks. Already one of the largest pure-play cybersecurity providers in the world, the company continues to expand and is focused on delivering cybersecurity as a service. Its flagship product, Sophos Central, is underpinned by an advanced data platform which relies on Aiven for Apache Kafka on AWS to move vast amounts of data at speed. By adopting Aiven’s Bring Your Own Cloud (BYOC) model, Sophos has kept the benefits of using the Aiven service while taking advantage of compute savings plans in its AWS account and gaining granular control over the Aiven managed service deployments and networking.
Sophos is on a mission to develop powerful and intuitive products and services that provide the world's most effective cybersecurity for organizations of any size. Sophos defends more than 600,000 organizations and more than 100 million users from adversaries, ransomware, phishing, malware and more.
In a multi-threat world, where cyberattackers constantly evolve their threat vectors, the company continues to innovate. It is particularly focused on delivering cybersecurity as a service — so it can give even more organizations the necessary protection from attack.
Cybersecurity is a data business. Sophos’s ability to provide early, comprehensive threat detection — and response — depends on its ability to rapidly handle phenomenal and increasing amounts of data every day. As cyber threats evolve, becoming more intelligent and more complex, Sophos must constantly adapt to new trends.
In 2018, Sophos launched its Cloud Management SaaS product, Sophos Central, which combined real-time information sharing between products with automated incident response and a unified management console. By aggregating threat, health and security information across Sophos’s network of customer products, applications and end-points, Sophos Central created much larger data pools. This provided deeper insights into potential cyberattacks, enabling Sophos analysts and customers to respond faster and more effectively to emerging threats.
When the company created Sophos Central, it had to build a powerful new data platform to underpin it. A key component was Apache Kafka® which ingests and processes all the streaming data in real time. Sophos chose Kafka as it’s considered to be the best-in-class solution. Due to the high volume of data and critical nature of the pipelines, the company decided not to manage Kafka internally but instead chose Aiven’s managed Kafka service.
“We enjoy support from the Aiven team which is responsible for versioning, upgrades and so on,” says Brian Campbell, Senior Director Software Engineering at Sophos. “I have a team of 30 people on the data platform. If we were managing the Kafka clusters in house, I’d need another team of five to six people.”
Sophos enriches its Kafka data streams by running sophisticated analytics, rules engines, AI models and other elements to gain additional context, insights and value. This helps identify and report anomalies and make a diagnosis about the nature of the threat, the level of risk, and how best to react.
“We handle complex processing, with multiple steps involved in data ingestion and enrichment. Kafka plays a crucial role in ensuring both reliable delivery and data resiliency throughout the process," says Campbell.
By early 2024, Campbell’s team had 56 Kafka clusters in production and another 22 in pre-production environments running across nine AWS regions globally. “Our Kafka pipelines had grown to the point that they ran in nine AWS regions, and in just one region, we had 50 terabytes of data running through them every day, increasing by 5% per month,” says Campbell. “Fortunately, Aiven proved that it could scale with us without any service degradation.”
However, Sophos Central had high operational costs, with Kafka itself a major contributor. The company was constantly looking for ways to reduce operational expenses, considering it a competitive advantage that helps bring down the overall unit costs of its products and services.
Of course, any reduction in cost could not come at the expense of performance. “Kafka is mission critical,” says Campbell. “Any data loss compromises our ability to detect and prevent future threats from cyberattacks.”
Initially, the Aiven for Kafka service was deployed on Aiven-managed infrastructure on AWS. In this arrangement, Sophos was not able to apply discounted pricing from AWS to Kafka.
Sophos looked into the Aiven’s BYOC model on AWS as a way of reducing its Kafka-related costs without sacrificing performance or scalability. With this model, Aiven would continue to manage Kafka, but importantly the clusters would run on cloud infrastructure within Sophos’s AWS account. This would allow Sophos to apply its AWS compute savings plans to the Aiven service.
Sophos began its BYOC migration on AWS with proofs of concept and testing before transitioning to live migration. Aiven's self-service automation for AWS BYOC streamlined the process, enabling a rapid and successful migration.
“We migrated all 79 Kafka clusters with BYOC, including some very large instances, across the nine AWS regions with zero downtime and zero data loss. And we did it in a single month, rather than the four months we’d anticipated,” says Campbell.
“Data loss is unacceptable in our business. If we had seen any data loss or downtime from the migration, it would have compromised our ability to safeguard our customers. Instead, all of them remain protected at every stage of the BYOC AWS migration.”
With the successful transition, Sophos is now taking advantage of an AWS savings plan. “Aiven’s BYOC on AWS solution has saved us between 30% and 40% of our total spend on the mission-critical Aiven for Kafka service,” says Campbell. “And cost savings translate into competitive advantage for us.”
All this was achieved with no performance degradation, which means Sophos has maintained its mean-time-to-detect (MTTD) of an attack to an industry-beating five minutes. Security is enhanced with BYOC on AWS by enabling more strategic and controlled access restrictions. For instance, workloads can be isolated from external networks, reducing potential vulnerabilities.
Overall, Sophos has successfully moved to a more cost-effective cloud data platform, capable of handling its projected 5% month-on-month growth in data volumes and enabling it to maintain its competitive edge in the ever-evolving security landscape. “Since moving Aiven for Kafka to a BYOC on AWS model, we’re better placed to outmaneuver the competition, stay ahead of threat actors and keep protecting our customers.”
As data volumes continue to grow, Sophos will continue to rely on Aiven for Kafka, and its BYOC on AWS model, to manage its data flows and power its cybersecurity solutions. Sophos is also exploring additional Aiven services, including Aiven for Valkey™ for high-performance caching, message queues and efficient data storage.
