Open Source Data Platforms: From Handling VAT Repayments in Real-Time to Creating Europe’s Largest Virtual Hospital Ward

“I love open source, but the community is even more important,” pronounced Heikki Nousiainen, Field CTO and Co-Founder, Aiven, in his opening address at Aiven Live 2023. “It's the community that enables us to experiment, innovate, pollinate ideas and inspire us with new ways of doing things.”

This theme of innovation set the scene for a day where industry influencers, technologists, strategists, developers and end-users, including GigaOm, Revenir, Hookdeck, Dojo and Doccla, talked about how open source data platforms are unlocking business value, and, critically, how data platform thinking is changing the way we do things.

Based on the knowledge sharing, ideas exchanges and conversations, here are my five big takeaways from the day.

1. The industry influencer’s observations: What is the problem we are trying to solve?

Jon Collins, Vice President of Engagement and Senior Analyst at GigaOm, gave a keynote presentation on "The Day 1-2-3 of Data Platforms." He discussed the challenges and opportunities of the current data landscape, and how organizations can use technology platforms to deliver business value.

Collins began by referencing the proliferation of different technologies and the challenges of managing a multi-cloud, multi-everything world when both legacy and emerging technologies must work together. How over the past decade we’ve seen a move towards cloud-based models and cloud native technologies, giving plenty of options for innovation but also creating technology fragmentation, complexity, operational overheads and increased risk.

He examined the trend towards FinOps, arguing that FinOps is an indication of cost overruns and that multi-cloud architectures are having to be managed alongside on-premise environments, adding to the complexity of managing IT costs. He argued that we need to move away from a fragmented approach and towards technology platforms, that is integrated layers of technology that can be built, deployed and managed as a whole, in-house or by a third party. This strategy will dictate the skills required in-house, plus processes and policies for procurement, deployment and operations. However, most companies don’t want or need to be experts in infrastructure and data, so third parties can offer real value here.

Collins concluded that technology platforms can be the basis for real innovation and can be efficient and effective right out of the box. And that the real winners in the AI space will be the data management companies, who are the experts in, well, data. But the focus should always start and end with “what is the problem we are trying to solve?”

2. The end user customer perspective. Can a data platform drive innovation?

Three of our customers, Brian Wagner, CTO at Revenir, a London-based fintech handling VAT repayments; Phil Leggeter, Head of DevRel & PLG at Hookdeck, an event gateway provider; and Sandeep Mehta, Engineering Lead, Data Platforms at Dojo, a card payment fintech, shared their insights on the theme: How can a data platform drive innovation?

They all talked to how the pressure to drive innovation rests on doing more with less. This has meant leaning in to address cost and performance optimization, multi-cloud operations and self-service data solutions.

Revenir began with cost optimization is always on their mind — calculating everything down to the cost of a single transaction, and how, for example, Aiven for Apache Kafka helps reduce costs. Leggeter really delved into how the company’s focus must be solely on its IP, citing: “… that’s where our value lies. If we were managing the infrastructure entirely ourselves, we wouldn’t be focusing on this value. We use Aiven to optimise performance and cost.” Dojo’s focus is steely on growth. However, Mehta talked about how important it is they’re not locked into one vendor, as they like to experiment themselves, and how the Aiven streaming platform was the right choice “…it was easy to adopt, it’s very close to open source, it has the ability to work with data coming in from multiple source and cloud environments, and we can lay down the foundation to scale without fear of lock in."

On the subject of cloud partnerships with the hyperscalers, while many are not quite multi-cloud yet, there was a unified voice that being able to choose technologies and solutions that work across many clouds will be paramount. This is also essential for Revenir, working with highly regulated banks who require multi-vendor, multi-cloud strategies.

While our guest panellists all talked about how data is core to the success of their operations, it is not their IP and so having a managed service to deliver this is essential for them to focus on delivering customer value.

3. The cloud view. How collaboration with hyperscalers and ecosystems ensures everything ‘just works’

Keeping the cloud and partnership thread going, Elena Zykova, VP of OEM and Partnerships, Aiven, kicked off her session by stating: “Our partner strategy starts with our customers. There are so many technologies out there and businesses need to find the right ones. We want to build meaningful connections with those who see the world the same way we do.”

Aiven’s partner strategy is based on understanding the tech ecosystem our customers are in. We build these partnerships to help our customers become more efficient both architecturally and business-process wise and help relevant partners on their cloud migration journey to help them meet their digital transformation goals. Through these partnerships we help customers reduce time on migration and realize accelerated time to value. We want to make managing a customer’s data estate easy. And through our partnerships we give customers the freedom to move between the clouds, whether for workload, compliance or regulatory reasons, and thrive in a multi-cloud world or within their own cloud environments (BYOC).

How we make cloud migrations and integrations simpler is best summed up here: “Aiven’s expertise and customer focus were key in simplifying the migrations of our critical workloads to AWS and help our teams drive continuous innovation,” said Andreas Hankel, CTO Idealo.

In summary, Zykova synthesised Aiven’s value though ‘decreasing the noise, increasing meaningful connections with other vendors, delivering one coherent message, and leaving more time for customers to focus on their own business. We want to make sure everything just works.’

4. The developer’s lens. Accelerating Generative AI development — but not doing it solo

Delivered by Francesco Tisiot, Staff Developer Advocate and data aficionado at Aiven, Tisiot honed in on the risks associated with embracing Generative AI and how an integrated data platform can accelerate a safe adoption. According to OpenAI, 92% of Fortune 500 companies have used generative AI ­­– but Francesco explained that most don’t want to do this by themselves.

Generative AI is being embraced to solve real problems and create better content, but Tisiot asked if Generative AI is perfect? “No, but it’s really good. And it’s only as good as the questions we ask relating to existing data sets. Public data is not enough anymore. We need to spend time focusing on which data assets are relevant to the type of interaction we want from public and private data.”

But, Tisiot cautioned that cutting and pasting data into the right set simply won’t work. With Generative AI applications relying on private, regulated, or commercially valuable data, feeding the model with the right information is crucial. Failure could result in adverse consequences such as poor performance, confidential information exposure or regulatory requirement breaches. We need to have the right tools to collect the right data and remove all the data we can’t expose to AI. And we must harvest data to organise it in real time and to build trust.

In reality this requires a lot of skills and navigating complexity. But this, Tisiot explained, is also where Aiven shines in the Generative AI space. Through one, integrated data platform, companies can pick and choose the tools they use to accelerate the development of generative AI applications by providing real-time data feeds, consolidating inputs from different sources, and defining clear information boundaries.

5. The industry picture: A data-driven journey to better patient care – the power of the open source community

Matthew Huxtable, Staff Platform Engineer at Doccla talked about the organization’s ambition to become Europe’s largest virtual hospital ward. Harnessing open source technology and Aiven's data platform services, this ambitious company is in a constant state of innovation and expansion, all with the core goal of empowering patients to achieve better health and wellbeing.

“We have really gone in on software automation to provide the best care at the best possible price,” said Huxtable. Talking about the regulatory, data sovereignty and scaling challenges Doccla faces, he spoke of the power of the open source community and also why as a small team, simply doing everything themselves doesn’t make business sense. “We are an open source friendly company and value this environment where I can contribute back to the community.” But he doesn’t want to build and run critical database infrastructure being a team of one, referencing how Aiven’s platform and service has given him the flexibility and knowledge to scale but with commodity software.

If you would like to discuss any of my take-aways or customer stories, or if you're interested in learning more about how open source data platforms can help you achieve your business goals, please do get in touch with me here.