Panu Kalliokoski is here to tell you AND your boss why Aiven is the way to go. Feel free to send the link along to the higher-ups in your company!
Panu Kalliokoski
|RSS FeedSenior Software Engineer at Aiven
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Before starting to work for Aiven, I was already an active Aiven user (that’s why I ended up working here). Aiven provided a very good solution to the question: how do I best run the data-oriented background services, such as databases, in a modern, automated way? But it wasn’t always so easy to make everyone else see it the same way. This article is a story about what kind of challenges I faced and what I did about them.
TL;DR: the really hard part is often to get management to accept good solutions. This is especially true when the initiative comes from specialists and the decision is not purely technical. But if you can muster enough good arguments, you can... manage it. (See what I did there?)
Particularly in more established and complex organizations, it may be sometimes very difficult to have everyone agree on how things should be done. Personal factors come into play, there’s pure inertia to deal with, and sometimes also power struggles. All this makes life difficult for people who just want to get things done.
Some managers are trusting and delegate real responsibility. Others are secretly suspicious that developers just want “toys to play with” and “don’t get the big picture”.
Aiven is a bit of an oddball. We don’t fit into people’s idea of what a cloud provider is. Aiven doesn’t host its own cloud; instead, Aiven provides a management and compatibility layer that works with multiple cloud providers.
A better way to describe us would perhaps be a cloud management system, managing your cloud providers for you.
With good enough arguments for the manager level, it’s usually possible to make everyone reach the same conclusion as you already did. Sometimes the best argument is “we already did it and it’s working very well”, but then you need to do a little work to make everyone feel happy about the right solution.
One of the tasks in my previous job (with the City of Helsinki) was to produce a CI pipeline and DevOps infrastructure that the development teams could use to provision environments for testing, staging, and production on a self-service basis. The goal was to give developers full control over their environments by putting the appropriate configuration in their Github repositories.
As the Product Owner, I defined how it should all work and what should get done first. As a bonus, I was also a user of the system, so I experienced the usability at first hand. In addition, I was the tech lead for other development teams, which helped with collecting requirements. The bulk of the actual implementation work was done mainly by our friends at Anders Innovations.
Not surprisingly, the technical part was not the hard one. Aiven played very well the role of a safe and efficient data storage layer, and infrastructure was managed centrally using Terraform for both Aiven and the software layer. The latter was run on Kubernetes in GCP (GKE) & Azure (AKS).
Why two different cloud providers? Because for production, the only “interoperability” requirement from upper management was to use Azure! So we ran the test environments in GCP where we could see what was going on, and production environments on Azure to make upper management happy.
Aiven accommodated to this design neatly by allowing us to run exactly the same kind of database service irrespective of which cloud we were using.
The hard part was to agree with management and other departments that this was a good setup. I’m not going to go much into detail about what happened in this particular project, but believe me when I tell you that the problems and solutions presented in this article are based on real experience...
I've listed some of the most commonly heard objections, and my responses to them. Feel free to borrow any/all.
Sometimes big organizations have a vetted cloud provider, probably one of the big ones (AWS, GCP or Azure). All cloud providers offer some kinds of hosted database solutions, so it’s easy to see why from the management point of view it doesn’t make sense to have another cloud provider for databases.
The sad truth is, the database solutions of big cloud providers are not always very thorough. Nor are they developed for highly specialised needs, such as optimizing performance for certain kinds of data. Doing data services properly is not most cloud providers’ main business, unless it’s some product of their own which is targeted at a specific customer segment. Many cloud databases don’t even offer backups out of the box. And let’s not even mention high availability, smooth failovers, rich health monitoring, and rolling upgrades.
Aiven provides all that, and also actually works on the cloud you’re using for other stuff. You can have the data services in the same data center(s) as the software services using them, and in the same private network too.
The big difference between data services and software services is that well designed software services are throwaway, only causing a couple of cut connections and maybe some cache misses. Data services are actually the most valuable asset of many companies. It’s always devastating to lose all of your data, and sometimes you can’t afford to lose any of it.
And it’s not always about server failures. When you lose a lot of data by admin error (a missing WHERE clause is sometimes all it takes), having functionality for proper point-in-time recovery is indispensable. Of course, Aiven provides that for many service types.
So, data services are not just another kind of services running on the cloud. They are the most valuable thing you have, and they require specialised expertise to run properly. It might make sense for your company to just hire that expertise, but it might also make sense to use a product that has all that working correctly to start with.
Many organisations are ordering all their services from one subcontractor just to make the accounting easier. However, it often creates problems for operations, and developers will also sometimes feel the consequences.
If your chosen vendor is AWS or GCP, you’re in luck. Aiven services can also be provisioned through the AWS and GCP marketplaces, so you get them billed as part of your main cloud provider expense. They’re more expensive that way, but hey, at least you’re keeping the accountants happy.
If that doesn’t work for you, just contact our billing. There are a lot of different options for receiving invoices, and maybe we can work out some kind of agreement with whoever usually handles your invoices.
Aiven’s pricing model is built for predictability. You always know how much your monthly expense will be. So from the budgeting point of view, the only unpredictable factor is whether your service usage will grow in huge proportions so that you need to start using a more beefy plan. But dealing with that is usually a positive problem - more usage usually translates to more cash flow.
This is actually a pretty valid point! As is the case with all of these open source databases (and other data services too, like Kafka), they are actively trying to make life easier for users and they are pretty easy to run.
On the other hand, if you’re actually relying on these services, you really need to have people who know what to do when something goes wrong. You also need to prepare for disaster - it doesn’t help to start sorting things out when there are problems. Aiven comes with sensible defaults, and you can get different levels of support from Aiven for your data services.
Aiven provides the same software as the upstream open source project, not some customised version of it. So you can expect everything to work exactly the same if you migrate into or out of Aiven.
It's also viable to first run your own services for development or prototyping. When the service actually becomes valuable, it’s very easy to switch to a properly managed platform such as Aiven. Once provided by Aiven, the services can also be scaled up and out without service interruptions, and support levels can be improved, too.
That said, it’s not very expensive to use Aiven directly from the beginning. The hobbyist plans are kind of test versions but already hosted by Aiven. Which way to go about service prototyping is up to you.
Unlike most cloud providers, Aiven services are the direct opposite to a vendor lock. We keep our customers by offering good service, not by tricking them to use vendor specific features and extensions.
Being based on unmodified open source means that the migration path out of Aiven is very easy and for many services you can do this without service interruptions. Also, you can run Aiven provisioned services side by side with your own and duplicate your data to both if you want to be extra sure that getting out is an option.
On a different level, Aiven helps enormously when you want to get out of other cloud vendor locks. Aiven features seamless migration of services between clouds. When you switch cloud providers, migrating data services is usually the worst and most error prone part of the process. Being able to do that by a simple API call is a lot of effort saved.
Even when you’re not migrating, the ability to run an exactly similar service on multiple clouds helps if you need to run in different environments - as we did in my previous job. Even if you’re not actively planning to move between clouds, having this option available takes away some of the stress of choosing one.
It might be true, but it might also be just a misconception. Aiven integrates very well with many existing observability and alerting services, and provides good support for IaC (infrastructure-as-code) tools. We have a Terraform provider, a Kubernetes operator, and a clean API for automating all kinds of setup tasks. So the answer to this of course depends on what you’d like to integrate into, but most likely the blocks are already there.
Sometimes it also happens that the building blocks provided by Aiven are better than what you’re currently using. We offer state-of-the-art open source logging and metrics services and you can run them side by side with the old ones. It might well happen that when management sees the new dashboards, they want to switch over immediately.
While the need for Aiven can be worked around with, the other options will probably incur more risks or be more expensive. If properly running a data service is not your primary business but not irrelevant to your success either, you will most likely get better service for less money if you get a managed data solution such as Aiven.
For the City of Helsinki, the story had a happy ending: we were able to defend our choice of using Aiven in face of all the questions that were raised.
I hope this article provides some talking points that clarify your arguments for adopting Aiven at your company. If you run into something new not discussed here, don’t hesitate to contact me or our solution architects. We will be very interested in discussing the forces involved in your case.
--
Not using Aiven services yet? Sign up now for your free trial at https://console.aiven.io/signup!
In the meantime, make sure you follow our changelog and blog RSS feeds or our LinkedIn and Twitter accounts to stay up-to-date with product and feature-related news.
