The replatforming challenge

As the observability market and its supporting technology have evolved, open source has become central to solving the data collection challenge. The bigger difficulty is scaling the data storage and improving analytics along with visualization. With technologies like OpenSearch® and Prometheus, we’ve seen new databases and data platforms take their place in many organizations to solve observability.

There are two major reasons for this change:

• Re-instrumenting applications and infrastructure when observability tooling changes is expensive and time-consuming.
• Organizations are tired of vendor lock-in where it comes to observability tooling.

Fighting the tide

The last of the current generation of observability tools are fighting hard to preserve their proprietary instrumentation, but it's only a matter of time before they must support open source. And lip service won’t be enough--they’ll have to provide meaningful support, equivalent to the way they have supported their own proprietary instrumentation.

As a longtime practitioner, analyst, engineering, and product leader, this is the reason I’ve focused on open source observability in my career for the last 3 years. I believe we have to look at the bigger picture of data platforms. This is why I joined Aiven recently. My hypothesis is that the observability and monitoring market will undergo a similar evolution that data platforms have been undergoing.

The same trend with proprietary components is apparent in data platforms and architectures. The major cloud providers are making the same mistake: creating proprietary components in their service offerings. For example, why would a cloud provider build their own messaging components, or database query languages, or APIs? Proprietary components create challenges for users who try to create portable environments. The users become locked in.

Some cloud providers are better at adopting open source software, but most of them do so in concert with a proprietary system. All major cloud providers have weaknesses in their open source offerings and compatibility, the depth of the lock-in is what varies between their service offerings.

The open source advantage

These are the primary reasons why I believe that Aiven’s approach of open source based data platforms and streaming which are fully portable, migratable, and easily managed across clouds is an advantage. I do not know of any other tools that can move these critical components across clouds with a click of a button or an API call, with little to no downtime. It’s impressive on many fronts, and the ease of management is unsurpassed.

I do not know of any other tools that can move these critical components across clouds with a click of a button or an API call, with little to no downtime.

Moving across clouds is more relevant than ever. Most organizations seem to undertake re-platforming their components every 5 years in good economic times. In the current economic downturn and the high cost of cloud services, these timelines for cloud portability will likely accelerate. It is therefore essential for the data platform to support all the major clouds along with portability.

Being able to move services easily helps you find the best possible pricing with cloud providers, by solving the migration of core data services. Keeping cloud providers competing with one another is critical in optimizing IT spend and can lead to considerable cost savings.

Data pipelines of the future

The future path for our data platform is to incorporate more user-defined code and application code to extend the platform within Aiven’s walls. These data pipelines, driven by open source technologies, provide flexibility to recompose and analyze data in unique ways.

This will be critical for the creation of future-proof data architectures. This in turn allows supporting code for the data pipeline to run close to the database or streaming platforms themselves.

We are so excited by what Apache Flink® and Apache Kafka® Connect can do for the Aiven platform. Expect more news on these trends soon.

To get the latest news about Aiven and our services, plus a bit of extra around all things open source, subscribe to our monthly newsletter! Daily news about Aiven is available on our LinkedIn and Twitter feeds.

If you just want to find out about our service updates, follow our changelog.