Aiven to release M3 as a service

UPDATE: Aiven for M3 is here!

Aiven is working on releasing M3 as a managed, distributed, time-series database-as-a-service.

The first M3 service release, as a beta launch, will contain basic time-series features and functionality, with advanced functionality like cross-cluster replication support to be added incrementally.

Background

With the explosion of time-series applications coinciding with the rise of Iot, it was only a matter of time when a single-threaded model for query processing could no longer massively expand: to operationalize the datasets of companies like Uber, it was necessary to handle up to 45 trillion metrics per day at 2500 queries per second.

Scalable, multi-threaded processing was needed even when the datastores themselves could already be distributed.

This prompted Uber to invent M3 to handle their internal workload. As many of their teams were already using Prometheus, they needed a solution that was compatible with existing dashboards. Highly-available from the get-go, clearly designed for scalability, and featuring efficient Gorilla compression, M3’s creators started with a few guiding principles:

1. Improved reliability and scalability;
2. Cross-data-center queries;
3. Low latency SLA;
4. Tagged metrics as first-class citizens: supports Prometheus’ (and others’) labels.

To accomplish this, let’s have a quick look at how M3 is built.

Architecture

M3 consists of many different kinds of nodes which form a pluggable architecture. The M3 design has multiple components but here are the most important ones:

• M3DB: the core storage model. A distributed-by-node time-series database. M3DB, built in Go language, comes with extensible, scalable storage and reverse-index time-series.

• M3 Aggregator: a metrics-aggregator service, providing stream-based downsampling, built on etcd. Data can be aggregated based on defined criteria.

• M3 Coordinator: a service coordinating reads and writes between systems and maintaining cluster consensus. These can accept multiple formats (e.g. Prometheus, collectd, fluentd and many more).

M3 integrates out of the box with Prometheus, Graphite, and Grafana for metrics and data visualization.

So how is M3 used?

Uses

M3 is the next evolutionary step in time-series databases; just imagine it faster, more reliable and scalable than previous time-series solutions. M3 can be used for almost all alerting and monitoring use-cases. In addition, M3 has been commonly used for tracking business metrics, monitoring network fabric bandwidth, tracking service latency across hundreds of microservices, capacity planning for clusters, and load balancing, to name a few.

Wrapping up

With M3 time series, you’ll have a scalably faster and more reliable solution for your data. We’ll be adding M3 as a service very soon, so remember to follow our blog or changelog RSS feeds; or, follow us on Twitter or LinkedIn to stay up to date.