Aiven has built a solid observability solution that our customers can also adopt by using the open source building blocks provided by Aiven.
Aiven makes setting up cloud databases so simple anyone can do it. Our set-it-and-forget-it solutions take the pain out of cloud data infrastructure.
"We promise our customers 99.99% uptime. That's a tall order. We need to see what our systems are doing all of the time, and when things go wrong we need targeted information about it in order to fix it quickly."
Mika Eloranta
Head of Technical Operations
We’ve been using this solution (or its direct predecessors) for years because it’s robust and versatile. The building blocks we’ve used to construct our solution are:
These plug into agents collecting metrics and logs, Telegraf™ and Logstash respectively.
Aiven actually started with InfluxDB® as our time series database, but the increasing number of metrics we collected forced us to find something new. On a practical level, the InfluxDB based solution generated duplicate alerts which was annoying.
“M3 was the best heavy-duty option,” says Markus Stenberg, one of Aiven’s Staff Software Engineers. “It’s used by several large companies and it’s horizontally scalable. Its architecture is also simple enough to be robust and comprehensible.”
“It’s not perfect by any means, but we’ve done our best to plug the biggest holes. For example, upstream M3 is network-intensive because it doesn’t compress traffic. We’ve built compression into our own branch, though, and now we’re just waiting for the upstream project to merge our work,” Stenberg says.
Our own use of M3 was what gave us the idea to start offering Aiven for M3 so that you, too, could benefit.
We use logs for multiple purposes. “They’re like the Swiss Army Knife of data collection,” says Eloranta. “Log entries are easy to collect, move and transform using open source components.”
In troubleshooting, it’s helpful that we collect so many logs. You often don’t know what you’ll need before you need it, and having fine-grained data helps us analyse fault situations quickly and accurately.
But Aiven doesn’t just rely on logs for failure detection. Many layers are needed to ensure that alerts are triggered correctly.
Aiven uses heartbeat-based monitoring to complement the telemetry collection. This ensures that if the telemetry-producing systems go down, the receiving systems don’t just sit there waiting for a signal that never comes. Instead, they know they’re supposed to be receiving a ping, so if nothing happens, it triggers an action because something is wrong.
We also keep working on our alert rules all the time to ensure that they cover all possible failure cases. That work will never be finished, because there’s no way for any list of failures to be complete and comprehensive, but we’re making the serious effort anyway.
“Let’s say one of our ops team members notices we’ve run out of file descriptors, for example,” says Eloranta. “This makes them realise it’s a good case to build an alert for. So because we already collect the metrics, it’s easy to create a graph in Grafana to discover the normal variation in file descriptors used. That figure can then become the basis for a new file descriptor use alert.”
When a fault has been detected, the Aiven team is notified by several concurrent and consecutive channels, like phone alerts and Slack messages.
But since Aiven’s system has so many alerts configured, doesn’t the team get a bad case of alert fatigue? Eloranta says no.
“It depends on how robust your system is,” he says. “If you have lots of alerts and your system is on fire every day, of course you’ll stop paying attention to them. That tells you to shift your focus from reactive to proactive and start fixing the system. Aiven has always addressed the root causes first. You can’t avoid alerts altogether in a big system, especially in the cloud, but you can try.”
"M3 was the best heavy-duty option. It’s used by several large companies and it’s horizontally scalable. Its architecture is also simple enough to be robust and comprehensible."
Markus Stenberg
Software Engineer
"We eat our own dogfood."
Mika Eloranta
Head of Technical Operations
