Ever wanted to know the breadth of logging and metrics tooling available with easy integration to the Aiven platform and on the platform itself? Look no further!
Celeste Horgan
|RSS FeedDeveloper Educator at Aiven
Subscribe for the latest news and insights on open source, Aiven offerings, and more.
Logging and metrics data, regardless of what your business does digitally, are some of the most prolific data types that any service or application can produce. Logging data is generally about specific events that you care about in your application and metrics data is generally about overall system performance and uptime. The main difference between these types of data is that metrics are usually numerical in nature and are generated at regular intervals, whereas logs typically have some amount of text attached to them and occur at irregular intervals.
The biggest challenge any business faces is knowing when to store and when to analyze, and how those choices affect latency, accuracy and usefulness of your data. In this article, we’ll go through two common techniques and when they make sense to use for you, and what Aiven services and Aiven integrations you can use to help!
There are two interrelated challenges to consider when working with logging and metrics data: the first is storage, and the second is analysis of that data.
This is the path that most businesses take for their data. Because both metrics and logs produce high volumes of data, usually a business decides on a relevant timeframe of ‘hot’ data to store in an easily accessible datastore like a database, and then relegates the rest of the data to cheaper ‘cold’ storage like an Amazon S3 bucket. ‘Hot’ data is most likely to be accessed by users using analytics or monitoring software to begin with, and relevant windows of ‘cold’ data can be retrieved, loaded from cold storage into a database, and displayed for the user if desired, though there will be lag due to the retrieval process.
This is the method preferred by most businesses, because it ensures reliability for all logging and metric data, and provides fast-enough retrieval most of the time, as users are typically analyzing events that happened in the recent past. From a setup perspective, however, it requires more systems in place and is more costly to maintain.
If the application generating metrics and logs is, say, a web frontend, a sample architecture might look like the following:
In most cases, architectures of this variety also include a buffer layer such as Apache Kafka®, to sort and otherwise structure the data in a consistent manner. The same usage metrics that drive analytics data, for example, are often used in event driven architectures to trigger specific interactions, and Kafka offers the ability to have multiple services subscribe to these same metrics.
While it may go against the grain, running analysis on your data before storage is a strategy that makes sense for some organizations.
In this architectural path, you do some basic sorting of the data using something like Apache Kafka or similar, and send that data directly to analytics or monitoring software. This strategy is particularly useful for metrics data and observability events, where the goal is to observe a system in near real time to address things like performance issues as they crop up. From there, the data is either destroyed or (more commonly) moved into cold storage if needed for analysis later.
Analyzing before storing suits observability and financial use cases, and other scenarios where you need to view data points in close to a realtime stream rather than a view composed by a dashboard. However, it does introduce the possibility of data loss, presenting data out of order, and increases your vulnerability to outages.
An architecture might look like the following in this use case, for example with OpenSearch®:
It’s important to note that many analytics suites, such as SuperSet, only function with an intermediary database rather than directly from streaming data, and thus this will not work with more complex systems.
In store-then-analyze use cases, Aiven can act as the reliable landing spot for your data, marshaling it into whatever shape it needs to be in for your analytics and monitoring services. Aiven offers a number of managed data services which can help:
Aiven offers two main products specifically geared at monitoring both Aiven services and external services:
Aiven also integrates with popular external metrics providers, all managed using the Aiven Console.
Effectively managing logging and metrics data is crucial for any business, when done correctly it provides a competitive advantage. The choice between storing first and then analyzing, or analyzing first then storing, depends heavily on the specific needs and priorities of the organization. The former approach offers greater reliability and is suitable for most use cases, the latter excels in real-time observability and time-sensitive applications. Aiven provides a robust ecosystem of tools and integrations to support both strategies with the hands-on expertise to help guide you in choosing the right approach.
From Aiven for OpenSearch for log analysis and Aiven for Clickhouse for metrics, to Aiven for Apache Kafka for data streaming and buffering, Aiven empowers businesses to build and manage their data pipelines effectively. Coupled with Aiven’s integrated monitoring solutions, including Aiven for Metrics (backed by Thanos) and Aiven for Grafana, coupled with integrations with leading external providers provide comprehensive observability and control over both Aiven services and external infrastructure all without breaking the bank. Ultimately, Aiven simplifies the complexities of log and metric management, which allows you to focus on deriving value from your data.
