External Elasticsearch logging output now supported by Aiven

Aiven customers now send their service logs to external Elasticsearch services; find out more about it in this post.

04 June 2019
John Hammink
John Hammink RSS Feed
Developer Advocate at Aiven

In April, we announced our syslog integration for sending your logs to an external service and that we'd soon be following it up with another. Well, it's here; we now support sending Aiven service logs to external OpenSearch (or Elasticsearch) services.

With this addition, you now have four options to get your Aiven service logs. You can get started with our latest option by following the steps outlined in our help article; but, let's check out some of the basics before you do.

A look inside...

Sending your logs to an external OpenSearch (or Elasticsearch) service works on the same principle as our other logging integrations which use the rsyslog protocol. This particlar integration works in the following way:

The external OpenSearch integration works by sending the log entries as JSON documents to an OpenSearch index, named as prefix-YYYY-MM-DD.

Fields that are passed to the new index include these fields:

  • HOSTNAME = name of service + name of node
  • MESSAGE = actual log message
  • SESSION_ID = ID of the running session
  • SYSTEMD_UNIT = linux service (the actual systemd service name)
  • TIMESTAMP = time the message was logged

Here's an example message, sent from a PostgreSQL service:

 {
       "_index" : "logs-2019-05-31",
       "_type" : "journal_msg",
       "_id" : "YD40EFzCwZCbzWQxl4z8",  #fake, for demo purposes
       "_score" : 1.0,
       "_source" : {
         "HOSTNAME" : "pg-1",
         "SYSTEMD_UNIT" : "postgresql-11.service",
         "MESSAGE" : "user=,db=,app=,client= LOG:  listening on IPv6 address \"::\", port 22889",
         "timestamp" : "2019-05-31T05:40:46.020508"
       }
     },

Wrapping up

Our latest logging integration brings the number of available logging options to 4. In addition to integrating with external OpenSearch (or Elasticsearch) services, you can:

  1. access the past two days' logs via our console, API, or CLI;
  2. integrate your logging with one of Aiven's own OpenSearch services for unlimited retention right in Aiven;
  3. or use our syslog integration.

As things evolve, feel free to stay up to date with our blog or changelog RSS feeds, or follow us on Twitter or LinkedIn.

announcementsintegrationselasticsearchopensearch

Let‘s connect

Aiven for Apache Kafka, Aiven for Apache Kafka Connect, Aiven for Apache Kafka MirrorMaker 2, Aiven for M3, Aiven for M3 Aggregator, Aiven for Apache Cassandra, Aiven for OpenSearch, Aiven for PostgreSQL, Aiven for MySQL, Aiven for Redis, Aiven for InfluxDB, Aiven for Grafana are trademarks and property of their respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.