External Elasticsearch logging output now supported by Aiven

In April, we announced our syslog integration for sending your logs to an external service and that we'd soon be following it up with another. Well, it's here; we now support sending Aiven service logs to external OpenSearch (or Elasticsearch) services.

With this addition, you now have four options to get your Aiven service logs. You can get started with our latest option by following the steps outlined in our help article; but, let's check out some of the basics before you do.

A look inside...

Sending your logs to an external OpenSearch (or Elasticsearch) service works on the same principle as our other logging integrations which use the rsyslog protocol. This particlar integration works in the following way:

The external OpenSearch integration works by sending the log entries as JSON documents to an OpenSearch index, named as prefix-YYYY-MM-DD.

Fields that are passed to the new index include these fields:

• HOSTNAME = name of service + name of node
• MESSAGE = actual log message
• SESSION_ID = ID of the running session
• SYSTEMD_UNIT = linux service (the actual systemd service name)
• TIMESTAMP = time the message was logged

Here's an example message, sent from a PostgreSQL service:

 {
       "_index" : "logs-2019-05-31",
       "_type" : "journal_msg",
       "_id" : "YD40EFzCwZCbzWQxl4z8",  #fake, for demo purposes
       "_score" : 1.0,
       "_source" : {
         "HOSTNAME" : "pg-1",
         "SYSTEMD_UNIT" : "postgresql-11.service",
         "MESSAGE" : "user=,db=,app=,client= LOG:  listening on IPv6 address \"::\", port 22889",
         "timestamp" : "2019-05-31T05:40:46.020508"
       }
     },
Copy to clipboard

Wrapping up

Our latest logging integration brings the number of available logging options to 4. In addition to integrating with external OpenSearch (or Elasticsearch) services, you can:

1. access the past two days' logs via our console, API, or CLI;
2. integrate your logging with one of Aiven's own OpenSearch services for unlimited retention right in Aiven;
3. or use our syslog integration.

Send your Aiven service metrics and logs to AWS CloudWatch with the Aiven Amazon CloudWatch integration.

As things evolve, feel free to stay up to date with our blog or changelog RSS feeds, or follow us on Twitter or LinkedIn.