External Elasticsearch logging output now supported by Aiven

Aiven customers now send their service logs to external Elasticsearch services; find out more about it in this post.

04 June 2019
John HamminkDeveloper Advocate at Aiven

In April, we announced our syslog integration for sending your logs to an external service and that we'd soon be following it up with another. Well, it's here; we now support sending Aiven service logs to external Elasticsearch services.

With this addition, you now have four options to get your Aiven service logs. You can get started with our latest option by following the steps outlined in our help article; but, let's check out some of the basics before you do.

A look inside...

Sending your logs to an external Elasticsearch service works on the same principle as our other logging integrations which use the rsyslog protocol. This particlar integration works in the following way:

The external Elasticsearch integration works by sending the log entries as JSON documents to an Elasticsearch index, named as prefix-YYYY-MM-DD.

Fields that are passed to the new index include these fields:

  • HOSTNAME = name of service + name of node
  • MESSAGE = actual log message
  • SESSION_ID = ID of the running session
  • SYSTEMD_UNIT = linux service (the actual systemd service name)
  • TIMESTAMP = time the message was logged

Here's an example message, sent from a postgresql service:

       "_index" : "logs-2019-05-31",
       "_type" : "journal_msg",
       "_id" : "YD40EFzCwZCbzWQxl4z8",  #fake, for demo purposes
       "_score" : 1.0,
       "_source" : {
         "HOSTNAME" : "pg-1",
         "SYSTEMD_UNIT" : "postgresql-11.service",
         "MESSAGE" : "user=,db=,app=,client= LOG:  listening on IPv6 address \"::\", port 22889",
         "timestamp" : "2019-05-31T05:40:46.020508"

Wrapping up

Our latest logging integration brings the number of available logging options to 4. In addition to integrating with external Elasticsearch services, you can:

1. access the [past two days' logs via our console, API, or CLI](https://help.aiven.io/en/articles/2357598-viewing-service-logs); 2. integrate your logging with one of [Aiven's own Elasticsearch services](https://help.aiven.io/en/articles/2090163-getting-started-with-elasticsearch-log-integration) for unlimited retention with Elasticsearch's power right in Aiven; 3. or, use our [syslog integration](https://help.aiven.io/en/articles/2933115-remote-syslog-integration).

As things evolve, feel free to stay up to date with our blog, changelog RSS feeds, or follow us on Twitter or LinkedIn.


Start your free 30 day trial

Test the whole platform for 30 days with no ifs, ands, or buts.

Aiven logo

Let‘s connect

Apache Kafka, Apache Kafka Connect, Apache Kafka MirrorMaker 2, M3, M3 Aggregator, Apache Cassandra, Elasticsearch, PostgreSQL, MySQL, Redis, InfluxDB, Grafana are trademarks and property of their respective owners. All product and service names used in this website are for identification purposes only and do not imply endorsement.