How to integrate with Elastic Stack via Logstash

Collect and transform logs with Logstash

A previous post explored how to integrate a Dockerized application with Elastic Stack using the Elastic Logging Plugin and transform the incoming data with Ingest Pipelines.

Ingest pipelines are built-in to Elasticsearch so getting started with them is really easy. However, compared to Logstash their capabilities are more limited.

This short post is about integrating a Spring Boot application with Elastic Stack using Logstash to receive and process the logs.

Ingesting logs with the Elastic stack and Logstash

Start Elastic Stack with a Logstash node

This time I also started from the compose file provided by Elastic. Similarly to the previous post I use a single Elasticsearch node, but this time Logstash is also part of the mix.

version: '3'
 
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - ELASTIC_PASSWORD=changeme
      - xpack.security.enabled=true
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - elastic
  logstash:
    image: docker.elastic.co/logstash/logstash:7.12.0
    networks:
      - elastic
    depends_on:
      - elasticsearch
    ports:
      - 5000:5000
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
  kibana:
    image: docker.elastic.co/kibana/kibana:7.12.0
    container_name: kibana
    environment:
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=changeme
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch
    networks:
      - elastic
volumes:
  elasticsearch-data:
    driver: local
networks:
  elastic:
    driver: bridge

Note the volume mounting the logstash/pipeline folder from the host machine which includes the logstash.conf file:

input {
    tcp {
        port  => "5000"
        codec => json_lines
    }
}
output {
    elasticsearch {
        hosts    => ["http://elasticsearch:9200"]
        index    => "myapplication-%{+YYYY.MM.dd}"
        user     => "elastic"
        password => "changeme"
    }
}

The config defines a Logstash pipeline that accepts JSON documents on port 5000 and persists them into an index called "myapplication-".

The name of the index is important because it determines which Index Template to use when the index is created. Among others, an Index Template defines the data structure to be used and it can cause some trouble if they are not properly aligned.

Elasticsearch has built-in index templates like logs-* to handle the data as data streams. Data streams are a recent additions to the Elastic stack, and Logstash 7.12 has no support for them; it can only work with simple indexes. So, if the index name would be "log-something", it wouldn't work and most likely it would fail with an error similar to what is reported in this forum thread.

Update: since I started writing this post, Logstash 7.13 got released which can be configured to append data to streams.

The bottom line is that you should always make sure to use an index name that does not match any default Index Template, or make sure it matches a default you like to use.

When the stack is running, Index Templates can be managed in the the Stack Management > Index Management menu where you can check the defaults and also create new ones.

Run docker-compose up and access the Kibana UI at http://localhost:5601.

Next, lets create a simple application to push some logs, so there'll be something to see.

Push log entries from a Spring Boot application

By default, Spring Boot uses Logback for logging which can be fine-tuned by creating a logback-spring.xml configuration:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <include resource="org/springframework/boot/logging/logback/defaults.xml"/>
    <include resource="org/springframework/boot/logging/logback/console-appender.xml"/>
 
    <appender name="logstashNetworkAppender" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
        <destination>localhost:5000</destination>
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        </encoder>
        <keepAliveDuration>5 minutes</keepAliveDuration>
    </appender>
 
    <root level="INFO">
        <appender-ref ref="logstashNetworkAppender"/>
        <appender-ref ref="CONSOLE"/>
    </root>
</configuration>

This snippet adds the LogstashTcpSocketAppender which sends the log entries in JSON format to the Logstash node and it also keeps using the default Console Appender.

After starting the application, the logs will be visible in Kibana.

Setup the index in Kibana

In order to observe the logs, a few things need to be configured in Elastic stack once some data is already there.

First, an Index pattern have to be set up under Stack Management > Index patterns. Create one with the pattern "myapplication-*" to match all indexes that are created by the Logstash pipeline.

Then go to the Logs page, and under the Settings submenu, add the same index pattern to the comma separated list for the Log indices.

Once it's done, visit the logs in the Discovery or in the Logs page.

Logs in the Kibana UI

Enrich the data with Logstash

To demonstrate some of the capabilities of Logstash, let's modify the pipeline to do something interesting by adding a filter that creates a new field called message_uppercase by creating a copy of the message field and applying a mutation to it:

input {
    ...
}
filter {
    mutate {
        add_field => { "message_uppercase" => "%{message}" }
    }
    mutate {
        uppercase => [ "message_uppercase" ]
    }
}
output {
    ...
}

After sending some log entries to this pipeline, the new field can be added in the Logs panel showing the results:

Enriched logs in the Kibana UI

Summary

By the end of this post, you have an Elastic Stack running locally with all 3 components: Elasticsearch, Logstash and Kibana, relying on Logstash to accept log entries in the form of JSON documents.

August 24, 2021
In this article