What is LogStash How does it work

Central logging in distributed systems with ELK

The architecture of software services has evolved from monoliths to microservices in the last few years or decades. Micro services are easily scalable and can therefore be used very flexibly. For operation in production, this development opens up many options for adapting the services to the required load.

Monitoring is an important part of systems in production, especially finding errors in log files. While searching the log files in a single-system service, in which there are only one or possibly several chronologically divided log files, is still quite easy, things are very different with distributed microservices. An incoming request is forwarded from one service to the next, so troubleshooting can be difficult and take a lot of time.

The solution to this problem lies in central logging, i.e. the log files are collected and processed at a central point. There are countless frameworks that address exactly this problem. Splunk, Loggy, Graylog or the ELK stack are just a few examples. We want to take a closer look at the ELK stack here, as it is becoming increasingly popular and the technologies can be used free of charge.


The ELK stack basic structure

ELK stands for the technologies on which the logging framework is based: E.lasticsearch, L.ogstash and Kibana. Logstash collects the log files from various sources, can filter them, change them and pass them on to several databases. One possibility is the Logstash, which receives log entries via TCP or UDP. In the ELK stack, the log entries from Logstash are passed on to Elasticsearch and saved. Elasticsearch is a document-based NoSql database.

Elasticsearch is based on Apache Lucene and is extremely efficient when it comes to searching for texts and is therefore perfect for log file entries. Kibana is a graphical user interface that reads and displays the log entries from Elasticsearch. The aggregated log data can be visualized graphically with diagrams and summarized in a dashboard

Fig. 1 ELK-Stack basic structure: each service sends the log entries to Logstash via TCP or UDP


The ELK stack can also be built a little differently. Logstash can not only react to TCP or UPD calls and receive the log entries in this way, but can also read them directly from a log file. In this setup, Logstash must be installed for each service and one or more files are defined as input. Logstash monitors the specified log files, reacts to changes in the file and sends new entries to Elasticsearch.


A logstash per service

This setup has the disadvantage that Logstash has to be deployed on every host system, but it also has some advantages. If the network connection fails, the log entries are not lost, but continue to be saved on the service. In order to achieve this in the first setup, the log entries must also be written to a file. The application is also relieved because it only has to log into a file and no longer has to worry about sending the log entries. Last but not least, application-dependent log files can also be imported into the central logging, such as IIS or system logs of the operating system.

Fig. 2 A Logstash per service which directly monitors the log file and sends changes to Elasticsearch

Beats instead of Logstash

After Logstash is rather heavyweight, the Beats family was developed. Filebeat, Metricbeat or Heartbeat are just a few members of this family. Metricbeat, for example, collects metrics about the host system or Heartbeat can determine whether an application is still working properly on the host system. Filebeat does the same as Logstash with input type: File, but is significantly lighter than Logstash and requires fewer resources on the host system. Filebeat doesn't have as many filters as Logstash for this. A common setup would therefore be to send the filebeat to a central Logstash installation and filter it there.

Fig. 3 Beats instead of Logstash, Filebeats can also be used to read the log entries from the log file and send them to Logstash


Each of these setups has certain advantages and disadvantages. Which one should be used has to be decided depending on the application.