Analyze raw logs, with Elastic it’s easy!

What do you do when you have to maintain a system and need to know how the system is doing every day but there are hundreds of logs that all have different log formats?
This was the challenge that we at Compose IT faced. Using Elastic, we designed a solution that makes log analysis simple.

If you have developed your own system or adapted a COTS system (standard system), it is likely that there are logs with different formats within the same solution. New logs are created when systems are adapted or expanded, and then it is common for the new logs to have a slightly different format than the existing ones. The format may even differ between log levels in the same log, or a log message may be split into multiple lines. The different formats can then cause problems when the logs are to be read in and analyzed at a central point because you have to be able to normalize the log messages and separate them from each other.

When we were faced with the challenge of collecting many different types of logs spread over many servers, we chose to use Elastic. There were several reasons why we chose Elastic. The main reasons were that Elastic’s database (Elasticsearch) has the capacity to very efficiently index and analyze text and that Elastic offers a comprehensive solution with everything from collection to visualization to be able to analyze unformatted logs. In addition, the entire solution is open source and thus free to use.

With Beats and Logstash (parts of Elastic), we collect logs from more than 50 servers spread over four different network instances into the central Elasticsearch database. Logstash normalizes the logs to one and the same format and processes them so that the messages can be analyzed per instance, across all instances and error messages separately. With this solution, it becomes easy to get an overview of the entire system’s status and the status of each individual instance. Elasticsearch makes it possible to search the log content for individual words or entire sentences. The information is displayed graphically and efficiently analyzed using Kibana, the Elastic GUI.

Parts of the information in these logs are sensitive and must therefore be handled securely. Elastic supports SSL/TLS and user-based security, which means that all log information can be collected, stored and presented securely.

In summary, this solution made it possible to easily analyze and find errors in logs that had completely different formats and contained different types of information from different instances.

Without Elastic, it was not possible to analyze the entire content of all logs within a reasonable time. With Elastic, it became possible to analyze all content and find errors within minutes.

With Elastic, log management becomes easy, fast and secure!