Elastic
Elastic has become a matter of course for both us at Compose IT and not least for several of our customers. The scalability, the speed, and more of the characteristics of Elastic mean that we naturally pack it in our portfolio. More information about Elastic, and how we at Compose work with Elastic, can be found in the feed below
Compose IT is often faced with new types of challenges that require new ways of thinking and new methods. Our customers’ various businesses with existing, established solutions must be linked together and enriched, and you also want to take advantage of functions that have already been incorporated. We at Compose IT have therefore for a long time used various open source tools to supplement our system solutions with possibilities that are not limited by either old technology or expensive costs. The toolbox that Elastic offers suits us well and is both effective and fun.
Here is a brief introduction to Elastic:
Elastic has products for collection, processing, data storage and visualization. All products are made to work both independently and together. Thanks to the solution being open source, there is a large amount of plug-ins created by Elastic’s large community. This makes it possible to load, process and visualize all kinds of data.
What parts are included in Elastic then and what makes it so good?
Elastic mainly consists of four different building blocks.
Beats
Small agents used for collection.
Logstash
Data processor that transforms and refines data.
Elasticsearch
The database and the heart of an Elastic solution.
Kibana
GUI for visualization.
In addition to these products, there are a number of functions that can be purchased from Elastic, e.g. for generating alarms, user management, machine learning and more. Some of these functions can also be replaced with open source alternatives.
In addition to the fact that all products work very well together, each part has many strengths that mean that they can also be used individually. An example of this is Logstash, which can be advantageously used together with IBM’s AI produkt Predictive Insight.
Elasticsearch, the database, is extremely powerful. In the following two applications, after evaluation, we have concluded that Elasticsearch is the best option:
• Where you want to collect many different types of data, log data, server monitoring, APM etc. in the same place
• Where there is a need for text-based search
Elasticsearch is a document-based database based on the open source project Apache Lucene, which is a high-performance text-based search library. The database can contain both structured and unstructured data, which makes it possible to store information without pre-defining a data structure. By analyzing each field in a document and indexing the fields based on its data type, the data storage is adapted to the storage type that is optimized for that particular data type. The ability to customize the data storage makes it possible to quickly index the documents and return search results incredibly quickly despite large amounts of data. This ability is what makes Elasticsearch the best solution when you want to gather many different types of information in one place to easily correlate different types of information. Elasticsearch is also the best solution when you want to analyze text. Strings of text are indexed into an inverted index that describes exactly where specific words and sentences are found. This means that you do not need to analyze each document at the time of the search because you already know where all the data is and you can present the results quickly despite large amounts of text.
By using Beats and Logstash together, it is possible to collect, enrich and process log files, metrics such as cpu usage, network data and availability status. Messages can be customized to exactly the format you require to find what you’re looking for in Elasticsearch. Depending on the type of data collected, different forms of processing and enrichment can be done. It is also possible to read data from several sources and adapt the processing according to what is written in the messages. It is possible to add information such as geographic information based on IP addresses, enrich messages with data that is already in Elasticsearch, data that is in a MySQL database or integrate with the REST API. There are many functions for normalizing data by e.g. split a log row into multiple messages, customize fields, remove fields or drop entire messages.
There are many different functions for processing, customizing and enriching data, too many to tell about here. But it is possible to load and process all kinds of data from logs to network information using Elastic. Ingest Pipeline is one of the efficient way used to organize data.
Elastic also supports SSL/TLS and user-based security so that all data can be collected, stored and presented securely. Basic security is free, but if you want to use integrations for e.g. LDAP for user management, it is possible to buy for it at a very reasonable cost.
Elastic with all its building blocks is a competitive monitoring tool that is also completely free. We have implemented Elastic for several different purposes, including as a log manager and as a historical database for Netcool Fault Management.
Read more:
Elastic and Instana
Advantages of Using Elastic and Instana This is a question we frequently get from our clients. Elastic and Instana are two powerful tools offering distinct advantages for organizations with mission-critical […]
Elastic Ingest Pipeline
Organize data with Elastic Ingest Pipeline Transforming and processing data before indexing is a very useful feature of Elastic. This can be done using the Ingest Pipeline tool, which makes […]
Elastic APM
Elastic APM, Application Performance Management With the ELK stack (Elasticsearch, Kibana and Logstash), Elastic has become best known for its data storage and analysis tools. In a simple way, you […]
Analyze raw logs
Analyze raw logs, with Elastic it’s easy! What do you do when you have to maintain a system and need to know how the system is doing every day but […]
