How to import data into Metric Manager (formerly Predictive Insight) and in what cases does it add the most value?

We at Compose IT have for many years collaborated with IBM and worked with their products. In 2018, we did a deep dive into IBM’s Metric Manager to see how it could complement our other products.

In our last article on Metric Manager, we described what the product can do and how it works. We also mentioned two test cases we have worked on and how to transfer data to Metric Manager. In this article we will go deeper into how to transfer data to Metric Manager. We will also tell you more about the two test cases we mentioned and describe both a case where Metric Manager’s results were invaluable and a case it is not optimized for.

The solution supports transferring data from many different types of systems. There are several out-of-the-box mediation packages from IBM that transfer data from a source to Metric Manager. There are mediation packages for several systems, including Splunk, Nagios, SCOM and ITNM. Data can also be transferred directly from a relational database, CSV files, JSON files and JSON http posts to Metric Manager.

When CSV or JSON is used as the import method, there may be a need to process the data into a suitable format. It may be a selection of fields to be exported to PI or perhaps a field needs to be formatted. The most common processing that takes place before data is imported is to format the timestamp as Metric Manager requires timestamps in the format of microseconds. If there is a processing need then Elastic Stacks Logstash is the option that IBM recommends. Elastic Stack, of which Logstash is a part, is an open source solution that we at Compose IT also have experience with. Since Logstash is open source based, there are several modules for collecting, filtering and exporting data available. IBM has created modules for Logstash  that format the data and write it to a CSV file with readable structure. With the help of Logstash, it is thus possible to format and import data independently of the source’s data format.

When data is imported from a relational database, Metric Manager’s mediation tool is used.
When we imported data for the two test cases mentioned in our previous article, two different import and processing methods were used, and in both test cases production data was used.

In the first test case, we had a known issue with a script intermittently not executing correctly. The problem occurred every night between one and three. To try to find the root cause of the problem, we started collecting and sending performance data from the server running the script as well as information about how the script was executed to Metric Manager. This data was collected, compiled into JSON format, written to Kafka, read from Kafka and imported into Metric Manager in real-time. In this case, a script was used to extract data from Kafka and import it into Metric Manager, but Logstash works just as well. We recommend the combination Kafka -> Logstash as we think Kafka is a very good product and because it is easy to read from Kafka using Logstash.

The idea was that Metric Manager would be able to find relationships between the data values ​​that would help us find the root cause of the problem. However, since the problem already existed and occurred on a regular basis when we started feeding Metric Manager this data, the problem was considered part of normal behavior. Metric Manager only shows correlations between data values ​​when there are two active deviations that relate to each other. Because this problem was considered part of normal behavior, there was no deviation and no relationships were shown. Metric Manager found some other interesting things but nothing related to the known issue. AI products like Metric Manager are in many cases very good, but this is not one of the cases they are optimized for.

In the second test case, we weren’t looking for anything in particular, but were curious to see what Metric Manager would find if it were allowed to analyze irregular non-time series data. To do this, we collected events that had taken place over a period of six months from the event manager Netcool. This data was written to CSV files that were processed using Metric Manager’s mediation tool and imported into Metric Manager. Metric Manager’s analysis of the data values ​​took a few hours, we let the analysis run overnight and the next morning it was ready.

Analyzing non-time series data in Metric Manager can be difficult and in some cases completely useless. Because Metric Manager  needs a large enough data set to be able to find normal behavior and thus find anomalies. If there are too few data points during an interval, the system considers missing data to be normal and each data record received becomes an anomaly. For example. if there are very few events, then no events are considered normal and as soon as an event occurs, it is seen as an anomaly. This can be corrected somewhat by setting the aggregation interval, but it along