700 million, that’s a sizeable number; 2 billion is bigger still. The first is an estimate of the number of items of machine data generated by the commercial transactions undertaken in the average European enterprise during a year. The second figure is the equivalent for a telco. IT-generated machine data is all the background information that gets generated by the systems that drive such transactions; database logs, network flow data, web click stream data and so on.
Such data, enriched with data from other sources, is a potential gold mine. How to use all this data effectively and turning it into operational intelligence is the subject of a new Quocirca research report, Master of Machines. The report shows the extent to which European organisations are turning machine data into operational intelligence.
The numbers involved are big, so processing machine data involves volume. In fact it fits all the 5 Vs definition of big data well. V for volume is as described above; another v is for variety—the range of sources, with their wide variety of formats. If machine data can be used in near real time it gives, v for velocity; it can add lots of v for value to operational decision making. All of which gets an organisation closer to the truth about what is happening behind the scenes on their IT systems; v for veracity—machine data is what it is; you cannot hide from the facts the mining can expose.
Typically, operational intelligence has been used by IT departments to search and investigate what is going on their IT systems; over 80% already use it in this way. More advanced organisations use the data for proactive monitoring of their IT systems, some providing levels of operational visibility that were not possible before. The most advanced are providing real-time business insights derived from machine data.
To provide commercial insight, the most advanced users of operational intelligence are making it available beyond IT management. 85% of businesses provide a view to IT managers, whereas only 62% currently get a view through to board level execs. In both cases, many recognise a need to improve the view provided. 91% of the most advanced users of operational intelligence are providing a board level view compared to just 3% of the least advanced.
Although there is broad agreement around the value of operational intelligence and the need to open it up to a wide range of management, most are relying on tools that are not designed for the job. These include traditional business intelligence tools and spreadsheets; the latter were certainly not designed to process billions of items of data from a multitude of sources. 27% say they are using purpose built tools for processing machine data to provide operational intelligence. The organisations using such tools gather more data in the first place and will find it easier to share it in a meaningful manner across their organisation.
Quocirca’s research was commissioned by Splunk, which provides a software platform for real-time operational Intelligence. Quocirca and Splunk will be discussing the report and its findings at a webinar on April 3rd 2014. Find out more and register here.