There are lots of technologies that when you first read about their capabilities your first reaction is one of an element of disbelief, you think it does that much, for a price like that, sure some marketer is having a laugh! Others you think great, but unless I am a rocket scientist how am I or any other ordinary Joe supposed to use that in the real world. And then very occasionally you come across something that changes all of your preconceived notions, and you think actually the marketers are probably underplaying the capabilities!

Splunk is an oddly named, but amazingly capable, example of one of those technologies that have the capacity, to shock, amaze and bring a big grin of satisfaction to your face. So just look at some of what it can do. You can download a starter, which is size limited but still highly functional, and capable of doing something useful, for free, is 100% true. That you only pay to index once, and that licensing is only for the scale of the data you index, and not for the numbers of users, the number of queries, the number of servers used etc, is 100% true. That it can index machine datafiles in minutes and not hours, is 100% true. That is can index terabytes of data every day, is 100% true. That is runs on commodity priced servers is 100% true. That its query language allows natural line of thoughts analysis, allowing you to dive down through the data (to Splunk !) to find essential truths to questions you may not have previously realised you needed to ask, let alone that you have the data to answer them, is 100% true.

So what is Splunk? We live in an era of exploding data. We attach sensors to everything, monitor and generate facts left, right and centre. This explosion of data covers everything that goes on within an enterprise, from online user interactions, to transactions, to system generated outcomes, it covers normal line of business activity, security threats, fraudulent activity and everything in-between. This is what Big Data is all about, large volume, extensive variety, volatility and relentless updates. Yet within that challenging mass of data lie the answers to all that C level executives, and all who make critical decisions, need to know in order to understand and make better decisions.

Splunk takes all of that machine generated data, gathers it together, holds it in flat time series files without any predetermined structure to influence its analysis by preconceived ideas; it indexes the data at collection time in a process similar to the technology search engines indexing the web, so it is fast and scales, and makes the results available rapidly making real time analysis a feasible idea. The data is then made available for analysis using tools that are considerably less intimidating than the data mining tools that used to be the basis of large-scale analysis.

Splunk is an enterprise level search capability. It collects, it indexes, monitors and it exposes the hidden detail. It is already in use by many of the usual suspects, the Telco's with their mass of CDR data, the Financial Services companies with their complex nexuses of transactional and behavioural data, and that growing element of a modern economy the online gaming companies with their plethora of complex interrelated customer actions. It can tackle all machine-generated data, regardless of volatility and scale, so it would be wrong and too limiting to attempt to categorise and pigeon hole its capabilities, but to give a flavour of what it can do typical uses include application management, system and network security, system compliance, web intelligence, customer behaviour analysis, and revenue assurance. So people are finding that regardless of volume and complexity Splunk has the capability to roll back the layers and expose to view the critical facts. This can cut across the siloed nature of modern business, it allows the inertia and opacity of modern companies to be ignored, providing insight and the opportunity to exert control where previously all seemed to be confusion and impossible scale and volatility.

So in a world of Big data, where the established vendors seem to be trying to use fear, uncertainty and doubt to encourage companies to handover increasing amounts of their vital infrastructure to a monopoly supplier, leaving the business feeling increasingly isolated and with no one to turn to Splunk can cut through the corporate inertia and offer a solution that out of the box can provide a working solution that can tackle just about any data problem. Splunk is supported not just by the company but also by a growing, energetic and enthusiastic community of proponents who are sharing their capability to tackle these big issues head on. As you can download Splunk for free, why not try it and see what it can do? For all insomniacs, whose list of things that keep them awake at night just seems to grow and grow, I think Splunk could be the answer.