Technology -> Big Data
By: Robin Bloor, Co-Founder, The Bloor Group
Published: 27th March 2013
Copyright The Bloor Group © 2013
The main thrust of Big Data is analytics. The simple fact is that we never had the possibility of analyzing vary large data heaps until recently becasue of practical factors, like the cost of buying the hardware and networking it, the absence of analytics software to run in parallel across server grids, fast scale-out databases, the lack of cloud deployment options and so on. Most of what was needed has gradually emerged and because of that Big Data is off and running.
The Upside and the Downside
We can look at this in two ways, from the positive side and the negative side. The positive side requires little explanation. Companies can quickly assemble Big Data pools that were previously too expensive or too slow to work with. In ananlyzing them they may discover extremely valuable knowledge that they can apply to good effect. This is the promise of Big Data and everyone knows it.
I'll deal with the downside simply by providing a list:
The world of events
Behind all of this, a fundamental change is occurring in the world of IT. We are moving from the processing of transactions to the processing of events. Most machine generated data, for example, is event data. Transcations are events too, but now they are in the minority. When we move to the "Internet of Things" there will be an explosion of event data way beyond its current volume, which already exceeds transaction data by a wide margin.
Put simply, Big Data is about events. They have become the atoms of data.
Posted: 27th March 2013 | By Philip Howard :
Robin, you raise an interesting point about the distinction between events and transactions (even though the latter is a subset of the former). If we assume that there are some businesses and/or companies where transactions will continue to predominate over events, does this mean that we end up with twin track solutions? So, there's a transaction architecture and an architecture around big data and some companies are in one stream and some in the other? And then there's probably a gradual transition from the transaction-based environment to the big data environment? Which probably means that we need some sort of sensible migration path from one to the other.
Posted: 27th March 2013 | By Robin Bloor :
You ask a good question.
I'm not sure what the eventual outcome will be - it's difficult to predict beyond the "near horizon." As you suggest, there are some businesses that may not need to care about events, or maybe not care much about them, or may even care about them in part of the business but not for the whole business.
This is likely to be the case in many parts of the mid-market. Let's say you manufacture widgets and just two or three companies buy from you. Maybe your existing ERP system which is transaction based, tells you everything you need to know about the business. Your BI will be transactional. No need to change, at least not in the near future.
Alternatively, at the other extreme, think of Google, Linked-in, Yahoo and other such web companies. Of course there are transactions there in the mix, but there are constant tsunami waves of events going on all the time. They have to make sense of them.
I suspect (but I'm not sure) that the former situation is fundamentally database oriented and the current data warehouse architecture is stable and works reasonably well. I suspect that the latter situation is fundamentally data flow oriented and it is not so easy to work out what data heaps to deploy and how to flow data between them.
Data flow orientation suggests a distributed arrangement of databases or as James Kobielus refers to it "logical data warehouse" (see his comment on http://www.it-director.com/technology/big-data/content.php?cid=13743)
So I guess a migration architecture would be to move towards a distributed arrangement. This would be complicated enough if all you were concerned about was data access at a practical latency for applications, but when you add in data cleansing, MDM, governance, and the rest it could get really complex.
I've not come across any technology that makes that easy. Have you?
I'm not even sure many companies see the problem coming at them yet.
The messages above were all contributed by IT-Director.com readers. Whilst we take care to remove any posts deemed inappropriate, we can take no responsibility for these comments. If you would like a comment removed please contact our editorial team.
We automatically stop accepting comments 180 days after a post is published. If you would like to know more about this subject, please contact us and we'll try to help.
Published by: IT Analysis Communications Ltd.
T: +44 (0)190 888 0760 | F: +44 (0)190 888 0761