Analysis

The impact of MDM on data warehousing

Philip Howard By: Philip Howard, Research Director - Data Management, Bloor Research
Published: 25th April 2008
Copyright Bloor Research © 2008
 Logo for Bloor Research

There hasn't been a lot of discussion about the impact of MDM (master data management) on data warehousing. But there should be.

What has been discussed is where to house MDM implementations. The argument here is that if you need to synchronise operational applications such as CRM systems then what you are doing most of the time is to create, read and update records, which are essentially transactional activities that are best hosted on an OLTP database rather than in a data warehouse. Conversely, if all you want to do is support analytics and business intelligence as opposed to operational applications, then the warehouse would be fine. If you want to do both then it is probably better (unless you are Teradata or Kalido, in which case you have an axe to grind) to host the system outside the warehouse and then federate the master data with the warehouse by means of an EII (enterprise information integration) platform.

However, this doesn't really impact on the data warehouse per se. What does is a consideration of MDM as the "system of record". Traditionally, this has been one of the ideas behind an enterprise data warehouse (EDW): that it represented the system of record and that data could safely be extracted from there to populate data marts against which you could run various forms of analytics. However, when MDM is implemented then that becomes the system of record instead of the EDW. Even if that is held within the warehouse the system of record is still logically within the MDM system and not the EDW.

So, what is the implication of this for the EDW? Put simply: you don't need one. Being the system of record was the raison d'être for the historic concept of the EDW though there is a movement in favour of what is known as EDW 2.0 (ugh!). However, that's another story.

So, if you don't need an EDW how do you do data warehousing? And the simple answer to this question is that you have federated data marts with a virtual platform, such as that of Composite Software, to link the data marts together for making cross-data mart queries. And, for that matter, queries that span operational systems and data marts. Nor is this pie in the sky. I know a couple of major organisations that are doing, or planning to do, exactly this.

But, you may ask, how does this environment link to the MDM system of record? And the answer is again simple: you use the same federated platform for linking between the MDM system and the warehouse environment.

Funnily enough I have run across a number of companies recently that have federated MDM deployments (usually either by domain or geography), all of which makes it look as if the federated platform is becoming an essential part of the whole environment. Indeed, it is not difficult to come up with an architecture in which the federated software is the hub for the whole environment.

Further, MDM systems can cater for structural change to data, which means that the warehouse doesn’t require the same level of capability in this regard (for example, slowly changing dimensions). And this doesn’t just apply to analytic MDM providers like Kalido but also to more general-purpose vendors such as Siperian.

There may be many implications for data warehousing arising out of the uptake of MDM but one will be a move away from traditional EDW-based architectures, at least by some companies, while a second will be an upsurge in the market for federated query capabilities.

Reader Comments

Sorry, we are no longer accepting comments on this item. We suggest trying to contact the author directly.

26th April 2008: 'Phil Bailey' said:

Maybe I'm showing my ignorance around the whole idea of 'federated warehousing' here because it's not clear to me is how this model will deal with the impact on the source systems of continuous ad hoc queiries throughout the day, and there's also the question of whether this method effectively limits the amount of history available for querying to that of the contents of the source system.

So surely you'd only be satisfying the 'farmers' with their hot, up-to-date data with the federated approach, and would always end up having a data warehouse anyway, to meet the requirements for warm and cold data...

I think MDM only really belongs in the operational domain. If it's done in the MI domain, then it's less MDM, and more Meta Data Management or Data Cleansing.

Reply to Phil Bailey?

29th April 2008: 'Cliff Longman' said:

I think the idea is not so much that queries would run on the operational systems, more that the data could be queried from the mart environment and the mart environment could be fed directly from the MDM environment rather than having a data warehouse (in its current form) in between.

By way of commenting on Philip's original piece, I think MDM as an addition to data wareouse infrastructure is a really good idea for anyone trying to work out where to start with MDM. Why take on a big risky operational systems project as a first step? why not cut your teeth on a project to establish MDM and data governance around your existing data warehouse(s) where you already have much of the infrastructure in place. You can move on to more ambitious operational MDM projects as a second step after you have ironed out the (much more challenging) organizational and governance issues.

Reply to Cliff Longman?

29th April 2008: 'Phil' said:

I see what you mean Cliff, but I personally think that doing MDM in the MI domain is just asking for trouble as you're effectively adding a whole layer of complexity in your 'back-end systems' that doesn't necessarily relate back to the front end. I think the real problem of integration and MDM etc lie more in the EAI space, although I do agree that it's a lot less daunting and costly to do it at the back end...

I wonder if anyone has ever implemented MDM first of all in the MI domain, and then successfully moved the focus into the operational domain?...

In my mind at least, DW and MI would be a whole lot easier if integration was tackled at the front end, so you have all your systems fully enabled to talk to each other within an appropriate context - but I suppose it's going to take eons for companies to achieve this with the likes of SOA and all it's future spin-offs...

Reply to Phil?

The messages above were all contributed by IT-Director.com readers. Whilst we take care to remove any posts deemed inappropriate, we can take no responsibility for these comments. If you would like a comment removed please contact our editorial team.