By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 3rd February 2012 Copyright Interarbor Solutions © 2012

This guest post comes courtesy of Tony Baer’s OnStrategies blog. Tony is a senior analyst at Ovum.

To date, Big Storage has been locked out of Big Data. It’s been all about direct attached storage for several reasons. First, Advanced SQL players have typically optimized architectures from data structure (using columnar), unique compression algorithms, and liberal usage of caching to juice response over hundreds of terabytes. For the NoSQL side, it’s been about cheap, cheap, cheap along the Internet data center model: have lots of commodity stuff and scale it out. Hadoop was engineered exactly for such an architecture; rather than speed, it was optimized for sheer linear scale.

Over the past year, most of the major platform players have planted their table stakes with Hadoop. Not surprisingly, IT household names are seeking to somehow tame Hadoop and make it safe for the enterprise.

Up ' til now, anybody with armies of the best software engineers that Internet firms could buy could brute force their way to scale out humungous clusters and, if necessary, invent their own technology then share and harvest from the open source community at will. Hardly a suitable scenario for the enterprise mainstream, the common thread behind the diverse strategies of IBM, EMC, Microsoft, and Oracle toward Hadoop has been to, not surprisingly, make Hadoop more approachable.

What’s been conspicuously absent so far was a play from Big Optimized Storage. The conventional wisdom is that SAN or NAS are premium, architected systems whose costs might be prohibitive when you talk petabytes of data.

Similarly, so far there has been a different operating philosophy behind the first generation implementations from the NoSQL world that assumed that parts would fail, and that five nines service levels were overkill. And anyway, the design of Hadoop brute forced the solution: replicate to have three unique copies of the data distributed around the cluster, as hardware is cheap.

As Big Data gains traction in the enterprise, some of it will certainly fit this pattern of something being better than nothing, as the result is unique insights that would not otherwise be possible. For instance, if your running analysis of Facebook or Twitter goes down, it probably won’t take the business with it. But as enterprises adopt Hadoop—and as pioneers stretch Hadoop to new operational use cases such as what Facebook is doing with its messaging system—those concepts of mission-criticality are being revisited.

And so, ever since EMC announced last spring that its Greenplum unit would start supporting and bundling different versions of Hadoop, we’ve been waiting for the other shoe to drop: When would EMC infuse its Big Data play with its core DNA, storage?

Today, EMC announced that its Isilon networked storage system was adding native support for Apache Hadoop’s HDFS file system. There were some interesting nuances to the rollout.

Big vendors feeling their way
It’s interesting to see how IT household names are cautiously navigating their way into unfamiliar territory. EMC becomes the latest, after Oracle and Microsoft, to calibrate their Hadoop strategy in public.

Oracle announced its Big Data appliance last fall before it lined up its Hadoop distribution. Microsoft ditched its Dryad project built around its HPC Server. Now EMC has recalibrated its Hadoop strategy; when it first unveiled its Hadoop strategy last spring, the spotlight was on the MapR proprietary alternatives to the HDFS file system of Apache Hadoop. It’s interesting that vendor initial announcements have either been vague, or have been tweaked as they’ve waded into the market. For EMC’s shift, more about that below.

For EMC, HDFS is the mainstream
MapR’s strategy (and IBM’s along with it, regarding GPFS) has prompted debate and concern in the Hadoop community about commercial vendors forking the technology. As we’ve ranted previously, Hadoop’s growth will be tied, not only to megaplatform vendors that support it, but the third party tools and solutions ecosystem that grows around it.

For such a thing to happen, ISVs and consulting firms need to have a common target to write against, and having forked versions of Hadoop won’t exactly grow large partner communities.

Regarding EMC, the original strategy was two Greenplum Hadoop editions: a Community Edition with a free Apache distro and an Enterprise Edition that bundled MapR, both under the Greenplum HD branding umbrella. At first blush, it looked like EMC was going to earn the bulk of its money from the proprietary side of the Hadoop business.

What’s significant is that the new announcement of Isilon support pertains on to the HDFS open source side. More to the point, EMC is rebranding and subtly repositioning its Greenplum Hadoop offerings: Greenplum HD is the Apache HDFS edition with the optional Isilon support, and Greenplum MR is the MapR version, which is niche targeted towards advanced Hadoop use cases that demand higher performance.

Coming atop recent announcements from Oracle and Microsoft that have come clearly out on the side of OEM’ing Apache rather than anything limited or proprietary, and this amounts to an unqualified endorsement of Apache Hadoop/HDFS as not only the formal, but also the de facto standard.

This reflects emerging conventional wisdom that the enterprise mainstream is leery about lock-in to anything that smells proprietary for technology where they still are in the learning curve. Other forks may emerge, but they will not be at the base file system layer. This leaves IBM and MapR pigeonholed—admittedly, there will be API compatibility, but clearly both are swimming upstream.

Central Storage is newest battleground
As noted earlier, Hadoop’s heritage has been the classic Internet data center scale-out model. The advantage is that, leveraging Hadoop’s highly linear scalability, organizations could expand their clusters quite easily by plucking more commodity server and disk. Pioneers or purists would scoff at the notion of an appliance approach because it was always simply scaling out inexpensive, commodity hardware, rather than paying premiums for big vendor boxes.

In blunt terms, the choice is whether you pay now or pay later. As mentioned before, do-it-yourself compute clusters require sweat equity—you need engineers who know how to design, deploy, and operate them. The flipside is that many, arguably most, corporate IT organizations either lack the skills or the capital. There are various solutions to what might otherwise appear a Hobson’s Choice:

• Go to a cloud service provider that has already created the infrastructure, such as what Microsoft is offering with its Hadoop-on-Azure services;
• Look for a happy, simpler medium such as Amazon’s Elastic MapReduce on its DynamoDB service;
• Subscribe to SaaS providers that offer Hadoop applications (e.g., social network analysis, smart grid as a service) as a service;
• Get a platform and have a systems integrator put it together for you (key to IBM’s BigInsights offering, and applicable to any SI that has a Hadoop practice)
• Go to an appliance or engineered systems approach that puts Hadoop and/or its subsystems in a box, such as with Oracle Big Data Appliance or EMC’s Greenplum DCA. The systems engineering is mostly done for you, but the increments for growing the system can be much larger than simply adding a few x86 servers here or there (Greenplum HD DCA can scale in groups of 4 server modules). Entry or expansion costs are not necessarily cheap, but then again, you have to balance capital cost against labor.
• Surrounding Hadoop infrastructure with solutions. This is not a mutually exclusive strategy; unless you’re Cloudera or Hortonworks, which make their business bundling and supporting the core Apache Hadoop platform, most of the household names will bundle frameworks, algorithms, and eventually solutions that in effect place Hadoop under the hood. For EMC, the strategy is their recent announcement of a Unified Analytics Platform (UAP) that provides collaborative development capabilities for Big Data applications. EMC is (or will be) hardly alone here.

With EMC’s new offering, the scale-up option tackles the next variable: storage. This is the natural progression of a market that will address many constituencies, and where there will be no single silver bullet that applies to all.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 2nd February 2012 Copyright Interarbor Solutions © 2012

Good data management, analytics, and helping to shape the goals of the business are keys to transforming the enterprise through impactful enterprise architecture (EA). That was the theme, from different perspectives, presented by a series of plenary speakers this week at The Open Group Conference in San Francisco.

Jeanne Ross, Director and Principal Research Scientist at MIT's Center for Information System Research, opened Monday's plenary session, telling the attendees that the stakes are high for EA, which needs to show swift success in the new digital economy. Enterprise architects also now need to help their organizations better use new services and instill a "value cycle." [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

Coming from the siloed past in IT, companies are now moving to business service-driven processes across various resources, Ross said. But they need to recognize the forces around consumption of such services, not just the implementation.

Making good data management a priority, a "single source of truth" is also at the heart of making EA valuable, said Ross. Ensuring the quality of data and the speed of data refresh will help enterprise architects rise in performance appreciation more than just about anything else, she said. Ross studies how firms develop competitive advantage through the implementation and reuse of digitized platforms.

She is also the co-author of three books: IT Governance: How Top Performers Manage IT Decision Rights for Superior Results, Enterprise Architecture As Strategy: Creating a Foundation for Business Execution, and IT Savvy: What Top Executives Must Know to Go from Pain to Gain.

I also interviewed Ross on enterprise transformation issues before the conference.

IT-enablement isn't enough, Ross said, because companies typically under-utilize new systems and applications. It's not that we can't build them, she said of systems, but that companies aren't using them to their potential. Architects need to consider this and then market and evangelize solutions.

And EAs need to be more involved with making quality data center stage in their companies. "You don't get good analytics with bad data," Ross said, "The secret to good EA is to put information in every person's hands so they can use data better." And that in turn will help transform the business and spur added innovation using IT systems and good architecture principles.

Most senior executives aren't very good at combining business and technology strategies, Ross said, and she outlined the architect's elevated role in helping their bosses deliver increased business value:

• Help senior execs clarify business goals
• Identify architectural capabilities that can be readily exploited
• Present options and their implications for business goals
• Build capabilities incrementally

She closed out, getting applause from the audience, by predicting, "Some day CIOs are going to report to the enterprise architect, because that's the way it ought to be."

Impressive cost reduction
The second plenary speaker, Celso Guiotoko, Corporate Vice President and CIO of Nissan Motor Co, Ltd., told how business value is at the top of IT principles for Nissan, information as an asset comes next, and then reducing complexity.

Using these principles, Nissan in 2005 developed "BEST" as an IT mid-term plan and significantly improved the efficiency of its information systems. BEST is an acronym for business alignment, EA, selective sourcing, and technology simplifications.

This was followed in 2009 with the development of the "Change" program, which provided the basis for further advances by changing people, technology, and "process." And, in 2011, the next IT mid-term plan "VITESSE" was launched, designed to bring direct profit to the company. VITESSE encompasses value, innovation, technology, simplification, and service excellence. Through the various initiatives, Nissan has reduced IT cost by over 40 percent, going from a cost per user of $1.09 to $0.63.

The transformed enterprise
Andy Mulholland, Global Chief Technology Officer and Corporate Vice President at Capgemini, focused on the transformed enterprise and cloud trends, as well as the effect of new devices and social networking. Forty million tablets and 70 million smartphones are having a huge impact on how workers and consumers expect to work and shop.

The "bring your own device" phenomenon is forcing a change in thinking for enterprises, Mulholland said, as two environments are developing—inside IT and outside IT. Typically back-end activities operate inside the firewall, while front-end people and activities operate outside the firewall, yet people nowadays want to be able to use smartphones and tablets for both personal and work tasks.

This has led to a situation in which workers are increasingly going outside IT to buy services. Mulholland quoted a Gartner prediction that up to 35 percent of IT expenditures will be outside the IT department by 2015. Other industry analysts like IDC have placed the figure higher.

Because of this, IT faces a huge “re-integration project” to bring together the inside and outside services in a rational way, Mulholland said, adding that the transformed enterprise needs to focus on the productivity of people and innovative business models.

I interviewed Mulholland a few weeks ago and we delved even deeper into the cloud duality issues now coming to the fore of enterprise technology issues and planning. I was also intrigued by a Wall Street Journal piece today on how the US faces a new tech boom. It was aligned with much of what Mulholland was saying.

The key to doing this “re-integration project,” according to Mulholland, is governance, and the industry really lacks a good cloud governance model, meaning that many businesses are already in trouble. However, enterprises shouldn't let that get in the way of progress. Mulholland advised, "If business wants something radically different from you, don't try to stop it. Try to understand it and take control of it."

Driving IT transformation
Lauren States, Vice President and Chief Technology Officer, Cloud Computing and Growth Initiatives, IBM, emphasized that transforming the enterprise requires a huge emphasis on analytics, and a successful integration of analytics and IT.

States drew on IBM's decades-long journey of constant transformation, relying on business process excellence, values-based culture, and IT-enablement. This has led to $1.5 billion in IT savings since 2005 as well as avoiding over $20 million in expenses over five years with a private analytics cloud, she said.

According to States, CMOs are overwhelmingly underprepared for the data explosion and recognize the need to invest in and integrate technology and analysis and consider analytics as business differentiators.

CEOs and CIOs are both highly focused on insights, clients, and people skills, States said, feeding into what she called the "new reality," the need to harvest and pass insights and build trusted relationships.

States' takeaway: We're at the beginning of a major change, much like the PC revolution three decades ago. The cloud's sweet spot now, she says, is in bringing new innovation and insights to marketing, sales and customer service.

No need to wait
Speaker Bill Rouse, executive director, Tennenbaum Institute at Georgia Tech, said that many enterprises wait too long to change, with the decision to transform dragging on until the damage is beyond repair. As evidence, he said that in the past 25 years, 1000 companies have dropped from the Fortune 500 list—showing enterprise transformation has high failure rate, and that waiting for the right time change is a risky business plan.

Moreover, for those enterprises seeking transformation, they need to look at the full ecosystem that a business operates in to effectively transform, says Rouse. Business ecosystems are co-creating high-value services, expanding transformation across supply chains, says Rouse. This is an important nee dimension, he added.

Using analytics better to support evidence-based decision making is transformative and should be a priority, says Rouse. And architecture-oriented thinking can be transformative in itself, he said.

Cyber security threats
On the topic of cyber security, plenary speaker Joseph Menn, cyber security correspondent for the Financial Times and author of Fatal System Error: The Hunt for the New Crime Lords Who are Bringing Down the Internet, made it clear that business as usual won't do.

Joe has covered security since 1999 for both the Financial Times and then before that, for the Los Angeles Times. Fatal System Error is his third book, he also wrote All the Rave: The Rise and Fall of Shawn Fanning's Napster. I also recently interviewed him.

"It's in no one's interest to tell us how bad it really is" when it comes to cyber crime and security, said Menn. And the Stuxnet affair is huge as a harbinger of things to come, he said.

As a result, more taxpayer money will be needed for effective government-level defenses against cyber attacks, he suggested. But government intervention won't do the job alone. Increasingly, corporations will need to play more than just defense on attacks, many of which come from Russia and China and from groups that blend state and criminal interests.

Counter attacks may be a strong defense when it comes to cyber risks, and US government may "turn a blind eye", says Menn. We may even see cyber crime bounty hunters that corporations hire on the QT to go after those that attack them, he said.

Meanwhile, IT groups and enterprise architects can play a bigger role. Knowing what you have helps you know when something has been taken, so improve tracking of assets, Menn told them. He also suggusted that companies keep their most critical data offline, and protect their intellectual property by burying it in and among fake data.

Allen Brown, President and CEO of The Open Group, said that more than 400 corporations are now members of The Open Group, showing strong growth over past 12 years since its founding. TOGAF 9 certification rates growing rapidly worldwide, he said.

FACE standard
In other news from The Open Group on Monday, The Future Airborne Capability Environment (FACE) Consortium, announced the official release of the FACE Technical Standard, which provides guidelines for creating a common operating environment to support applications across multiple Department of Defense avionics systems. See my interview on FACE as it was just getting under way.

The standard is designed to enhance the U.S. military aviation community’s ability to address issues of limited software reuse and accelerate and enhance warfighter capabilities, as well as enabling the community to take advantage of new technologies more rapidly and affordably.The FACE technical standard will enable developers to create and deploy a wide catalog of applications for use across the spectrum of military aviation systems through a common operating environment. Product development efforts by industry and procurements by government customer organizations are already underway based on the FACE standard.

“The introduction of the FACE Technical Standard is an important milestone in extending interoperability among the armed forces and creating a common platform for avionics that enables systems to work together across each of the branches of the U.S. military,” said Brown.

And on Tuesday, The Open Group announced the arrival of ArchiMate 2.0, the latest version of the organization's open and independent modeling language for enterprise architecture. This version is more tightly aligned to TOGAF, so enterprise architects using the language can improve the way key business and IT stakeholders collaborate and adapt to change.

ArchiMate 2.0 improves collaboration through clearer understanding across multiple functions, including business executives, enterprise architects, systems analysts, software engineers, business process consultants and infrastructure engineers, according to the release. The new standard enables the creation of fully integrated models of an organization's Enterprise Architecture, the motivation behind it, and the programs, projects and migration paths to implement it.

"By combining TOGAF and ArchiMate, TOGAF becomes more easy to apply in any organization," said Harmen van den Berg, partner and co-founder at BiZZdesign. "Having a reference model makes them both easier to apply in any industry or vertical."

He added: "Architects like to make models, and this now helps them to use those models to create change in the organization, for something that means more to the business."

Making the EA function a chief weapon of enterprise transformation in a time of roiling change and complexity, that's the main message from the conference. No time to wait.

By: Simon Holloway, Practice Leader - Process Management & RFID, Bloor Research Posted: 27th January 2012 Copyright Bloor Research © 2012

When I first wrote the RFID market overview, one of the key things I identified was that RFID hardware couldn't exist without RFID middleware and applications, and neither could RFID middleware and applications exist without RFID hardware. What has also become clear is that no longer are organisations just looking at passive or active tags, what they want is for their RFID middleware and applications to be able to work with a mix of different tags, both active and passive, and even at different frequencies. It is a case of choosing the right horse for the course!

On January 12th, Zebra announced they have entered into a "cooperative relationship and licensing agreement" with Checkpoint Systems. This relationship brings together Zebra's active location solutions with the passive RFID, auto-ID, Wi-Fi and sensor capabilities of Checkpoint division OATSystems' OATxpress middleware. The objective is to provide increased visibility of assets across an enterprise. The agreement is a non-exclusive contract and provides Zebra with an OEM software license for OATxpress

A reminder for those of you who are not sure about the two organisations involved. Zebra is one of the leading suppliers of bar code, receipt, card, kiosk and RFID printers and supplies, as well as real-time location solutions. Over the last year or so they have also developed a real-time location solution (RTLS), WhereNet ISO/IEC 24730-2. This provides robust location performance both indoors and outdoors with a long tag to sensor range. WhereLAN III RTLS tag delivers 1 meter locating accuracy, lower deployment and ownership costs, lower power consumption, and 802.11 b/g Wi-Fi backhaul.

Checkpoint Systems is a leader in shrink management, merchandise visibility, apparel labeling and asset tracking solutions. Checkpoint has some 40 years of experience of RF technology and shrink management requirements. In 2008, Checkpoint Systems acquired one of the leading RFID middleware companies, OATSystems (see OATSystems acquired by Checkpoint). This strengthened their RF capability and RFID customer base and has allowed OATSystems, as a division of Checkpoint, to further develop supply chain, manufacturing and inventory management applications on top of their RFID middleware for a number of verticals ranging from Apparel to Aerospace.

So what we have with this agreement is that Zebra can now offer Checkpoint's OATxpress device and data management capabilities in conjunction with their WhereNet RTLS solution. This makes it easier for a potential customer to purchase a complete solution from one point. From Checkpoint's viewpoint it gives access to Zebra customers and to the Zebra partner network thus providing further global access. From Zebra's viewpoint it can be summed up by a quote from Phil Gerskovich, senior vice president, new growth platforms at Zebra Technologies, "The addition of OAT's passive RFID and other auto-ID technologies capabilities will enable Zebra to play a larger and more meaningful role in helping organizations to make smarter decisions in managing their operations." Zebra has stated that they will announce details around its first product with the capability to implement applications that combine both active and passive RFID in the coming months, so watch this space!

In my view this relationship makes perfect sense to everyone and, most importantly, to potential and existing customers of Zebra and Checkpoint Systems.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 25th January 2012 Copyright Bloor Research © 2012

There have been a couple of things floating around in the ether about CEP (complex event processing) recently. The first is the question, supposedly credited to Curt Monash, of whether it should be called something different.

I've been going back through my records. When I first wrote a product evaluation of what is now Progress Apama in 2002, I stated that "the company's contention is that conventional approaches to real-time queries are only suitable for small scale environments or those in which limited numbers of data feeds are being monitored. In particular, its view is that these solutions cannot cope with environments where large numbers of data feeds need to be combined in a complex and dynamic fashion." That was the only use of the word "complex" in a total of nearly 3,000 words. Events were mentioned several times, streams not at all.

As an aside, look at Wikipedia and other sources about the development of CEP and you'll see lots of mentions of David Luckham, who coined the CEP term in his book "The Power of Events", published in 2001. You will also see various other attributions to American scholarship but no mention at all to Cambridge (that's UK not Harvard), which is where Apama came from. I guess all the writers are American.

Anyway to get back to the subject, I wrote the following in our report on CEP, published in 2006: "the subject under discussion is frequently referred to as either complex event processing (CEP) or as event stream processing (ESP). We believe that both of these names are misleading: the former suggests that the technology is not also suitable for processing simple events, while you could infer from the latter (processing streams) that this was only about high performance. We prefer event processing as a neutral term to cover all of these possibilities." Frankly, I gave up this argument years ago.

The second piece of discussion that has recently hit the blogosphere is from Chris Carlson at Informatica. He is suggesting, quite rightly, that CEP isn't simply about real-time processing and that, in fact, it is misleading to refer to it as such. Again, from our 2006 report, "event processing is suitable for use in a wide range of diverse environments. Some of these are more about event streaming (low latency) and some are more about complex processing and some potentially both" and "what event processing does is to reduce the data latency, insight latency and, sometimes, the decision latency involved in taking action when compared to traditional approaches." In other words, CEP is about processing real-time data - it isn't necessarily about making real-time decisions.

Finally, I want to make a point about big data and CEP. Back before Christmas I wrote about how only StreamBase and IBM, of the major vendors, are currently, as far as I can tell, targeting their products at general-purpose operational intelligence environments as opposed to specific areas such as capital markets and security services or applications environments such as SOA. I did mention that SAS will be bringing out a product later this year and that there is also Darkstar from Cloud Event Processing (CEP - ycch!) which, naturally, runs in the Cloud. What I didn't mention was that there are a number of companies/products that have been specifically designed to work in conjunction with Hadoop, namely HStreaming, S4 (from Yahoo!) and Storm (from Twitter). I haven't looked at any of these in detail so can't comment on them but that is definitely an area that is heating up.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 23rd January 2012 Copyright Bloor Research © 2012

"It was the best of times. It was the worst of times." For Dickens, in a Tale of Two Cities, this was how he saw the era of the French Revolution. I think that we live in a time that could be described as being both the worst of times, and the most exciting of times. We face unprecedented economic challenges; challenges that will only be met by business people making more, faster and better decisions about every aspect of their trade. In response to that challenge we are seeing technologies that are just staggeringly exciting with the potential they offer, and one such is Alteryx.

I see two major trends in the BI and analytics space. The two are closely linked to the market challenge. Firstly we have the emergence of what is being described as Agile BI; for me what Agile BI is about is providing the most sophisticated of analytics power to the business user on their desktop, with tools that do not require the skills of a statistician or an IT technician to understand and master, capable of delivering results that are rapid, reliable and readily understood at an affordable price. Then we have the rise of Big Data, which has been written about many times by my colleagues in Bloor, and is not so much about Big as in massive, as being big as a consequence of the range and diversity of data that should, and now can, be included in the decision making process for all but the most trivial of business decisions. It is in this sweet spot that Alteryx sit.

It should be noted that in the big study that McKinsey undertook into Big Data and its implications that they identified that one of the biggest barriers to successful roll out was the shortage of skilled knowledge workers to fill the management posts to use the information wisely. Unfortunately we do not have the time to set up the University courses to fill that void in the near term, but what we do have is products like Alteryx.

Alteryx talk a lot about Data Artisans: by this they mean the people in the business who are tasked with improving marketing responses, improving merchandising, deciding where to locate the next store, reducing churn etc. This is their day job and they do not want an IT project to give them a one-off solution, they need a way to create an ongoing and evolving answer to their needs. Those needs are complex, but they do not want a dumbed down solution, they want one that retains integrity, power and reliability, but which allows them to merge transactional data with operational data, with demographic data, with location intelligence, with social media trends, with market data etc etc. to make an informed decision.

By desktop to cloud they are saying that the whole process of identifying, gathering and loading data - then analysing that data to turn data into information - must be made available on the desktop, with the results then being broadcast via the cloud, be that public or private.

From such descriptions it would be easy to dismiss this as just marketing. But this is not an immature product, it has been going for several years in North America, and is has a very prestigious looking client list with household names like Wal-Mart, and Ford and even Apple. It has a number of vertical apps for groups like Telcos and Retailers that have prebuilt a lot of the functions that people need to do their everyday jobs. These are clearly very successful as 7 out of the 10 North American mobile operators have it deployed because it tackles things like churn, integrating operational data off the network with customer data coming in from the call centre.

As it is sold as a subscription service, these are not just one-off evaluation purchases but renewed deployments that are significant investments by the users. And although they do not a have a direct sales force in EMEA at present (it is planned) they have a channel partner, which is Experian, and Experian themselves use the product as a delivery channel for their data. If you go onto the web site you can see many endorsements by their customers, which indicates how happy they have been with the product.

Because this product essentially has the whole BI stack deployed in the desktop there is a danger that IT people will see it as a threat to the traditional tools, and try to discourage its use. This would be wrong. I do not see this as replacing the existing BI technologies, but it is highly complementary, and it does point to the way that all tools must go in the future. Alteryx clearly do not see their platform as an all-encompassing solution, because they are planning to build connectors to provide Hadoop support via ODBC drivers to Hive, giving them read/write into Hadoop clusters. This illustrates the extremely complex nature of full enterprise BI solutions, as they are now emerging from the era when we saw a single vendor suite on top of a data warehouse as the solution.

I am extremely impressed by what I have seen and will be following the progress of Alteryx with keen interest, and expect to see it putting established analytics players, like IBM and SAS, under pressure to improve their usability and responsiveness in the hands of people outside of the Analytics teams. This is another product to watch in 2012. it should make giant strides.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 13th January 2012 Copyright Interarbor Solutions © 2012

This guest post comes courtesy of Tony Baer’s OnStrategies blog. Tony is a senior analyst at Ovum.

When we last left Oracle’s big data plans, there was definitely a missing piece. Oracle’s Big Data Appliance, as initially disclosed at last fall’s OpenWorld, was a vague plan that appeared to be positioned primarily as an appliance that would accompany and feed data to Exadata. Oracle did specify some utilities, such as an enterprise version of the open source R statistical processing program that was designed for multithreaded execution, plus a distribution of a NoSQL database based on Oracle’s BerkeleyDB as an alternative to Apache Hive. But the emphasis appeared to be extraction and transformation of data for Exadata via Oracle’s own utilities that were optimized for its platform.

As such, Oracle’s plan for Hadoop was competition, not for Cloudera (or Hortonworks), which featured a full Apache Hadoop platform, but EMC, which offered a comparable, appliance-based strategy that pairs Hadoop with an Advanced SQL data store; and IBM, which took a different approach by emphasizing Hadoop as an analytics platform destination enhanced with text and predictive analytics engines, and other features such as unique query languages and file systems.

Oracle’s initial Hadoop blueprint lacked explicit support of many pieces of the Hadoop stack such as HBase, Hive, Pig, Zookeeper, and Avro. No more. With Oracle’s announcement of general availability of the big data appliance, it is filling in the blanks by disclosing that it is OEM’ing Cloudera’s CDH Hadoop distribution and, more importantly, the management tooling that is key to its revenue stream. For Oracle, OEM’ing Cloudera’s Hadoop offering fully fleshes out its Hadoop distribution and positions it as a full-fledged analytic platform in its own right; for Cloudera, the deal is a coup that will help establish its distribution as the reference. It is fully consistent with Cloudera’s goal to become the Red Hat of Hadoop as it does not aspire to spread its footprint into applications or frameworks.

Question of acquisition
Of course, whenever you put Oracle in the same sentence as OEM deal, the question of acquisition inevitably pops up. There are several reasons why an Oracle acquisition of Cloudera is unlikely.

1. Little upside for Oracle. While Oracle likes to assert maximum control of the stack, from software to hardware, its foray into productizing its own support for Red Hat Enterprise Linux has been strictly defensive; its offering has not weakened Red Hat.
   
   
2. Scant leverage. Compare Hadoop to MySQL and you have a Tale of Two Open Source projects. One is hosted and controlled by Apache, the other is hosted and controlled by Oracle. As a result, while Oracle can change licensing terms for MySQL, which it owns, it has no such control over Hadoop. Were Oracle to buy Cloudera, another provider could easily move in to fill the vacuum. The same would happen to Cloudera if, as a prelude to such a deal, it began forking from the Apache project with its own proprietary adds-ons or substitutions.

OEM deals are a major stage of building the market. Cloudera has used its first mover advantage with Hadoop well with deals with Dell, and now Oracle. Microsoft, in turn, has decided to keep the “competition” honest by signing up Hortonworks to (eventually) deliver the Hadoop engine for Azure.

OEM deals are important for attaining another key goal in developing the Hadoop market: defining the core stack – as we’ve ranted about previously. Just as Linux took off once a robust kernel was defined, the script will be identical for Hadoop. With IBM and EMC/MapR forking the Apache stack at the core file system level, and with niche providers like Hadapt offering replacement for HBase and Hive, there is growing variability in the Hadoop stack. However, to develop the third party ecosystem that will be vital to the development of Hadoop, a common target (and APIs for where the forks occur) must emerge. A year from now, the outlines of the market’s decision on what makes Hadoop Hadoop will become clear.

The final piece of the trifecta will be commitments from the Accentures and Deloittes of the world to develop practices based on specific Hadoop platforms. For now they are still keeping their cards close to their vests.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 19th December 2011 Copyright Interarbor Solutions © 2011

The latest BriefingsDirect discussion targets the need to make sense of the deluge and complexity of the data and information that is swirling in and around modern enterprises. Most large organizations today are able to identify, classify, and exploit only a small portion of the total data and information within their systems and processes.

Perhaps half of those enterprises actually have a strategy for improving on this dismal fact. But business leaders are now recognizing that managing and exploiting information is a core business competency that will increasingly determine their overall success. That means broader solutions to data distress are being called for.

This discussion then examines how metadata-driven data virtualization and improved orchestration can help provide the inclusion and scale to accomplish far better data management. Such access then leads to improved integration of all information into an approachable resource for actionable business activities.

With us now to help better understand these issues—and the market for solutions to these problems—are Noel Yuhanna, Principal Analyst at Forrester Research, and Todd Brinegar, Senior Vice President for Sales and Marketing at Stone Bond Technologies. The panel is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: Stone Bond is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: It’s hard to overstate that the size and rate of growth of data and information is just overwhelming the business world. Why is it a critical stage now to change how we're addressing these issues?

Yuhanna: We have customers who have 55,000 databases, and they plan to double this in the next three to four years. Imagine trying to manage 55,000 databases. It’s a nightmare. In fact, they don’t even know what the count is actually.

The data has been growing significantly over the last few years because of different application deployments, different devices, such as mobile devices, and different environments, such as globalization. These are obviously creating a bigger need for integration.

Then, they're dealing with unstructured data, which is more than 75 percent of the data. It’s a huge challenge trying to manage this unstructured data. Forget about the intrusions and the hackers trying to break in. You can’t even manage that data.

Then, obviously, we have challenges of heterogeneous data sources, structured, unstructured, semi-structured. Then, we have different database types, and then, data is obviously duplicated quite a lot as well. These are definitely bigger challenges than we've ever seen.

Gardner: We're not just dealing with an increase in data, but we have all these different data sources. We're still dealing with mainframes.

It seems to me that you can’t just deal with big data. You have to deal with the right data. What’s the difference between big data and right data?

Yuhanna: It’s like GIGO, Garbage In, Garbage Out. A lot of times, organizations that deal with data don’t know what data they're dealing with. They don’t know that it’s valuable data in the organization. The big challenge is how to deal with this data.

The other thing is making business sense of this data. That's a very important point. And right data is important. I know a lot of organizations think, "Well, we have big data, but then we want to just aggregate the data and generate reports." But are these reports valuable? Fifty percent of times they're not, and they've just burned away 1,000 CPU cycles for this big data.

That's where there's a huge opportunity for organizations that are dealing with such big data. First of all, you need to understand what this big data means, and ask are you going to be utilizing it. Throwing something into the big data framework is useless and pointless, unless you know the data.

Brinegar: Noel is 100 percent correct, and it is all about the right data, not just a lot of data. It’s interesting. We have clients that have a multiplicity of databases. Some they don’t even know about or no longer use, but there is relevant data in there.

When you were talking about the ability to attach to mainframes, all legacy systems, as well as incorporated into today’s environments, that's really a big challenge for a lot of integration solutions and a lot of companies.

So the ability to come in, attach, and get the right data and make that data actionable and make it matter to a company is really key and critical today. And being able to do that with the lowest cost of ownership in the market and the highest time to value equation—so that the companies aren’t creating a huge amount of tech on top of the tech that they already have to get at this right data—that’s really the key critical part.

Gardner: What’s with this notion about orchestrating, metadata, and data virtualization? Why are some of these architectural approaches being sought out, especially for real-time uses?

Yuhanna: You have to look at the holistic data set. Today, most organizations or business users want to look at the complete data sets in terms of how to make business decisions. Typically, what they're seeing is that data has always been in silos, in different repositories, and different data segregations. They did try to bring this all together like in a warehouse trying to deliver this value.

But then the volumes of data, the real-time data needs are definitely a big challenge. Warehouses weren't meant to be real-time. They were able to handle data, but not in real time.

So this whole data segregation delivers a yet even better superior framework to deliver real-time data and the right data to consumers, to processes, to applications, whether it’s structured data, semi-structured, unstructured data, all coming together from different sources—not only on-premise, also off-premise, such as partner's data and marketplace data coming together and providing that framework toward different elements.

We talked about this many years ago and called it the information fabric, which is basically data virtualization that delivers this whole segregation of data in that layer, so that it could be consumed by different applications as a service, and this is all delivered in a real-time manner.

Now, an important point here is that it's not just read-only, but you can also write back through this virtualized layer, so that it can get back at the data.

Definitely, things have changed with this new framework and there are solutions out there that offer this whole framework, not only just accessing data and integrating data, but they also have frameworks, which includes metadata, security, integration, transformation.

Gardner: For the companies that you work with at Forrester, when they do this correctly, what sort of benefits are they able to gain?

Yuhanna: The good thing about data virtualization is that it's not just a single benefit. There are many, many benefits of data virtualization, and there are customers who are doing real-time business intelligence (BI), business with data virtualization. As I mentioned, there are drawbacks and limitations in some of the older approaches, technologies, and architectures we've used for decades.

We want real-time BI, in the sense that you can’t just wait a day for this report to show up. You need this every hour or every minute. So these are important decisions you've got to make for that.

Real-time BI is definitely one of the big drivers for data virtualization, but also having a single version of the truth. As you know, more than 30 percent of data is duplicated in an organization. That’s a very conservative number. Many people don’t know how much data is duplicated.

And you have different duplication of data—customer data, product data, or internal data. There are many different types of data that is duplicated. Then the data has a quality issue, because you may change customer data in one of the applications that may touch one database, but the other database is not synchronized as such. What you get is inconsistent data, and customers and other business users don’t really value the data actually anymore.

A single version of the truth is a very important deliverable from solutions, which has never been done before, unless you have one single database actually, but most organizations have multiple databases.

Also it's creating this whole dashboard. You want to get data from different sources, be able to present business value to the consumers, to the business users, what have you, and the other cases like enterprise search, you're able to search data very quickly.

Imagine if an auditor walks into an organization, they want to look at data for a particular event, or an activity, or a customer, searching across a thousand resources. It could be a nightmare. The compliance initiative through data virtualization becomes a lot simpler.

Then, you're doing things like content-management applications, which need to be delivered in federation and integrate data from many sources to present more valuable information. Also, smart phones and mobile devices want data from different systems so that they all tie together to their consumers, to the business users, effectively.

So data virtualization has quite a strong value proposition and, typically, organizations get the return on investment (ROI) within six months or less with data virtualization.

Brinegar: This is exactly the fabric and the framework that Enterprise Enabler, Stone Bond’s integration technology, is built on.

What we've done is look at it from a different approach than traditional integration. Instead of taking old technologies and modifying those technologies linearly to effect an integration and bring that data into a staging database and then do a transformation and then massage it, we've looked at it three-dimensionally.

We attach with our AppComms, which are our connectors, to the metadata layer of an application. We don’t agent within the application. We get at the data of the data. We separate that data from multiple sources, unlimited sources, and orchestrate that to a view that a client has. It could be Salesforce.com, SharePoint, a portal, Excel spreadsheets, or anything that they're used to consuming that data in.

Gardner: Just to be clear, Todd, your architecture and solution approach is not only for access for analysis, for BI, for dashboards and insights—but this is also for real-time running application sets. This is actionable data?

Brinegar: Absolutely. With Enterprise Enabler, we're not only a data-integration tool, we're an applications-integration tool. So we are EAI/ETL. We cover that full spectrum of integration. And as you said, it is the real-time solution, the ability to access and act on that information in real time.

Enterprise Enabler provides the ability to virtualize, federate, orchestrate, all in real-time and is a huge value. The biggest thing is time to value though. How quickly can they get the software configured and operational within their enterprise? That is really the key that is driving a lot of our clients’ actions.

When we do an installation, a client can be up and operational doing their first integration transformations within the first day. That’s a huge time-to-value benefit for that client. Then, they can be fully operational with complex integration in under three weeks. That's really astounding in the marketplace.

I have one client that on one single project calculated $1.5 million cost savings in personnel in the first year. That’s not even taking into account a technology that they may be displacing by putting in Enterprise Enabler. Those are huge components.

HP is a great example. HP runs Enterprise Enabler in their supply chain for their Enterprise Server Group. That group provides data to all the suppliers within the Enterprise Server Group on an on-time basis.

They are able to build on demand and take care of their financials in the manufacturing of the servers much more efficiently than they ever have. They were experiencing, I believe, a 10-times return on investment within the first year. That’s a huge cost benefit for that organization. It's really kept them a great client of ours.

We do quite a bit of work in the oil business and the oil-field services business, and each one of our clients has experienced a faster ROI and a lower total cost of ownership (TCO).

We just announced recently that most of our clients experienced a 300 percent ROI in the first year that they implemented Enterprise Enabler. CenterPoint Energy is a large client of Stone Bond and they use us for their strategic transformation of how they're handling their data.

Gardner: Let’s go back to Noel. Do you have a sense of where companies that are successful at doing this have begun?

Yuhanna: One is taking an issue, like an application-specific strategy, and building blocks on that, or maybe just going out and looking at an enterprise-wide strategy. For the enterprise-wide strategy, I know that some of the large organizations in the financial services, retail, and sales force are starting to embark on looking at all of these data in a more holistic manner:

"I've got customer data that is all over the place. I need to make it more consistent. I need to make it more real-time." Those are the things that I'm dealing with, and I think those are going to be seen more in the coming years.

Obviously, you can’t boil the ocean, but I think you want to start with some data which becomes more valuable, and this comes back to the point that you talked about as the right data. Start with the right data and look at those data points that are being shared and consumed by many users, business users, and that’s going to be valuable for the business itself.

The important thing is also that you're building this block on the solution. You can definitely leverage some existing technologies, if you wanted to. I would definitely recommend now looking at newer technologies, because they definitely are faster. They do a lot of caching. They do a lot of faster integration.

As Todd was mentioning, quicker ROI is important. You don’t have to wait for a year trying to integrate data. So I think those are critical for organizations going forward. But you also have to look at security, availability, and performance. All of these are critical, when you're making decisions about what your architecture is going look like.

We've actually done extensive research over the last four or five years on this topic. If you look at Information Fabric, this is a reference architecture we've told customers to use when you're building a data virtualization yourself. You can build the data virtualization yourself, but obviously it will take a couple of years to build. It’s a bit complex to build, and I think that's why solutions are better at that.

But Information Fabric reports are there. Also, information as a service is something that we've written about—best practices, use cases, and also vendor solutions around this topic of discussion. So information as a service is something that customers could look at and gain understanding.

We have use cases or case studies that talk about the different types of deployments, whether it’s a real-time BI implementations or doing single version of fraud detection, or any other different types of environments they're doing. So we definitely have case studies as well.

There are case studies, reference architectures, and even product surveys, which talk about all of these technologies and solutions.

Gardner: Todd, how about at Stone Bond? Do you have some white papers or research, reports that you can point to in order to help people sort through this and perhaps get a better sense of where your technologies are relevant and what your value is?

Brinegar: We do. On our website, stonebond.com, we have our CTO, Pamela Szabó's, blog, which has a great perspective of data, big data, and the changing face of data usage and virtualization.

I wish everybody would explore the different opportunities and the different technologies that there are for integration and really determine not what you need today—that’s important—but what will you need tomorrow. What’s the tech that you're going to carry forward, and how much is the TCO going to be as you move forward, and really make that value decision past that one specific project, because you're going to live with the solution for a long time.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 19th December 2011 Copyright Bloor Research © 2011

It's that time of year. Here are a few thoughts about what's to come, in no particular order.

1. Real-time everything. Hardly a surprise. It's been real-time everything for a few years. What I think is interesting is the growth in the data replication market specifically to support real-time BI as opposed to failover, disaster recovery, zero-downtime migrations and the like. I would not be at all surprised if we see the introduction of lightweight BI-only data replication products into the marketplace.
2. Continuous BI. I think we'll hear a lot more about this as a generic market for complex event processing as opposed to the vertical markets that CEP has previously addressed.
3. CEP adoption by SIEM vendors. I have been arguing for the last two years that SIEM architectures are generally antiquated. The big breakthrough will come if (I think it more likely to be when) IBM announces that InfoSphere Streams has been integrated with QRadar. Now you have a big beast on the one hand and smaller, more agile companies like Red Lambda and Tier-3 all offering CEP in this space and the other suppliers will have to follow suit or a) appear out-of-date and slow (which they are) or b) limit themselves to the SME market.
4. Warehousing adoption by SIEM vendors. This is the other thing I have been calling for. How can you claim to offer analytics against security and log data if you don't have an analytic platform to support it? Anyway, what's the betting that IBM links up Netezza with QRadar (and, for that matter Guardium)? If they do, there could be a scramble to catch up by the other suppliers, which will be good news for other data warehousing companies and also for Sensage, which is the only company in the SIEM space that actually seems to understand how important this is and has done something about it.
5. Growth in PMML adoption. Talk to more or less anyone outside the data mining community and they have hardly ever heard of the Predictive Modelling Mark-up Language, the standard for porting data mining models. Well, bear in mind that InfoSphere Streams supports PMML and that whenever SAS comes out with its CEP product it is highly likely to support PMML and you have a situation where leading players in both the continuous BI and SIEM spaces are supporting this standard. Isn't it likely that others will have to follow suit?
6. Lots more big data. Well, of course. Unfortunately, I don't expect to see any more clarity during 2012. Indeed, the reverse. As more products and companies enter this space, or claim to, the more murky the whole big data thing will become.
7. The emergence of the Data Scientist. This is a new class of information worker that appears to have emerged recently. Back in the 90s when data mining first started to become popular the people who found useful needles in haystacks of information were usually data mining specialists. With Hadoop you do not need data mining skills in quite the same way but you have similar tasks to perform. That's what the data scientist does: in effect, determining what the business should be monitoring.
8. The logical data warehouse. I wrote about the death of the traditional EDW as the home for all things great and good some time ago and the logical data warehouse, at least as a concept, seems to me pretty much a done deal now. The complication is that vendors like Sybase (SAP) can support a logical EDW within a single physical installation but that's an implementation issue rather than a conceptual one.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 15th December 2011 Copyright Interarbor Solutions © 2011

You don’t need to go very far in IT nowadays to find people who are diligently working to do more with less, even as they're working to transform and modernize their environments.

One way to keep the interest high—and those operating and investment budgets in place—is to show fast results, and then use that to prime the pump for even more improvement—and even more funding—with perhaps even growing budgets.

The latest BriefingsDirect discussion then explores how to build quick data center project wins, by leveraging project tracking and scorecards, as well as by developing a common roadmap for both facilities and IT infrastructure.

We'll hear from a panel of HP experts on some of their most effective methods for fostering consolidation and standardization across critical IT tasks and management. This is the second in a series of podcasts on data center transformation (DCT) best practices and is presented in conjunction with a complementary video series.

With us now to explain how these solutions can drive successful data center transformation is our panel, Duncan Campbell, Vice President of Marketing for HP Converged Infrastructure and small to medium-sized businesses (SMBs); Randy Lawton, Practice Principal for Americas West Data Center Transformation & Cloud Infrastructure Consulting at HP, and Larry Hinman, Critical Facilities Consulting Director and Worldwide Practice Leader for HP Critical Facility Services and HP Technology Services. The panel is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Campbell: We've seen that when a customer is successful in breaking down a large project into a set of quick wins, there are some very positive outcomes from that.

Number one, it breeds confidence, and this is a confidence that is actually felt within the organization, within the IT team, and into the business as well. So it builds confidence both inside and outside the organization.

The other key benefit is that when you can manifest these quick wins in terms of some specific return on investment (ROI) business outcome, that also translates very nicely as well and gets a lot of key attention, which I think has some downstream benefits that actually help out the team in multiple ways.

It's not just about attracting the best talent and executing well, but it's about marketing the team’s results as well.

One of the benefits in that is that you can actually break down these projects just in terms of some specific type of wins. That might be around standardization, and you can see a lot of wins there. You can quickly consolidate to blades. You can look at virtualization types of quick wins, as well as some automation quick wins.

We would advocate that customers think about this in terms of almost a step-by-step approach, knocking that down, getting those quick wins, and then marketing this in some very tangible ways that resonate very strongly.

Gardner: When you start to develop a cycle of recognition, incentives, and buy-in, we could also start to see some sort of a virtuous adoption cycle, whereby that sets you up for more interest, an easier time evangelizing, and so on.

Campbell: A virtuous cycle is well put. That really allows the team to get the additional green light to go to the next step in terms of their blueprint that they are trying to execute on. It gets a green light also in terms of additional dollars and, in some cases, additional headcount to add to their team as well.

What this does is, and I like this term the virtuous cycle, not only allows you to attract key talent, but it really allows you to retain folks. That means you're getting the best team possible to duplicate that, to get those additional wins, and it really does indeed become a virtuous cycle.

A good example is where we have been able to see a significant total cost of ownership (TCO) type of savings with one of our customers, McKesson, that in fact was taking one of these consolidated approaches with all their development tools. They saw considerable savings, both in terms of dollars—over $12.9 million—as well as a percentage of TCO savings that was upwards of 50 percent.

When you see tangible exciting numbers like that, that does grab people’s attention and, you bet, it becomes part of the whole social-media fabric and people want to go to a winner. Success breeds success here.

Lawton: Many of the transformation programs we engage in with our customers are substantially complex and span many facets of the IT organization. They often involve other vendors and service providers in the customer organization.

So there’s a tremendous amount of detail to pull together and organize in these complex engagements and initiatives. We find that there’s really no way to do that, unless you have a good way of capturing the data that’s necessary for a baseline.

It’s important to note that we manage these programs through a series of phases in our methodology. The first phase is strategy and analysis. During that phase, we typically run a discovery on all IT assets that would include the data center, servers, storage, the network environment, and the applications that run on those environments.

From that, we bridge into the second phase, which is architect and validate, where we begin to solution out and develop the strategies for a future-state design that includes the standardization and consolidation approaches, and on that begin to assemble the business case. In a detailed design, we build out those specifications and begin to create the data that determines what the future-state transformation is.

Then, through the implementation phase, we have detailed scorecards that are required to be tracked to show progress of the application teams and infrastructure teams that contribute to the program in order to guarantee success and provide visibility to all the stakeholders as part of the program, before we turn everything over to operations.

During the course of the last few years, our services unit has made investments in a number of tools that help with the capture and management of the data, the scorecarding, and the analytics through each of the phases of these programs. We believe that helps offer a competitive advantage for us and helps enable more rapid achievement of the programs from our customer perspective.

In these complex engagements it’s normally some time before there are quick-win type of achievements that are really notable.

For example, in the HP IT transformation program we undertook over several years back through 2008, we were building six new data centers so that we could consolidate 185 worldwide. So it was some period of time from the beginning of the program until the point where we moved the first application into production.

All along the way we were scorecarding the progress on the build-out of the data centers. Then, it was the build-out of the compute infrastructure within the data centers. And then it was a matter of being able to show the scorecarding against the applications, as we could get them into the next generation data centers.

If we didn't have the ability to show and demonstrate the progress along the way, I think our stakeholders would have lost patience or would not have felt that the momentum of the program was going on the kind of track that was required. With some of these tools and approaches and the scorecarding, we were able to demonstrate the progress and keep very visible to management the movements and momentum of the program.

A very notable example is one of our telecom customers we worked with during the last year and finished a program earlier this year. The company was purchasing the assets of another organization and needed to be able to clone the applications and infrastructure that supported business processes from the acquired company.

Within the mix of delivery for stakeholders in the program, there were nine different companies represented. There were some outsourced vendors from the application support side in the acquiree’s company, outsourcers in the application side for the acquiring company, and outsourcers in the data centers that operated data center infrastructure and operations for the target data centers we were moving into.

What was really critical in pulling all this together was to be able to map out, at a very detailed level, the tasks that needed to be executed, and in what time frame, across all of these teams.

The final cutover migration required over 2,500 tasks across these 9 different companies that all needed to be executed in less than 96 hours in order to meet the downtime window of requirements that were required of the acquiring company’s executive management.

It was the detailed scorecarding and operating war rooms to keep those scorecards up to date in real-time that allowed us to be able to accomplish that. There’s just no possible way we would have been able to do that ahead of time.

Gardner: Has there usually been a completely separate direction for facilities planning in IT infrastructure? Why was that the case, and why is it so important to end that practice?

Hinman: If you look over time and over the last several years, everybody has data centers and everybody has IT. The things that we've seen over the last 10 or 15 years are things like the Internet and criticality of IT and high density and all this stuff that people are talking about these days. If you look at the ways companies organized themselves several years ago, IT was a separate organization, facilities was a separate organization, and that actually still exists today.

One of the things that we're still seeing today is that, even though there is this push to try to get IT groups and facilities organizations to talk and work each other, this gap that exists between truly how to glue all of this together.

If you look at the way people do this traditionally—and when I say people, I'm talking about IT organizations and facilities organization—they typically will model IT and data centers, even if they are attempting to try and glue them together, they try to look at power requirements.

So we took this whole complex framework and data center program and broke it into four key areas. It looks simplistic in the way we've done this, and we have done this over many, many years of analysis and trying to figure out exactly what direction we should take. We've actually spun this off in many directions a few times, trying to continually make it better, but we always keep coming back to these four key profiles.

Business and risk is the first profile. IT architecture, which is really the application suite, is the second profile. IT infrastructure is the third. Data center facilities is the fourth.

One of the things that you will start to hear from us, if you haven’t heard it already via the data center transformation story that you guys were just recently talking about, is this nomenclature of IT plus facilities equals the data center.

Look at that, look at these four profiles, and look at what we call a top-down approach, where I start to get everybody synchronized on what risk profiles are and tolerances for risk are from an IT perspective and how to run the business, gluing that together with an IT infrastructure strategy, and then gluing all that into a data center facility strategy.

What we found over time is that we were able to take this complex program of trying to have something predictable, scalable, all of the groovy stuff that people talk about these days, and have something that I could really manage. If you're called into the boss’s office, as I and others have been over the many years in my career, to ask what’s the data center going to look like over the next five years, at least I would have some hope of trying to answer that question.

One of the the big lessons learned for us over the years has been this ability to not only provide this kind of modeling and predictability over time for clients and for customers. We had to get out of this mode of doing this once and putting it on a shelf, deploying a future state data center framework, keep the client pointing in the right direction.

The data gets archived, and they pick it up every few years and do it again and again and again, finding out that a lot of times there's an "aha" moment during those periods, the gaps between doing it again and again.

We've taken all of our modeling tools and integrated them to common databases, where now we can start to glue together even the operational piece, of data center infrastructure management (DCIM), or architecture and infrastructure management, facilities management, etc., so now the client can have this real-time, long-term, what we call a 10-year view of the overall operation.

So now, you can do this. You get it pointing the right direction, collect the data, complete the modeling, put it in the toolset, and now you have something very dynamic that you can manage over time. That's what we've done, and that's where we have been heading with all of our tools and processes over the last two to three years.

Gardner: I also remember with great interest the news from HP Discover in Las Vegas last summer about your EcoPOD and the whole POD concept toward facilities and infrastructure. Does that also play a part in this and perhaps make it easier when your modularity is ratcheted up to almost a mini data center level, rather than at the server or rack level?

Hinman: With the various what we call facility sourcing options, which PODs are certainly one of those these days, we've also been very careful to make sure that our framework is completely unbiased when it comes to a specific sourcing option.

What that means is, over the last 10 plus years, most people were really targeted at building new green-field data centers. It was all about space, then it became all about power, then about cooling, but we were still in this brick and mortar age, but modularity and scalability has been driving everything.

With PODs coming on the scene with some of the other design technologies, like multi-tiered or flexible data center, what we've been able to do is make sure that our framework is targeted at almost a generic framework where we can complete all the growth modeling and analysis, regardless of what the client is going to do from a facilities perspective.

It lays the groundwork for the customer to get their arms around all of this and tie together IT and facilities with risk and business, and then start to map out an appropriate facility sourcing option.

We find these days that POD is actually a very nice fit with all of our clients, because it provides high density server farms, it provides things that they can implement very quickly, and gets the power usage effectiveness (PUE) and power and operational cost down. We're starting to see that take a stronghold in a lot of customers.

Gardner: As we begin to wrap up, I should think that these trends are going to be even more important, these methods even more productive, when we start to factor in movement toward private cloud. Any thoughts about how scorecards and tracking will be even more important in the future, as we move, as we expect we will, to a more cloud-, mobile-, and eco-friendly world?

Lawton: In a lot of ways, there is added complexity these days with more customers operating in a hybrid delivery model, where there may be multiple suppliers in addition to their internal IT organizations.

Just like the example case I gave earlier, where you spread some of these activities not only across multiple teams and stakeholders, but also into separate companies and suppliers who are working under various contract mechanisms, the complexity is even greater. If that complexity is not pulled into a simplified model that is beta driven, that is supported by plans and contracts, then there are big gaps in the programs.

The scorecarding and data gathering methods and approaches that we take on our programs are going to be even more critical as we go forward in these more complex environments.

Operating the cloud environments simplifies things from a customer perspective, but it does add some additional complexities in the infrastructure and operations of the organization as well. All of those complexities add up too, meaning that even more attention needs to be brought to the details of the program and where those responsibilities lie within stakeholders.

Gardner: Larry Hinman, we're seeing this drive toward cloud. We're also seeing consolidation and standardization around data center infrastructure. So perhaps more large data centers to support more types of applications to even more endpoints, users, and geographic locations or business units. Getting that facilities and IT equation just right becomes even more important as we have fewer, yet more massive and critical, data centers involved.

Hinman: Dana, that's exactly correct. If you look at this, you have to look at the data center facilities piece, not only from a framework or model or topology perspective, but all the way down to the specific environment.

It could be that based on a specific client’s business requirements and IT strategy that it will require possibly a couple of large-scale core data centers and multiple remote sites and/or it could just be a bunch of smaller types of facilities.

It really depends on how the business is being run and supported by IT and the application suite, what the tolerances for risk are, whether it’s high availability, synchronous, all the groovy stuff, and then coming up with a framework that matches all those requirements that it’s integrating.

We tell clients constantly that you have to have your act together with respect to your profile, and start to align all of this, before you can even think about cloud and all the wonderful technologies that are coming down the pike. You have to be able to have something that you can at least manage to control cost and control this whole framework and manage to a future-state business requirement, before you can even start to really deploy some of these other things.

So it all glues together. It's extremely important that customers understand that this really is a process they have to do.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 14th December 2011 Copyright Bloor Research © 2011

All the recent compliance headlines in the financial services sector, at least in the UK and Europe, have been around Solvency II, Basel III and MiFid II. A regulation that has been largely overlooked (except by Trillium (which has just announced the Trillium FATCA Compliance Data Assessment service) by the IT industry is FATCA.

FATCA (foreign account tax compliance act) is a US law that comes into effect on 1st January 2013. It is designed to ensure that US citizens who hold assets abroad pay relevant taxes. So, suppose I lived in Boston (Massachusetts not Lincolnshire) and had an account with a UK-based bank, through which I held various investments. Today, I might be able to get away with not paying US tax on any profit I made from these investments. FATCA has been designed to ensure that that will not be possible in future.

FATCA applies to both US financial institutions that have any dealings overseas and to so-called foreign financial institutions: USFIs and FFIs respectively. These include banks, insurance companies, alternative investment companies, private equity companies, hedge funds and so on and (subject to their being some level of non-US interaction) to any financial company that either has US citizens as customers or which holds US assets.

FFIs can either register as participating or as non-participating. Non-participation means that you are effectively opting out. However, if you do this, or if you are a participating company and fail to comply with the regulations, then the US tax authorities will apply a 30% withholding tax against any sales of US assets. Moreover, this is not against profits but against revenue so you could sell a stock at a loss and then have the 30% deducted. It is difficult to imagine any company that has any significant US business not wanting to both participate and comply.

If you decide to participate then you must be able to recognise which of your clients are US citizens and you will be required to provide relevant information about those clients. You must also have relevant processes in place to recognise whether new clients are American or not. The same is also true if you formally decide not to participate: you will need to demonstrate that you have procedures in place to recognise if new clients are American and, therefore, reject them as clients.

Unfortunately, the requirement for participating FFIs to provide relevant information about their US clients will fly in the face of the data protection laws of a number of countries. Where this is the case then the FFI will need to obtain a waiver from each of its clients to confirm that that information can be passed to the IRS or it will need to close that account.

Needless to say there are significant data governance implications in order to support FATCA, whether you are a USFI or are an FFI. You will need to know which clients are US citizens, ensure that they have signed a waiver, if relevant, have procedures for identifying whether new clients are US citizens or not, and have processes that ensure that only information about US citizens is provided upon request and that you do not break data protection laws by inadvertently sending information about non-US citizens. You will also need to be very clear about your data quality processes and careful about de-duplication and merging of records.

I have to say that this makes me feel a little sorry for financial services companies. In the UK they have only recently had to comply with FSCS regulations and the insurance sector and banks (those that provide asset management) have to comply with Solvency II, which is the same official start date (it may be delayed) as FATCA. That's a lot to do in a short space of time (not to mention MiFID II and Basel III waiting in the wings). The one consolation is that you need good data governance for all three of these. Those that thought they could get away without seriously addressing data governance for FSCS may not be wishing that they had done it properly the first time.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 12th December 2011 Copyright Bloor Research © 2011

I have just got back from the Netezza user conference and there was an interesting presentation on using IBM's Information Server in conjunction with Netezza. The major bone of contention during the presentation, which caused much discussion, was that while most customers and prospects for Netezza were drawn, at least in part, to its simplicity, the last thing they wanted was complex data integration software front-ending to it.

You can see their point. Conversely, if you have 15 data sources, or 50, then you are dealing with a complex problem and hiding that complexity isn't necessarily easy. However, it did get me thinking about how much easier it could be. So here are some thoughts (which are not specific to IBM by the way).

The first thing I would address is semantics, though I don't think that having a semantic layer (à la expressor) is actually necessary: at this level a thesaurus is all you really need. This would know that "account_no" is the same as "custID" and "clientnum" for example. It would be shipped with common synonyms pre-built but should be extensible so that users can define their own equivalences where necessary. It should probably also include support for concatenations and decompositions. For example, you might have a product category field ("AB") and a product number field ("12345") that you want to join together in the warehouse or, conversely, you might have a combined field that you want to split.

Now, of course you can do this stuff using rules but what I am really advocating here is that rules creation and management should be reserved for those transformations that are genuinely complex and the rest should be automated. By separating the thesaurus from rules management you can apply automated capabilities to the former where you cannot do so more broadly.

The second area where more automation could be provided is in datatype conversions. If "account_no" in your OLTP system is numeric but it is alphanumeric in your warehouse then a) the data integration software should recognise this fact and b) it should be able to perform the transformation automatically without you have to tell it do so. Tools like expressor that use semantic typing can do this but, again, you don't necessarily need semantics for this purpose.

Reuse could also be made simpler. If you are going to perform some sort of complex transformation on data from tables X and Y, say, then it cannot be beyond the wit of man for the software to automatically display to you (in a pop-up window perhaps) all the existing transformations that have been defined  across those two tables. The big problem with encouraging reuse is that's it's a hassle finding what's been done previously: so make it really simple.

There's one other biggie that could make a big difference towards removing complexity but unfortunately IBM mentioned it to me rather than the other way around and as it is in their roadmap (not all of the above are) and they want it to be an unpleasant surprise for their competition, you'll have to wait until that feature is announced.

Finally, one of the interesting things that IBM has done with Information Server to support Netezza is to allow jobs to be run in either ETL or ELT-mode. So, you can push joins and processing to either sources or targets, or run on the Information Server platform, as suits you best. This is an excellent idea, I like it a lot. However, the problem is that you have to define even more rules about when you do this: for example, "between 8.00am and 5.00pm use an ETL-based approach but otherwise use ELT". Wouldn't a much simpler option (for the user) be if the data integration product had its own optimiser, collected statistics and so forth, just like a database does, generated an "integration plan" (as opposed to a query plan) and generally automated the whole process of where you perform relevant tasks?

The Netezza users are right: data integration is complex but it behoves vendors of data integration solutions to make it as simple for users as possible.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 9th December 2011 Copyright Bloor Research © 2011

I have written previously about how the major vendors of CEP (complex event processing) solutions are still primarily focused on either capital markets or supporting business process and SOA and are not really focusing on business intelligence and analytics per se. The exceptions are IBM and StreamBase. However, this is by no means true when it comes to the operational and big data markets. In the former area, Tier-3 and Red Lambda both use CEP for SIEM (security information and event management) while AccelOps employs it for data centre monitoring.

However, it is on big data more generally that I want to focus. Here there are a number of new and not so new ventures that provide what is becoming known as "continuous BI". First, it is worth explaining what continuous BI is. Normally, when you run a query that's it: it's a one-time shot. If you want to run the same query again then you have to activate it again. A continuous query, on the other hand, is one that you fire up and, once instigated, it continues to run until you tell it not to. What happens is that the results of the query change as new data passes through the query. It is, if you like, a sort of sophisticated real-time monitoring that doesn't simply record what has happened but supports complex analysis and pattern matching (and, of course, simpler things) against the incoming data.

Now, this is typically a big data problem but it won't work to simply load the data into Hadoop and then query it. That won't be continuous. So you need some sort of CEP engine to do it for you. Four such products are (or will be, they are not all commercially available yet) S4 from Yahoo!, which is part of the Apache Incubator programme; Storm from BackType Technology, which describes its product as "The Hadoop of Real-time Processing"; DarkStar from Cloud Event Processing, which provides CEP-based streaming MapReduce; and HStreaming (from the company of the same name). As its company name implies, DarkStar is a cloud-based and HStreaming also offers a cloud-based version.

No doubt all of these products have their merits. However, I have two issues with all of them. Firstly, as far as I can tell (their web sites are singularly uninformative) these are all platforms and unless you are a geek you don't want a platform for BI, you want a solution: you want a user interface where business people can define queries either for prolonged use or on ad hoc basis, and you want visualisation and graphics that can display the results of those queries to them. What these products all offer (and the same is true for IBM Streams) is a platform on which developers can build query capabilities. But I don't think that's enough and it's certainly not what end-users want. The only company that has actually built this sort of interface (or, more precisely, is starting to build it: this is only version 1) on top of a CEP platform, as far as I know, is StreamBase, with its recently released LiveView product. I'll discuss this in detail in a separate posting but suffice it to say that it is a mile ahead of all these other vendors in offering continuous BI in so far as I can tell from their web sites (and, yes, I need to do some more research and, yes, I will report back having done so).

The second problem I have with these vendors is that yes it is a good idea and yes there is gap in the market but I don't think that gap in the market will last long. Certainly, I would expect IBM to integrate Cognos on top of Streams, SAS will be entering the market next year, it would reasonable to expect Sybase Aleri to be integrated with Business Objects, TIBCO could do the same thing with Spotfire and obviously Oracle also has that capability. None of this means that there won't be a market for open source and smaller suppliers but they are going to have to be careful about where they put their marketing dollars. One can imagine that Pentaho and JasperSoft might be interested in integrating with, and potentially acquiring, these vendors. No doubt an ISV market will also emerge but this may take some time.

As with all things associated with big data, continuous BI is an immature market and we will have to wait to see what happens but I expect the big boys to jump on this with both feet.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 5th December 2011 Copyright Interarbor Solutions © 2011

The latest BriefingsDirect podcast discussion centers on how new data and analysis approaches are significantly improving IT operations monitoring, as well as providing stronger security.

The conversation examines how AccelOps has developed technology that correlates events with relevant data across IT systems, so that operators can gain much better insights faster, and then learn as they go to better predict future problems before they emerge. That's because advances in big data analytics and complex events processing (CEP) can come together to provide deep and real-time, pattern-based insights into large-scale IT operations.

Here to explain how these new solutions can drive better IT monitoring and remediation response—and keep those critical systems performing at their best—is Mahesh Kumar, Vice President of Marketing at AccelOps. The discussion is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: AccelOps is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: Is there a fundamental change in how we approach the data that’s coming from IT systems in order to get a better monitoring and analysis capability?

Kumar: The data has to be analyzed in real-time. By real-time I mean in streaming mode before the data hits the disk. You need to be able to analyze it and make decisions. That's actually a very efficient way of analyzing information. Because you avoid a lot of data sync issues and duplicate data, you can react immediately in real time to remediate systems or provide very early warnings in terms of what is going wrong.

The challenges in doing this streaming-mode analysis are scale and speed. The traditional approaches with pure relational databases alone are not equipped to analyze data in this manner. You need new thinking and new approaches to tackle this analysis problem.

Gardner: Also for issues of security, often as trying different types of attacks. So this needs to be in real-time as well?

Kumar: You might be familiar with advanced persistent threats (APTs). These are attacks where the attacker tries their best to be invisible. These are not the brute-force attacks that we have witnessed in the past. Attackers may hijack an account or gain access to a server, and then over time, stealthily, be able to collect or capture the information that they are after.

These kinds of threats cannot be effectively handled only by looking at data historically, because these are activities that are happening in real-time, and there are very, very weak signals that need to be interpreted, and there is a time element of what else is happening at that time. This too calls for streaming-mode analysis.

If you notice, for example, someone accessing a server, a database administrator accessing a server for which they have an admin account, it gives you a certain amount of feedback around that activity. But if on the other hand, you learn that a user is accessing a database server for which they don’t have the right level of privileges, it may be a red flag.

You need to be able to connect this red flag that you identify in one instance with the same user trying to do other activity in different kinds of systems. And you need to do that over long periods of time in order to defend yourself against APTs.

Gardner: It's always been difficult to gain accurate analysis of large-scale IT operations, but it seems that this is getting more difficult. Why?

Kumar: If you look at trends, there are on average about 10 virtual machines (VMs) to a physical server. Predictions are that this is going to increase to about 50 to 1, maybe higher, with advances in hardware and virtualization technologies. The increase in density of VMs is a complicating factor for capacity planning, capacity management, performance management, and security.

In a very short period of time, you have in effect seen a doubling of the size of the IT management problem. So there are a huge number of VMs to manage and that introduces complexity and a lot of data that is created.

Cloud computing is another big trend. All analyst research and customer feedback suggests that we're moving to a hybrid model, where you have some workloads on a public cloud, some in a private cloud, and some running in a traditional data center. For this, monitoring has to work in a distributed environment, across multiple controlling parties.

Last but certainly not the least, in a hybrid environment, there is absolutely no clear perimeter that you need to defend from a security perspective. Security has to be pervasive.

Given these new realities, it's no longer possible to separate performance monitoring aspects from security monitoring aspects, because of the distributed nature of the problem. ... So change is happening much more quickly and rapidly than ever before. At the very least, you need monitoring and management that can keep pace with today’s rate of change.

The basic problem you need to address is one of analysis. Why is that? As we discussed earlier, the scale of systems is really high. The pace of change is very high. The sheer number of configurations that need to be managed is very large. So there's data explosion here.

Since you have a plethora of information coming at you, the challenge is no longer collection of that information. It's how you analyze that information in a holistic manner and provide consumable and actionable data to your business, so that you're able to actually then prevent problems in the future or respond to any issues in real-time or in near real-time.

You need to nail the real-time analytics problem and this has to be the centerpiece of any monitoring or management platform going forward.

Gardner: So we have the modern data center, we have issues of complexity and virtualization, we have scale, we have data as a deluge, and we need to do something fast in real-time and consistently to learn and relearn and derive correlations.

It turns out that there are some advances in IT over the past several years that have been applied to solve other problems that can be brought to bear here. You've looked at what's being done with big data and in-memory architectures, and you've also looked at some of the great work that’s been done in services-oriented architecture (SOA) and CEP, and you've put these together in an interesting way.

Kumar: Clearly there is a big-data angle to this.

Doug Laney, a META and a Gartner analyst, probably put it best when he highlighted that big data is about volume, the velocity or the speed with which the data comes in and out, and the variety or the number of different data types and sources that are being indexed and managed.

For example, in an IT management paradigm, a single configuration setting can have a security implication, a performance implication, an availability implication, and even a capacity implication in some cases. Just a small change in data has multiple decision points that are affected by it. From our angle, all these different types of criteria affect the big data problem.

There are a couple of approaches. Some companies are doing some really interesting work around big-data analysis for IT operations.

They primarily focus on gathering the data, heavily indexing it, and making it available for search, thereby derive analytical results. It allows you to do forensic analysis that you were not easily able to with traditional monitoring systems.

The challenge with that approach is that it swings the pendulum all the way to the other end. Previously we had very rigid, well-defined relational data-models or data structures, and the index and search approach is much more of a free form. So the pure index-and-search type of an approach is sort of the other end of the spectrum.

What you really need is something that incorporates the best of both worlds and puts that together, and I can explain to you how that can be accomplished with a more modern architecture. To start with, we can't do away with this whole concept of a model or a relationship diagram or entity relationship map. It's really critical for us to maintain that.

I’ll give you an example. When you say that a server is part of a network segment, and a server is connected to a switch in a particular way, it conveys certain meaning. And because of that meaning, you can now automatically apply policies, rules, patterns, and automatically exploit the meaning that you capture purely from that relationship. You can automate a lot of things just by knowing that.

If you stick to a pure index-and-search approach, you basically zero out a lot of this meaning and you lose information in the process. Then it's the operators who have to handcraft these queries to have to then reestablish this meaning that’s already out there. That can get very, very expensive pretty quickly.

Our approach to this big-data analytics problem is to take a hybrid approach. You need a flexible and extensible model that you start with as a foundation, that allows you to then apply meaning on top of that model to all the extended data that you capture and that can be kept in flat files and searched and indexed. You need that hybrid approach in order to get a handle on this problem.

Gardner: Why do you need to think about the architecture that supports this big data capability in order for it to actually work in practical terms?

Kumar: You start with a fully virtualized architecture, because it allows you not only to scale easily ... but you're able to reach into these multiple disparate environments and capture and analyze and bring that information in. So virtualized architecture is absolutely essential.

Maybe more important is the ability for you to auto-correlate and analyze data, and that analysis has to be distributed analysis. Because whenever you have a big data problem, especially in something like IT management, you're not really sure of the scale of data that you need to analyze and you can never plan for it.

Think of it as applying a MapReduce type of algorithm to IT management problems, so that you can do distributed analysis, and the analysis is highly granular or specific. In IT management problems, it's always about the specificity with which you analyze and detect a problem that makes all the difference between whether that product or the solution is useful for a customer or not.

A major advantage of distributed analytics is that you're freed from the scale-versus-richness trade-off, from the limits on the type of events you can process. If I wanted to do more complex events and process more complex events, it's a lot easier to add compute capacity by just simply adding VMs and scaling horizontally. That’s a big aspect of automating deep forensic analysis into the data that you're receiving.

I want to add a little bit more about the richness of CEP. It's not just around capturing data and massaging it or looking at it from different angles and events. When we say CEP, we mean it is advanced to the point where it starts to capture how people would actually rationalize and analyze a problem.

The only way you can automate your monitoring systems end-to-end and get more of the human element out of it is when your CEP system is able to capture those nuances that people in the NOC and SOC would normally use to rationalize when they look at events. You not only look at a stream of events, you ask further questions and then determine the remedy.

To do this, you should have a rich data set to analyze, i.e. there shouldn’t be any hard limits placed on what data can participate in the analysis and you should have the flexibility to easily add new data sources or types of data. So it's very important for the architecture to be able to not only event on data that is stored in traditional models or well-defined relational models, but also event against data that’s typically serialized and indexed in flat file databases.

Gardner: What's the payoff if you do this properly?

Kumar: It is no surprise that our customers don’t come to us saying we have a big data problem, help us solve a big data problem, or we have a complex event problem.

Their needs are really around managing security, performance and configurations. These are three interconnected metrics in a virtualized cloud environment. You can't separate one from the other. And customers say they are so interconnected that they want these managed on a common platform. So they're really coming at it from a business-level or outcome-focused perspective.

What AccelOps does, under the covers, is apply techniques such as big-data analysis, complex driven processing, etc., to then solve those problems for the customer. That is the key payoff—that customer’s key concerns that I just mentioned are addressed in a unified and scalable manner.

An important factor for customer productivity and adoption is the product user-interface. It is not of much use if a product leverages these advanced techniques but makes the user interface complicated—you end up with the same result as before. So we’ve designed a UI that’s very easy to use, requires one or two clicks to get the information you need; a UI-driven ability to compose rich events and event patterns. Our customers find this very valuable, as they do not need super-specialized skills to work with our product.

What we've built is a platform that monitors data center performance, security, and configurations. The three key interconnected metrics in virtualized cloud environments. Most of our customers really want that combined and integrated platform. Some of them might choose to start with addressing security, but they soon bring in the performance management aspects into it also. And vice versa.

And we take a holistic cross-domain perspective—we span server, storage, network, virtualization and applications. What we've really built is a common consistent platform that addresses these problems of performance, security, and configurations, in a holistic manner and that’s the main thing that our customers buy from us today.

Most of our customers start off with the free trial download. It’s a very simple process. Visit www.accelops.com/download and download a virtual appliance trial that you can install in your data center within your firewall very quickly and easily.

Getting started with the AccelOps product is pretty simple. You fire up the product and enter the credentials needed to access the devices to be monitored. We do most of it agentlessly, and so you just enter the credentials, the range that you want to discover and monitor, and that’s it. You get started that way and you hit Go.

The product then uses this information to determine what’s in the environment. It automatically establishes relationships between them, automatically applies the rules and policies that come out of the box with the product, and some basic thresholds that are already in the product that you can actually start measuring the results. Within a few hours of getting started, you'll have measurable results and trends and graphs and charts to look at and gain benefits from it.

Gardner: It seems that as we move toward cloud and mobile that at some point or another organizations will hit the wall and look for this automation alternative.

Kumar: It’s about automation and distributed analytics and about getting very specific with the information that you have, so that you can make absolutely more predictable, 99.9 percent correct of decisions and do that in an automated manner. The only way you can do that is if you have a platform that’s rich enough and scalable and that allows you to then reach that ultimate goal of automating most of the management of these diverse and disparate environments.

That’s something that's sorely lacking in products today. As you said, it's all brute-force today. What we have built is a very elegant, easy-to-use way of managing your IT problems, whether it’s from a security standpoint, performance management standpoint, or configuration standpoint, in a single integrated platform. That's extremely appealing for our customers, both enterprise and cloud-service providers.

I also want to take this opportunity to encourage those of your listening or reading this podcast to come meet our team at the 2011 Gartner Data Center Conference, Dec. 5–9, at Booth 49 and learn more. AccelOps is a silver sponsor of the conference.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 29th November 2011 Copyright Bloor Research © 2011

Gerry Brown (my colleague who specialises in BI and Marketing solutions) wrote in a recent analysis piece of the damage that was done to the CRM market by vendors over-promising and under-delivering, and turning the original promise of CRM, a sustainable means of building lifetime value in one's customer base, and limiting it to become a quick and dirty sales cycle aide. These are sentiments which I share, and it has bred a widespread cynicism about CRM. It therefore came as a very pleasant surprise to be invited to look at Sugar CRM by one of their UK based partners, Bestoutcome. Sugar is an open source product, available at four price points offering increasing capability and storage facilities, available both for in-house installation or provision via the cloud. What immediately struck me was the clarity of the interface; everything was there that you would expect and all was readily identifiable, there was no hide and seek for your favourite function. The last time I saw something similar was a few years ago when I looked at RightNow CRM, a SaaS offering that, in a similar fashion, had all that you need but nothing that was obviously superfluous or confusing, and they have just been bought by Oracle, so this is a good endorsement that the Sugar people are setting about this in the right way.

So the problem with most so-called CRM solutions is that they are not about the long term relationship and managing prospects to customers and on to being supporters, they place to much emphasise on the short term and on the capture of leads and their conversion to a sale - not so with Sugar. Yes, it covers that process but it does a lot more to actually nurture the relationship and allow a long term mutually beneficial relationship to be developed. So Sugar is not just about sales, it also covers marketing and support. All of these functions are linked together by an underpinning workflow system. This is the key to any effective CRM system; if you want to actually have a relationship with a customer you need to both ensure that all channels touching the prospect or customer actually have all of the available knowledge about the person to hand so there is no starting from scratch at each point of contact, and each contact must result in the desired outcome without a constant need to be prompted to do the things which were promised by the supposed recipient of the services! Sugar does all of that elegantly and intuitively, so you do not have to undergo weeks of training to make this solution do what you want.

So, what you are getting is, of course, the ubiquitous Sales Force Automation piece. This offers facilities to track and share contacts and opportunities and goes on to manage quotes and contracts. All of this is integrated with email systems such as Outlook, Gmail, Yahoo or even Lotus Notes, so all actions are captured, tracked and logged against the contact record, enabling everyone to share a common view. When entering a new contact the system also gives you a head start because it can be integrated with Social Media such as Facebook, Linkedin and Twitter to bring up background about the person, and also news feeds etc. to identify valid news stories in the case of companies etc. There are dashboards to enable performance to be visually monitored and forecasts can be created, and reports are available to manage the whole process. So comprehensive, but not with functional overload that masks the essentials with things that might be used once in a very long period!

The marketing element allows campaigns to be created with their budgets and goals captured. The system then can identify target lists from the contacts in the system. The system can capture leads from website landing pages, and leads can be assigned to sales to close down. Again the loop is closed by management reports that allow performance, including campaign ROI, to be reported.

The support element allows customer cases to be handled quickly and effectively, with inbound communications being logged against the customer record, then workflow ensures that actions are taken to complete the handling of the request, and learning can be shared via the knowledgebase, and there is reporting to ensure that the responsiveness is monitored.

Another very exciting feature for all those of us who find being kept apart from our iPad a painful experience, is that Sugar supports not just the iPad but also Android, BlackBerry and iPhone platforms so you can access all of this capability on a smartphone or tablet - this is fast becoming an essential capability.

So there you have it in a quick skim, a very well thought out, highly functional, CRM solution, supported by a vibrant open source user community putting forward add-ons etc to make the tool highly configurable, brought to you at a price that makes it affordable to groups like SMEs and charities who really need a CRM solution but really cannot afford the price tag of the big enterprise solutions and, to my mind, I can see no glaring omissions in what Sugar can do and I really do encourage people to go the website and take a look at how clean the interface is and how intuitive it is to grasp how to drive this application. In a world in which the economy is fragile everyone needs to take care of their customers like never before and, with Sugar, that can now be achieved admirably at a price that will only come as a shock, because it is not as much as you would think, having seen what it buys.

By: Mark McGregor, Research Director, Bloor Research Posted: 28th November 2011 Copyright Bloor Research © 2011

There are many words now being written, especially in marketing circles, about the "gamification" of BPM and process improvement. However, there appears to be little consensus on what it might be and what it might mean.

The linkage of game theory with process technology has been occurring for some time. However the most successful company in the space historically did not make a lot of noise about it. Alan Trefeler of Pegasystems is a chess master and has been using the gaming principles of chess as the core of the company's software for many years. While TIBCO founder Vivek Ranadive believes that in applying the principles of how great game players succeed to business will provide greater competitive advantage.

As someone who has been using scenario and role play based training for some years it is interesting to see how the tide is turning. It used to be that people laughed or, worse still, resisted the idea that you were going to encourage their people to play games in order to learn. Now it seems that teaching and learning via games is highly fashionable.

Singularity, BizzDesign and 21apps all make use of and promote the idea of game play in order to extract requirements and motivate people for change in the process arena. In the case of the first two their focus is on ensuring that their customers deliver better applications faster. As we know, any vendor needs happy referenceable customers and ensuring that the systems built with their technology are more effective is important to Singularity and BizzDesign. Both of these companies use structured role/scenario play to speed up leaning and enable people to quickly discover problems for themselves.

My impression is that, while the structure is extremely effective as a learning and discovery tool, the challenge is that people may not always follow through with buying your technology afterwards.

In the case of 21apps it is more a case of using games as tools, so continuing to use traditional techniques like SWOT analysis or brainstorming, but using game ideas to make the sessions more effective.

Two recent books are driving much of the current interest, "Gamestorming" by Dave Gray, Sunni Brown and James Macanufo and "Innovation Games" by Luke Hohmann. Both books are packed with ideas and games to help you in all sorts of different situations. Luke Hohmann also has his own web site where you can play many of the games online.

The rationale for playing games is grounded in good learning theory. We learn and retain information faster when playing. Cast your mind back to when you were 4 or 5 years old and think of the games you used to play, either on your own or with your friends. It might be that you were like me at that age and loved maths! Well actually I loved the games the teacher played and found that they helped me learn maths. At that age I found maths easy and fun - all because of the games we played. In my case, wind the clock forward 8 years to a bigger school where there were lectures and stern teachers and, surprise, surprise, I quickly learned to hate maths! And my skills failed to live up to the teacher's ideas of what they could or should be - how I wished they understood learning and games theory then. Your specific experiences will be different, but I suspect if you think hard enough you will find similar instances in your own past.

As an organisation looking to gain or build consensus, capture requirements, generate ideas, overcome resistance or any one of a hundred other things, the use of games via a skilled facilitator will speed up your results, increase motivation and overcome challenges. If you are not already applying games as a part of your process workshops then you are definitely missing an opportunity.

It is not all upside though; much role and game play success is down to the interactions among people and the dynamics of the group and this is an area of worry when it comes to vendors and tools.

There are online games, such as IBM's Innov8 and others, that make use of interactive worlds like Second Life. Their biggest appeal will always be to those who might normally play computer games, especially in an online community. This, for me, brings a high risk, for these situations might provide realistic business and process scenarios for us to work on, but fail to provide a great deal of real world interaction among people, thus diminishing some of the potential return.

In summary, all businesses should be looking at the latest developments in games and the application of games to learning. In order to leverage the techniques you will need to access or train good facilitators, measure success by the outcomes and not by the bulleted "you will learn lists", and be prepared for you and your teams to discover things about themselves and their work that they may not have ever thought of.

Lastly, do not be seduced by technology. The use of games theory, as applied by people like Trefler and Ranadive, has sound roots. They are helping to create systems that work more like we, as people, do. Others who take games and simply computerise them are not leveraging the underlying theory benefits and may, in fact, also destroy some of the interaction benefits they purport to support.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 25th November 2011 Copyright Bloor Research © 2011

The biggest barriers that I see to the widespread adoption of Big Data is the skills that are required to deliver the benefits that we all agree can be obtained. In the standard MIS layers of the BI tool suites (Reporting, Dashboards, and OLAP) we are seeing an increasing emphasis on what is being labelled Agile BI, a tool set that offers the same power as the traditional tools, but which costs less, is easier to use, is targeted at the business user and not the IT professional, is far more visual in how they are controlled and what they output, and which increase productivity in a step-change. But in the area in which Big Data offers the biggest potential return, that of data mining, the application of statistical and mathematical modelling to identify patterns of significance, there has been no comparable change, until now. Alpine Miner is the first offering I have seen that is clearly addressing the challenges of the scale and affordability of exploiting Big Data.

I have, for a long time, been a big fan of KXEN as an alternative to SPSS or SAS for those businesses that do not have the skills required to really make the most of the considerable power of the established market leaders, being easier to deploy and understand the results if you are not a statistician, whilst still delivering models of comparable statistical validity. But all of those technologies are, at present, going to struggle to economically cope with the scale of the data when it comes to Big Data. This is where Alpine Data Labs provide the first sight of a next generation of data mining solution, which copes with the scale of big data, but is still affordable, and is designed to be used by people in the business world and not just statisticians.

Alpine Data Labs are a spin-off from Greenplum (just prior to the EMC acquisition of Greenplum last year). Their primary product, Alpine Miner, is a data mining and analytics platform meant to leverage the processing capabilities of MPP databases like Greenplum and Oracle's Exadata. Alpine is headquartered in San Mateo, California with a sizeable development shop in Beijing. They have over 15 early adopter customers in both the US and China, and already over 500 evaluation downloads have been taken, so there is a lot of interest and the company is showing very solid growth based on quality opportunities.

This is a disruptive technology, aiming to bring advanced analytics into the hands of people capable using it to change the business landscape. The user interface is a drag and drop GUI, and the technology is designed to be cost effective, so there are few reasons at the business level not to invest. The promise of Big Data will only be achieved when the ROI is compelling. If the benefits can only be obtained by the deployment of very expensive technology, by very expensive consultants, operating over elongated time periods, Business will not bother.

So lets see how Alpine Data address these challenges. Firstly we should note that Alpine Miner is very much a work in progress. The company has a clear and compelling vision of where they are heading, and have already established the fundamental building blocks. The first challenge is one of scalability. This they are confident they have addressed and are well on the way to handling any size of data on any platform. Next they want to provide a platform that is well integrated so the modelling is not just providing insight; that insight must be readily actionable, so that it drives business improvement. Again, this is there already but will probably just get better and better over time. The third point they want to address is making data mining a participative technology, so that it is embedded into the business decision-making process, used naturally within the business to aid effective business management. This model is coming in 2012 with Alpine Miner v3.0, and finally they see some sort of a SaaS offering down the road.

The key to much of what Alpine delivers is that they are embedding the computation into the data, and not moving data to the tool. Alpine Miner is an analytics engine that connects directly to Greenplum, PostgreSQL and Exadata with offerings for Netezza and Hadoop on the roadmap.  Alpine runs all of the transformations, calculations, and analytic processes directly within the database itself, thus eliminating the need to extract data out of the database and sending it off to another (smaller) analytic server for processing. On the client side, a PC or Mac controls things through a point and click GUI, with no arcane statistical notation to navigate. This "in-database" approach leverages the MPP capabilities of the appliance, and eliminates many of the constraints on scalability and integration seen with traditional data mining tools. The tool is designed to be used easily by BI analysts and should be a natural extension of their BI toolkits.

This model is capable of revolutionising how analytics are deployed. The slow, expensive model of the past is being replaced by a quick to deploy, rapid time to results, affordable alternative tailored to the needs of the business user. None of this changes what we do, just how we achieve it. So we are still looking to find answers to understand the customer and how they value things, and how then to market to them those things that they will value the most, but the cost of doing that has been changing dramatically, making the ROI really compelling.

Alpine Miner is a really exciting offering, which makes the promise of Big Data analytics more of a reality, to a broader audience than has been true before. I suggest you track their progress.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 25th November 2011 Copyright Bloor Research © 2011

The latest entry into the BDDB (big data database) market is Hadapt, which has just announced the public beta (it has been in private beta since June) of its eponymous product.

What is interesting, and different, about Hadapt is that it has a hybrid database with two storage engines. One of these is HDFS (Hadoop distributed file system) and the other is a relational database based on PostgreSQL. So, you store unstructured data in HDFS and structured data in the relational database. While unusual, this is by no means a unique approach. For example, DB2 has two storage engines: one for relational data and one for XML.

However, one other point of note is that Hadapt has been designed specifically for the low-cost clustered environments that are common to BDDB environments and, especially, deployments of these within cloud-based environments. One of the problems with such public environments of this sort is that you can get what might be called 'node slowdowns' where a particular node inexplicably suddenly starts running much slower than normal (perhaps Amazon, or whoever, is performing maintenance). In order to avert this issue, not only is the unstructured data triplicated (as is standard with HDFS) in Hadapt but so too is the structured data and, if a node starts to slow, then processing can be diverted to one of the other nodes holding that data in order to ensure consistent performance. Of course, there is a downside in holding two copies of the data but since structured data is only going to be a small part of the whole this should not be onerous and, of course, you get failover and load balancing for free.

The one comment would be is that sometimes you are going to just want to query structured data, in which case you want great traditional analytic performance, and a fairly standard PostgreSQL row-based database probably isn't go to hack that. However, we understand from Hadapt that it would be relatively trivial to replace that data store with, say, a column-based one so the potential is there to offer a really powerful combination of structured and unstructured query support (which Hadapt calls multi-structured) within the same system.

In so far as querying the data is concerned, Hadapt is SQL-based. I understand that it has significantly more capabilities (and, in internal tests, is an order of magnitude faster) than Hive, though it is not yet (it will be) ANSI compliant. Nevertheless, this means that you should be able to use your existing business intelligence tools alongside Hadapt. After queries are received the database software does two things: it generates MapReduce to run against HDFS and it uses what the company calls 'Split Query Execution' (for which the company has applied for a patent) so that relevant SQL is directed at the structured data in the relational engine, so that you can have a single query running against both storage engines.

In so far as Hadoop itself is concerned, Hadapt is distribution-agnostic, so that if you are concerned about a NameNode single point of failure, for example, then you could choose to use Hadapt in conjunction with MapR, though most of the company's beta sites are using Cloudera.

It is far too early to say how successful Hadapt will be. Clearly, the ability to address multi-structured data within a single query can potentially open up new avenues of discovery but the product is not yet fully functional or mature. However, if it does prove to be successful then it raises one big question mark: to date, the assumption amongst the data warehousing community (vendors and analysts alike) has been that you will have a traditional data warehouse on the one hand and a BDDB alongside it, linked by some sort of data integration technology. Hadapt calls that whole approach into question: why have two products when one can do the job as well or better? If Hadapt proves successful then we may see other vendors moving in the same direction. For example, Aster Data (Teradata) already supports MapReduce within its warehouse and adding a BDDB storage engine certainly wouldn't be a stretch. Conversely, as I have already mentioned, IBM already has two storage engines under DB2: why not a third? It could be that Hadapt represents a shift in the way we think about the combination of big data and warehousing in much the same way that Netezza made us think again about data warehousing itself just a few years ago.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 24th November 2011 Copyright Bloor Research © 2011

Before I start I should say that this is not just about Hadoop but also about all its extensions, distributions, alternatives or replacements. For brevity's sake we need some terminology here so I will refer to these generically as BDDB (big data database) solutions.

As someone said to me recently, "it's only when you start to look at Hadoop that you start to realise what's important about the MS in DBMS". However, it's not management that I want to discuss and, actually, while a major focus of the BDDB vendors is to provide the sort of manageability you might expect, there is some way to go before these are fully robust.

What I want to discuss is governance, and there are a number of issues. Firstly, the advocates of BDDBs suggest that you might well want to include conventional structured data within your BDDB as well as unstructured data, because relevant queries require both types of data. Fair enough. But how do you ensure the quality of the structured data? I don't know any data quality tools (or profiling tools for that matter) that run against BDDBs. That means that a) you don't care about data quality or b) you cleanse the data at source (a good idea but many companies don't do it) or c) you load the data into your conventional warehouse first, cleanse it and then export the data out to the BDDB. In any case, governance goes beyond your traditional data quality. It is also about validating your business rules so that the logic for transformations is sound from a business perspective.

Secondly, there are quality issues around the unstructured data that you may load into your BDDB. Consider stock ticks or call detail records: both of these are frequently duplicated and those duplicates need to be filtered out. Similarly, lots of RFID events have no value, typically when nothing changes, and you also need filtering mechanism here too. (IBM has such a solution but, strangely, it is in the WebSphere portfolio rather than InfoSphere where all the big data stuff is - go figure). Further, this issue is not limited to sensor and event-based information. Consider tweets: I write a tweet (not often) and someone forwards it, someone else comments on it, it goes viral (not likely): how many tweets is that? If the original tweet gets copied 5,000 times does that count as one tweet or 5,001? I guess it depends on what you want to do with the information but there is certainly a governance issue.

Thirdly, there is the question of how you get meaningful information out of unstructured data? This is particularly important if you want to combine this information with structured data. Here, at least, there is now a product. Informatica has just released HParser, which is a parser optimised for Hadoop. That is, it runs on Hadoop (with all of its parallelism and so forth) and which is used to parse unstructured data and semi-structured data in much the same way that a data quality tool might parse a product description.

HParser runs on nearly any distribution of Apache Hadoop and Informatica distributes it via its marketplace. Informatica has also just announced a partnership with Hortonworks and you can also see the link to Informatica's market place from the Hortonworks data platform. The nicest thing about it is that doesn't require the analyst or developer to know anything about MapReduce as s(he) just works within a graphical user interface and the software takes care of the implementation. The parser will parse web logs, call detail records, various providers of financial information (Bloomberg et al) and, more generally, XML and JSON (JavaScript object notation). The software will discover relationships and hierarchies within the data and flatten them.

I dare say that HParser will not be the last product to start to tackle governance within the BDDB space but it is the first.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 22nd November 2011 Copyright Bloor Research © 2011

Last week I was asked to attend a roundtable discussion on Big Data and network security. This is not my usual area of interest but the more I thought about it the more intrigued I was. We are experiencing a phenomenal explosion in the volume of data that is being created, and as the data volume grows so the nature of that data evolves. Whereas a decade ago only a small proportion of the available data was personal or proprietary, now over 40% falls into those categories. The danger of such data is obvious. I have had a considerable sum taken from my online bank account by people who purported to be my bank and knew all of the details that should have been used to protect me, so I am very aware of this threat.

The Big Data story is intimately linked to rise of the Internet, and it is via the Internet that we leave the digital trail that describes increasing aspects of our personal and business life. Unfortunately most of us are rather lulled into feeling that we are safe in cyber space. We see ourselves as just a small cog operating in a big complex, and that we are fairly anonymous, so we should have nothing to fear. The truth is very far from this benign image that we hold. Sourcefire research indicates that there were nearly 300 million new pieces of malware were created last year, and that nearly 75% of them were attacks only seen on a single system.

So our thoughts at the roundtable were to gain a balanced perspective between the huge potential benefit that Big Data raises, in understanding customer behaviour, creating new niche markets, streamlining business operations etc. and the need to recognise that the very richness of the data that affords that potential also represents too tempting a lure to the unscrupulous.

Hackers are sophisticated, and the rewards are great, and there is a lack of a basic universal framework around identity management and security to protect us. The threat comes not just from technical malcontents, but also from national governments who sanction industrial and political cyber espionage and perhaps in terms of the scale, if not the severity of the threat, to commercial enterprises who create dummy personal accounts on social media sites to seed their message in what the users assume to be an open and genuinely representative forum for the free exchange of views.

Of particular concern, we felt, was the sheer persistence of data that is lodged in the Net, and the lack of recourse that exists to those who feel that they have been misrepresented. We have all heard the tales of how a great number of generation Y kids will have to reinvent themselves in order to escape their teenage past immortalised on Facebook, and only last week there was a radio article about an individual who was falsely accused of inappropriate behaviour on a social network that had seriously damaged his business and how no one was prepared to do anything about it. This led us to explore the very real danger that the Net and the Cloud with Big Data analytics have to open us up to a very sophisticated form of long-term fraud. We are all entrusting all of our details to these big bodies. They claim that it is all secure, yet they keep slipping out snippets of information that show that they cannot resist the temptation to snoop. So far it has been fairly harmless, like telling us just how many of us use the most popular passwords, but what if someone were to exploit that trust to just go that bit further into exploiting what they know about us?

An interesting line of thought was to look at what, to one group, is surplus data that could be disposed of with relatively careless handling as being of limited worth, but which may be of great use to another group to exploit.

What we all agreed was missing was a lack of awareness at all levels, from individuals through enterprises to governments, that there is a real need to protect identity and establish credentials by both sides in all transactions. If you look at how I was impacted by fraud the callers knew exactly how my bank establishes my identity but does not allow me to do the same to them, and having established their fidelity they then control the relationship and the dialogue.

So was our view just wholly one of buyer beware? Well, actually, we felt that the tools do exist to deploy the techniques of the Big Data analytics to identify and protect with as much ability to generate fresh insight, as is true in the marketing domain. Through sentiment analysis a sense of who are likely to be seen as a target can be discerned. Using forensic analysis of digital trails the paths and behaviours of the would-be attacker can be identified. Statistical analysis can filter out the false positives, enabling the subset of the most likely attacks to be presented to the security analyst, not only reducing the volumes they have to deal with, but also providing a full range of supporting data to assist in their analysis. We felt that the threat to national security, the threat to brand value of commercial enterprises and the danger of the violation of personal identity are becoming far higher up the agenda of everyone and resources will be diverted into security.

Clearly Sourcefire are already at the forefront of thinking in this area and that is highly commendable. People will find that they can provide a valuable leg up to overcome the perceived barriers to knowing where to start on what, at first sight, appears a challenge of unprecedented scale and complexity. I found the afternoon very challenging and thought provoking, an interesting time awaits us all!

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 22nd November 2011 Copyright Bloor Research © 2011

Red Lambda recently came to market with a new SIEM (security, information and event management) product. Its distinguishing feature is that it has a CEP (complex event processing) engine at the front-end because, in the company's own words, "log and security data is a big data problem". I have been preaching this, not in so many words, for the last couple of years but the only other SIEM vendor that I know of that uses CEP is Tier-3, though I also know that SAS has been doing work in this area (for example, for real-time identification of 'low and slow' attacks, which is notoriously difficult).

More generally, the CEP market segment is characterised by two use cases: supporting algorithmic trading and other functions within capital markets, and supporting business process management (BPM) and improvement within SOA (service oriented architecture) environments. So Progress, Streambase and Sybase are in the former camp while Oracle, Progress (again) and TIBCO are in the latter group, along with IBM's WebSphere Business Events.

What is surprising to me is that, with the exception of IBM, none of the other vendors that have long-established CEP products seems to have recognised the wider truth in Red Lambda's statement: that CEP is the solution to many of the issues raised by big data, especially where that is instrumented data arising from sensors, logs (web, security or otherwise), smart meters, RFID, GPS or similar. That's not to say that they haven't dabbled with employing their products in other environments but they certainly aren't, at least to my knowledge, targeting 'big data' as a generic issue.

The key, to my mind, is to be able to support data mining techniques to build a relevant model (or models) and then score incoming events against that model in real-time. For example, Red Lambda has implemented a data mining technique that it calls Neural Foam, which is like a neural network except that it doesn't require training (note: in case you are not into data mining this is a seriously important feature). More generally, you would want to have support for PMML (predictive modelling mark-up language), which is the industry standard for porting data mining models: so you can build a suitable model in your warehouse, port it to your CEP engine and then, as I say, score in real-time.

This is what IBM's InfoSphere Streams does and, given Oracle and TIBCO's presence in the business intelligence space (in the latter case with Spotfire), then you might think that they would do so too. In particular, by continuing to focus on CEP as a corollary to BPM and SOA they are leaving the field wide open for IBM. In the case of Oracle this seems very un-Larry-like.

More particularly, IBM is implementing Streams to augment its data warehousing offerings. Not only are there Streams installations in conjunction with DB2, but IBM already has Streams being used in conjunction with Netezza. In other words, it is easy to conceive of situations where Streams might make the difference between a customer licensing Exadata and/or the Oracle Big Data Appliance on the one hand and an IBM database or BigInsights (which integrates with Streams) solution on the other. As I said, most un-Larry-like to hand a major competitor such an advantage. Still, I'm sure that IBM is happy enough to have the playing field to itself.

Finally, just to return to SIEM for a moment: what's the betting that Streams will be integrated with QRadar, which IBM has gained with its acquisition of Q1 Labs? If anyone wants to place a bet to the contrary, please let me know: I'll be happy to take your money. And that doesn't spell good news for any of the other SIEM vendors out there: Tier-3 and Red Lambda may be in the game but everyone else (unless they're quick) is going to be left with an out-of-date architecture. But then, I think that's where they are already.

By: Fran Howarth, Practice Leader, Bloor Research Posted: 18th November 2011 Copyright Bloor Research © 2011

Location-based mobile applications such as Facebook, Google and others are used by a large percentage of adults and teenagers. Applications that pinpoint a user's physical location introduce unprecedented new risks. The potential threats range from fraud and identity theft to crimes such as burglary or physical violence.

Geolocation is your physical location and is derived by technology using data from your computer or mobile device. It could relate to your physical location (position on the earth's surface) or the virtual (internet) environment. Both can be collected in many ways:

• Web browsing via your computer (IP[1] address is your identification)
• Mobile phone usage
• GPS (Global Positioning System) devices
• Credit/debit card transactions
• Tags in photographs and postings (Facebook and Twitter).

Location can be collected in an active or passive mode. The active mode is a user device that provides the Geolocation using software to determine the user's position by wireless, GPS[2] or by "request and response". The passive mode is server-based and determines the position via IP (internet protocol), 3G or 4G and wireless positioning.

What are the benefits location brings?

• To the Customer: optimal request routing or navigation, instant purchasing decisions (shopping, restaurants), nearest station or bus stop and social networking opportunities.
• To Business: targeted marketing, delivery and asset management, insurance risk management, logistics etc. The list is endless.

Location, combined with other personally identifiable information, can be used or abused. The capabilities of this technology empower social networking, support law enforcement, enable many mobile services and also provide a serious concern in the hands of criminals.

Location information can be seriously abused. For example, an individual who announces holiday plans or activities on a social networking site may be signalling to a criminal that their house is currently unoccupied, leading to a higher risk of being burgled, whilst more general personal information could be used in social engineering attacks against them.

For organisations, location information can lead to unwarranted surveillance of their current activities. An example could be tracking the location of a company's executives. This could provide its competitors with pointers regarding ongoing business negotiations, such as potential mergers or acquisitions. This could affect the organisation's brand and reputation, or even dent it financially if the competitor were able to scupper the deal. Organisations must also be wary themselves when using location-based services. They should be careful that information collected regarding the location of their employees does not constitute illegal tracking of their activities outside of business hours. In addition, any location-based services offered to customers or suppliers should take into account the privacy and ethical concerns of those parties.

In dealing with such risks, ISACA[3], which provides issues and guidance with regard to the governance, security and audit of information systems, cautions that the legal obligations of users and developers of geolocation data are currently unclear. In the absence of legal guidelines, it cautions that organisations need to carefully consider what controls are appropriate. These could be strong access controls and anonymisation techniques or the use of encryption for all personally identifiable information. It urges all organisations using geolocation to develop its own framework to address privacy and security locations, making use of existing information security frameworks such as CobIT[4].

How to safeguard yourself? We quote the ISACA recommends this 5-step practice:

1. Read your mobile application agreements to see what information you are sharing.
2. Only enable Geolocation when the benefits outweigh the risks.
3. Understand that others can track your current and past locations.
4. Think before posting tagged photos to social-media sites.
5. Embrace the technology, and educate yourself.

With such safeguards in place, you will be in a much better position to embrace the exciting benefits that are offered by geolocation technologies.

This article was prompted by the discussion within "Why geolocation apps can be dangerous" and the ISACA's new white paper, "Geolocation: Risk, Issues and Strategies."

[1] IP - Internet Protocol
[2] GPS - Global Positioning Systems
[3] ISACA - Information Systems Audit Control Association
[4] CobIT - Control objectives for Information and related Technology

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 18th November 2011 Copyright Interarbor Solutions © 2011

Most enterprises, service providers and governments have ramped-up their use of virtualization over the past several years, with many impressive results. Those paybacks can only continue, however, if the overall service and support of these complex and dynamic environments keeps pace.

The problem of effectively troubleshooting issues across virtualized data centers consisting of many products from many suppliers is daunting. But there's an added element. The stakes have never been higher for keeping applications and business up and running. Indeed, a businesses' IT systems are, increasingly, the actual business itself. It's hard to separate them.

HP has made the service and support of global virtualization market leader VMware implementations a top priority. Keeping virtualized servers that support mission-critical applications and databases at top levels of performance 24 x 7 is a much different problem than for maintaining physical servers in traditional configurations. [Disclosure: HP and VMware are both sponsors of BriefingsDirect podcasts.]

Indeed, advanced and pervasive virtualization and cloud computing trends are driving the need for a better, holistic approach to IT support and remediation. And, while the technology to support and fix these virtualized environments is essential, the people, skills and knowledge to manage these systems is perhaps the most decisive element of ongoing performance success.

Live discussion
To find out more, I'll be moderating a live deep-dive discussion on Dec. 7, with a group of HP experts to explore how to make the most of the available people, technology and processes to provide an insurance policy against failure.

Overall, you'll hear recommendations for how IT support can and should be done—even amid a rapidly changing IT landscape of virtualized, hybrid and cloud computing. First in the hour-long multi-media presentation is the inside story of how modern service and support works from one of HP's top services experts, Cindy Manderson, Technical Solutions Consultant for Complex Problem Resolution & Quality for VMware Products, who has 27-plus years experience with HP, and eight-plus years supporting VMware.

She will provide a short overview on the HP/VMware relationship and how the HP/VMware software support model uniquely enables always-on support for enterprises, service providers and governments. She’ll also present several case studies of how the HP Call Center global support process has solved problems in VMware environments.

After Cindy's chat, viewers will be invited to participate in the interactive questions and answer session with actual HP VMware experts. Moreover, both questions and answers will be automatically translated into 13 languages to demonstrate how service and support services know no boundaries, time zones or language barriers.

Leading these interactive sessions to answer the audience's questions live will be several additional HP/VMware support experts, including Patrick Lampert, a Critical Service Senior Technical Account Manager and Team Leader responsible for delivery and management of VMware Technical Services for Fortune 500 HP Custom Mission Critical Service Customers.

He'll be joined by Sumithra Reddy, Virtualization Engineer with HP Technology Services in the Global Competency Center, a 27-year veteran of software support, with a current focus on VMware. Other experts will join from Europe and Asia.

Register to reserve a place for this free HP Expert Chat on Dec. 7.

In summary, attendees will see how the breadth of virtualization is extending from servers to networks, desktop clients, storage, and mobile clients. All must operate in conjunction with the rest, especially as virtualized workloads come and go based on dynamic demand. This means that understanding how VMware and its ecosystem of vendors supporting these advanced environments relate. Problems in these environments must be solved from an over-view and neutral perspective, with all the interdependencies considered and managed.

So join the online presentation, discussion and question-and-answer sessions in nearly any major language worldwide. This is the first in a series of Expert Chats that I'll be moderating and that will tackle serious IT issues, with full global language support. Look for more more information on future discussions here.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 15th November 2011 Copyright Bloor Research © 2011

Since IBM acquired Informix there has always been the question of how Informix was positioned versus DB2. The answer that IBM has promulgated has been that Informix was IBM's offering to support embedded applications where you could fire up the database and forget about it because it essentially needs no administration. In other words, Informix has primarily been targeted at ISVs and software houses that were developing appropriate applications. Of course, this didn't preclude direct sales of Informix but this was the main thrust of IBM's marketing.

This has now changed, with the emergence of instrumented (breakthrough) applications offering new opportunities for Informix. In particular, IBM is targeting time-series based applications with Informix. There are many instrumented environments where data is time-stamped and it is important to understand the sequence in which data is collected. An example is smart metering applications but there are many others.

Now, Informix is just about the only database that I can think of that supports both operational (transactional) data and stores data in time-series format. There are, of course, a number of data and event warehousing products that support time-series analysis (Netezza and Sensage spring to mind but these certainly aren't the only ones) but they do not natively store time-series data.

Informix, on the other hand, has supported the storage of time-series data since the company acquired Illustra back in the 90s, via what is known as a DataBlade. Previously, these DataBlades (of which there are several) were chargeable extras but now they are bundled with the database.

So, IBM is specifically targeting environments that need to understand time-series. In addition, because Informix also has a spatial DataBlade, the database can also be used where time-series needs to be combined with spatial information. One might also imagine that IBM might also target pure spatial applications with Informix but this is not the intention or, at least, not yet.

The difficulty, of course, is that IBM Netezza supports time-series analytics and also has a strong spatial capability so there is once again an overlap. So the key is to understand that Netezza is targeted specifically at (deep) analytics whereas Informix will be most likely used for instrumented applications where a combination of analytics and operational functions is required. However, IBM will need to set out clearly and explicitly when it will be most suitable to use Informix and when Netezza should be preferred. There are also some differences in approach (ignoring the storage issue for a moment) but we understand that the plan is that these will be brought into line so that, potentially, you might have Informix front-ending onto Netezza.

Leaving aside this issue it is clear that Informix is emerging from under its bushel and this is not just in terms of new marketing directions but also with respect to the product itself. For example, there was the introduction of the flexible grid technology last year and the company is now re-selling 4JS' Genero (which provides modern functionality for Informix 4GL users, as opposed to the late and not lamented Informix NewEra product, which failed to do precisely that). It's good to see Informix back (even though it never really went away).

By: Gerry Brown, Analyst - Digital Marketing & CRM, Bloor Research Posted: 15th November 2011 Copyright Bloor Research © 2011

Most companies have not fully 'got' social media yet. Sure, they have a Facebook page, a Twitter account and a LinkedIn presence, but then what? Wait for the world to beat a path to your door? Run a few promotional campaigns and see few results? Wait for the CEO to ask "why have we only got X number of Facebook followers?"

Many brands have a somewhat passive, reactive approach to social media marketing. EngageSciences is an interesting UK start-up with a pro-active social media formula for businesses and brands wishing to attract new social media followers, and then to monetize that presence.

Firstly, you need to sign up a critical mass of fans. EngageSciences encourages its clients to create viral campaigns on Facebook and Twitter that give away something for free. Digital marketers might run prize draws, online sweepstakes, contests and quizzes, or offer exclusive premium downloadable content, such as a white paper, for example.

These incentives can prove surprisingly effective. For example, Unilever (Pot Noodles) increased their Facebook fans from 3,000 to 25,000 and Play.com increased from 40,000 to 80,000 Facebook fans within 8 weeks of using EngageSciences.

Sign-up requires an opt-in email address to be provided, so that the prize winners can be notified. Email addresses are critical pieces of digital marketing data. For digitally-savvy vendors, this contact data can then be cross-correlated with other databases and company web site visits; so that a pattern of personal behaviour can be detected that triggers further more personalised marketing communications.

Secondly, you need your fans to spend money with you. The best incentives are those that can easily 'go viral' i.e. are forwarded to friends, family and other followers. Typically these include redeemable coupons and 'flash deals' (time-constrained discount offers), such as the '2 for 1' deals used by Café Rouge and Strada restaurants, that have proved so effective. Exclusive ("offers only for you, dear fan") web page offers also work well.

As a recent IBM CRM survey proved, contra to conventional wisdom, for the most part, consumers don't really want 'relationships', and 'connection' and 'dialogue' with corporates. Consumers want offers, discounts and the ecommerce ability to purchase online. ExactTarget research reveals that Facebook consumers who are promiscuous (who "Like" a lot of brands) and those a little older (age 27+) want something of value in return for their "Like".

The EngageSciences Fan Relationship Marketing Platform provides all the technology required to push attractive offers into the key social media channels i.e. Facebook and Twitter, and capture followers' contact details for segmentation and online re-marketing.

EngageSciences' SaaS-based hosted solution is a relatively low-cost method of quickly creating a cloud-based social marketing database as a repository for follower demographic details, social behaviour, and engagement with online social media marketing campaigns. A 'test' campaign or a monthly subscription costs c. £1,000.

EngageSciences is an invaluable short-cut to market for marketers wanting to get started with executing social media marketing campaigns. Many marketers are fumbling with DIY approaches or 'big ticket' solutions that take too long to develop.

Consumers are receptive to creative online promotions now. ExactTarget research reveals that 45% of Facebook customers currently "Like" a company at least once monthly. Already the average US Facebook user "Likes" an average of 14 companies / brands. Consumer fatigue will set in over time, and fan or follower acquisition will become increasingly more difficult and expensive.

To date EngageSciences has managed to attract some impressive high calibre brands as users - Nokia, TNT and Forbes for example. One fact is clear. EngageSciences can help to re-energise a lethargic social media presence - and there are plenty of companies out there in that broad category.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 10th November 2011 Copyright Bloor Research © 2011

Hadoop sounds great but it has a number of issues associated with it. The first is that there are problems around high availability. In particular, Hadoop has a single NameNode. This is where the metadata is stored about the Hadoop cluster. Unfortunately, there is only one of them, which means that the NameNode is a single point of failure for the entire environment. If you don't mind that then fine but otherwise you will either need a much more expensive and robust server to house the NameNode or you will need to take an alternate approach: there are several of these. One is to go with a different distribution of Hadoop such as MapR, which fixes the NameNode problem. Or there are companies such as ZettaSet that have built additional tooling around Hadoop, including NameNode high availability, but which do not fork the Apache distribution. Or, since this NameNode issue is specific to HDFS (Hadoop distributed file system), you could replace this with IBM's GPFS-SNC, which similarly averts this problem. GPFS is also POSIX (portable operating system for UNIX) compliant, which HDFS is not.

Another associated problem is with the JobTracker. This is used to manage the MapReduce tasks and assign tasks to relevant servers (close to where the data is stored). Unfortunately, JobTracker, too, usually runs only on a single node, so it also represents a single point of failure. Fortunately, the same approaches that fix the NameNode issue will generally also handle JobTracker failures.

The second major issue is that MapReduce requires programming skills, and programming skills, as we all know, are in short supply. As a result there have been a number of developments to make things easier and to de-skill requirements. For example, there is Pig, which actually consists of PigLatin (the language) and a runtime environment that executes PigLatin code. This doesn't really help lessen the programming issue but Pig has been designed specifically for analysis purposes and takes away having to understand map and reduce functions. Next there is Hive. This provides a SQL interface to Hadoop. Unfortunately, it is limited: some functions are not available and some perform very poorly. Next, there is Jaql, which is a query language based on JSON (JavaScript Object Notation) and which was donated to the open source community by IBM. In its BigInsights product IBM offers an ANSI-compliant SQL interface that sits on top of Jaql.

In addition to this menagerie, some traditional vendors are exploiting MapReduce directly within their own products. For example, Syncsort, in the newly announced DMExpress 7.0, supports MapReduce functions directly from its GUI. In other words you can define a data integration task directly with DMExpress, using traditional drag-and-drop methods, and it will take care of the exploitation of MapReduce for you. This is great for data integration but unfortunately it doesn't help with query processing.

The third issue is that Hadoop and a number of associated products perform poorly. Again, various vendors have stepped into the breach. Thus MapR has re-written Shuffle so that it is 30% faster, while ZettaSet has made Pig multi-threaded. MapR has also made numerous other improvements and it estimates that these will halve your hardware requirements. Then again, Pervasive DataRush supports Hadoop clusters and it can be used either in conjunction with HDFS as an alternative to MapReduce or in conjunction with Hadoop, in either case providing significantly improved performance. Pervasive also has a product called TurboRush for Hive which the company claims improves the performance of Hive queries but with half the hardware. In internal benchmarks the product was out-performing native Hive (which you don't have to change) by a factor of three.

Finally, Hadoop is not an easy environment to manage. Not surprising really, when you consider that you might have hundreds of servers in a cluster. Both alternate distributions (MapR and so forth) and build-around products (ZettaSet, BigInsights et al) aim to help here, and there is also the ZooKeeper project from Apache, which provides synchronisation, configuration management and other cross-cluster services.

The bottom line is that there are a lot of considerations around Hadoop. It is by no means a mature environment and it is likely that you will require multiple additional products to make it work properly, especially if you go down the open source route. If you are happy to go commercial then you will probably need fewer such add-ons but then, of course, you will have to pay for them.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 10th November 2011 Copyright Bloor Research © 2011

IBM probably (almost certainly) has the largest portfolio of products spanning the big data space. It has DB2 (or Smart Analytics Systems), Netezza, Informix, Hadoop (with HDFS) and BigInsights (Hadoop with GPFS) to store the data or, if the data is too big and too fast you can use InfoSphere Streams as a complex event processing (CEP) engine. To query the data you can use Cognos, SPSS or BigInsights (which includes text analytics). Also, if you are using BigInsights you can use JasperSoft, as the latter has just announced a partnership with IBM.

The problem with this, and I haven't even started on data governance or integration - a subject for another day - is that there is too much.

When do you use DB2 or Netezza? The answer from IBM is that you use DB2 for operational analytics and Netezza for 'deep' analytics. In other words, Netezza won't support lots of users with short running queries from, say, a call centre. Well, okay, but that's a development decision: it was certainly something that Netezza was working on before the company was acquired and I have no doubt that it would have succeeded in satisfactorily supporting operational queries in due course.

When would you use Netezza versus Informix? The particular strength of the latter is in supporting time-series storage and analysis. Netezza only does the latter. So you would use Netezza if you are only interested in analytics and Informix if you want to build instrumented applications around time-series. But it's all a bit murky.

Hadoop with HDFS or Hadoop with GPFS? I've discussed this in a previous article: GPFS is more robust and should out-perform HDFS. So, when would you want HDFS? Probably only if you want to try stuff out to see if the whole idea is worth pursuing and you want to stick with free (open source) software.

As for Streams, I have no issues. In my opinion it's probably the best CEP engine on the market. And it supports PMML (predictive modelling mark-up language), which many others don't. I'll discuss this further in another article.

When it comes to queries again there are multiple products. The biggest question must be why JasperSoft? Not that I have anything against JasperSoft, it's a nice product, but why not Cognos? If IBM wants a query and reporting solution alongside BigInsights why didn't it develop something out of the Cognos stable?

Okay, so there are issues. But this is an emerging market. I'd rather have too many choices than too few. And it's not as if competitive vendors have a single product either. Moreover, there isn't really anyone competing on the instrumented data side of the house: either for event processing (Oracle, for example, still sees CEP as a part of its SOA offering rather than having anything to do with big data) or for instrumented applications, where both Informix and Netezza have a role to play. Overall, I think IBM has got its approach about right, though some product rationalisation might be nice.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 8th November 2011 Copyright Bloor Research © 2011

In the previous article in this series I discussed what big data is and talked about the ability to query ALL data that is relevant to the organisation. There are several ways to look at this issue. In that previous article I focused on where the data comes from: transactional, content, instrumented and external. However, there are other viewpoints to be considered as well, notably the type of data involved and the type of query capabilities that are available. These are closely linked.

There are various forms of structured data: transactional data is structured and you can access it via conventional query tools, analytics and SQL. XML-based documents are also structured and can be accessed via XQuery. Several vendors have extended SQL capability that allows XML documents to be queried alongside transactional data. Unfortunately, neither of these approaches are much use with unstructured data.

There are various other forms of structured data. For example, the data in a spreadsheet is structured: it's just that there is no metadata to describe what the rows and columns mean. Other examples are sensor-based, clickstream and log data. All of these are essentially structured. However, they are not really relational. They don't typically, for example, have the primary-foreign key relationships that are typical of relational data. For this reason something like Hadoop is well-suited for storing this sort of information simply because you don't need the complexity (and expense) of a full relational database. Nevertheless, historically, relational databases and relational access methods have been used with these sorts of data or else flat file systems.

Instrumented data is also, frequently, time-stamped. This has two different requirements: the ability to store time series data and the ability to analyse it. The former is clearly an advantage when it comes to supporting the latter. Nevertheless, a number of data warehousing products support analysis but very few relational databases (or any others of other types) store data this way. One of the few exceptions is Informix, which has supported time series since it acquired Illustra back in the 90s.

The other type of data is unstructured. Search is the historic way of querying this or, if you have built a suitable taxonomy, then you can do a more thorough analysis of content. However, if you want to analyse tweets, for example, then it is unlikely that you will have such a taxonomy, in which case you will really need to parse the data. Consider a 140 character product description: data quality tools can parse this description for things like colour, model number, number of pixels, horsepower, voltage, dimensions and other characteristics of relevant products. In other words, you can extract structured information from the text. You would really like to be able to do the same with tweets and, as it happens, Informatica has just released HParser, which is parser for a Hadoop. While this is the first such product (as far as I know) it surely won't be the last.

Leaving that aside, traditional products aren't very good at querying text even if they have text indexing, which products such as Sybase IQ do, which is precisely where Hadoop comes into play.

Finally, it isn't just a question of structured and unstructured data. Frequently it will make sense to combine the two. There are different ways you can do this. One is to use a business intelligence tool that uses an index-based approach for both structured and unstructured data. Examples are Endeca (Oracle) Latitude and Connexica. These will run on top of a standard relational data warehouse. The second, theoretical, possibility is to put all the data into Hadoop, but you probably wouldn't want to be without your data warehouse. The third option is to use a data warehouse that directly supports MapReduce, such as Aster Data (Teradata); a fourth would be to implement HBase (a column-oriented store) on top of Hadoop; and the fifth is to use a data warehouse linked via federation (companies like Denodo and Composite Software support this) or ETL processes (IBM, Informatica, Talend, Syncsort et al) to Hadoop. Most companies are opting for this last option, an approach I will discuss further in a forthcoming article.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 2nd November 2011 Copyright Bloor Research © 2011

This is the first of I don't know how many articles about "Big Data". I was going to start the first article by saying that big data is not the same as Hadoop but then I realised that I had better describe what Hadoop is first, as otherwise that statement wouldn't make sense. So you can take this as a sort of preface to the whole series.

Hadoop is a) an Apache Software Foundation development project and b) the original developer's son's stuffed toy elephant (hence the Hadoop logo and the menagerie of complementary projects named after animals). It has three bits to it: a storage layer, a framework for query processing and a function library. The last is not terribly important so I'll concentrate on the first two. There are also loads of things built on top of or around Hadoop (Hive, Pig, Zookeeper et al) that are not part of Hadoop per se - I'll come back to these in a further article.

The storage layer is HDFS (Hadoop distributed file system) and it stores data in blocks. Unlike a relational database, in which block sizes are typically 32Kb or less, in HDFS block sizes are, by default, 64Mb. This supports serial processing, as opposed to the random I/O that you find in a relational database. It is good for reading large volumes of data but useless for transaction processing or operational business intelligence.

Further, HDFS has redundancy built-in. Designed to run across a cluster of hundreds or thousands of low cost servers, HDFS expects those servers and their disk drives to break down on a regular basis. For this reason each data block is stored, by default, three times within a Hadoop environment. Note the implication of this: however much data you want to store you will require at least three times as much capacity as data (bearing in mind factors such as compression). This also has implications for processing functions, which are despatched to the servers on which the data resides rather than bringing the data to the processing, which is the norm in relational environments. A further point to note is that new data is always appended to what is already present: you cannot insert data.

OK, enough about HDFS for the time being, what about MapReduce? MapReduce is a programming framework for parallelising queries, which greatly speeds up the (batch - this is not suitable for real-time) processing of large datasets running, in conjunction with HDFS, on low-cost hardware. While it is usually thought of as having two steps (Map and Reduce) it actually has three. The first, the Map phase, reads the data and converts it into key/value pairs (a discussion of key/value databases will be the subject of a further article or articles). An example of a key/value pair might be "Name: Howard" "Employee number: 666" (just kidding) where Name is the key and Employee number is the value. Imagine that you had multiple employee files, then you would have a separate map task for each file.

Secondly, there is a process called Shuffle, which assigns the output from mapping tasks to specific Reduce tasks, which combine these results. All keys with the same value must be sent to the same reduce task.

There is obviously more to it than I have outlined here but these are the essentials. So, to return to the initial question: what is Hadoop? The short answer is that it is a support framework for MapReduce. The key word is "a": you can run MapReduce on Aster Data (Teradata) so you don't need Hadoop at all, you can deploy HBase, which is a column-oriented but non-relational database management system on top of HDFS, or you can deploy RainStor alongside HDFS, or you can replace HDFS with IBM's GPFS-SNC or there are various other options. Why you might want to do any of these things are subjects for another day but the important thing to remember at this stage is that the key element is the query processing provided by MapReduce rather than anything else.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 31st October 2011 Copyright Interarbor Solutions © 2011

The latest BriefingsDirect podcast discussion targets two major pillars of proper and successful data center transformation (DCT) projects. We’ll hear from a panel of HP experts on proven methods that have aided productive and cost-efficient projects to reshape and modernize enterprise data centers.

Learn about the latest trends buttressing the need for DCT and then how to do it well and safely. Specifically, we’ll delve into why it's important to fully understand the current state of an organization’s IT landscape and data center composition in order to then properly chart a strategy for transformation.

Secondly, we'll explore how to avoid pitfalls by balancing long-term goals with short-term flexibility. The key is to know how to constantly evaluate based on metrics and to reassess execution plans as DCT projects unfold. This avoids being too rigidly aligned with long-term plans and roadmaps and potentially losing sight of how actual progress is being made—or not.

This is the first in a series of podcasts on DCT best practices and is presented in conjunction with a complementary video series.

With us now to explain why DCT makes sense and how to go about it with lower risk, we are joined by Helen Tang, Worldwide Data Center Transformation Lead for HP Enterprise Business; Mark Grindle, Master Business Consultant at HP, and Bruce Randall, Director of Product Marketing for Project and Portfolio Management at HP. The discussion is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Tang: We all know that in this day and age, the business demands innovation, and IT is really important, a racing engine for any business. However, there are a lot of external constraints. The economy is not getting any better. Budgets are very, very tight. They are dealing with IT sprawl, aging infrastructure, and are just very much weighed down by this decade of old assets that they’ve inherited.

So a lot of companies today have been looking to transform, but getting started is not always very easy. So HP decided to launch this HUB project, which is designed to be a resource engine for IT to feature a virtual library of videos, showcasing the best of HP, but more importantly, ideas for how to address these challenges. We, as a team, decided to tackle it with a series that’s aligned around some of the ways customers can approach addressing data centers, transforming them, and how to jump start their IT agility.

The five steps that we decided that as keys for the series would be the planning process, which is actually what we’re discussing in this podcast: data center consolidation, as well as standardization; virtualization; data center automation; and last but not least, of course, security.

To make this video series more engaging, we hit on this idea of IT as superheroes, because we’ve all seen people, especially in this day and age, customers with the clean budget, whose IT team is really performing superhuman feats.

We thought we’d produce a series that's a bit more light-hearted than is usual for HP. So we added a superhero angle to the series. That’s how we hit upon the name of "IT Superhero Secrets: Five Steps to Jump Start Your IT Agility." Hopefully, this is going to be one of the little things that can contribute to this great process of data center modernizing right now, which is a key trend.

With us today are two of these experts that we’re going to feature in Episode 1. And to find these videos, you go to hp.com/go/thehub.

Gardner: Mark, you've been doing this for quite some time and have learned a lot along the way. Tell us why having a solid understanding of where you are in the present puts you in a position to better execute on your plans for the future.

Grindle: There certainly are a lot of great reasons to start transformation now.

But, as you said, the key to starting any kind of major initiative, whether it’s transformation, data center consolidation, or any of these great things like virtualization, technology refresh that will help you improve your environment, improve the service to your customers, and reduce costs, which is what this is all about, is to understand where you are today.

Most companies out there with the economic pressures and technology changes that have gone on have done a lot to go after the proverbial low-hanging fruit. But now it’s important to understand where you are today, so that you can build the right plan for maximizing value the fastest and in the best way.

When we talk about understanding where you are today, there are a few things that jump to mind. How many servers do I have? How much storage do I have? What are the operating system levels and the versions that I'm at? How many desktops do I have? People really think about that kind of physical inventory and they try to manage it. They try to understand it, sometimes more successfully and other times less successfully.

But there's a lot more to understanding where you are today. Understanding that physical inventory is critical to what you need to understand to go forward, and most people have a lot of tools out there already to do that. I should mention that those of you who don’t have tools that can get that physical inventory, it’s important that you do.

I've found so many times when I go into environments that they think they have a good understanding of what they have physically, and a lot of times they do, but rarely is that accurate. Manual processes just can't keep things as accurate or as current as you really need, when you start trying to baseline your environment so that you can track and measure your progress and value.

Of course, beyond the physical portions of your inventory, you'd better start thinking about your applications. What are your applications? What language are they written in? Are those traditional or supportable commercial-off-the-shelf (COTS) type applications? Are they homegrown? That’s going to make a big difference in how you move forward.

And of course, what does your financial landscape look like? What’s going in the operating expense? What’s your capital expense? How is it allocated out, and by the way, is it consistently allocated out.

I've run into a lot of issues where a business unit in the United States has put certain items into an operating expense bucket. In another country or a sub-business unit or another business unit, they're tracking things differently in where they put network cost or where they put people cost or where they put services. So it's not only important to understand where your money is allocated, but what’s in those buckets, so that you can track the progress.

Then, you get into things like people. As you start looking at transformation, a big part of transformation is not just the cost savings that may come about through being able to redeploy your people, but it's also from making sure that you have the right skill set.

If you don’t really understand how many people you have today, what roles and what functions they’re performing, it's going to become really challenging to understand what kind of retraining, reeducation, or redeployment you’re going to do in the future as the needs and the requirements and the skills change.

You transform, as you level out your application landscape, as you consolidate your databases, as you virtualize your servers, as you use more storage carrying all that great technology. That's going to make a big difference in how your team, your IT organization runs the operations. You really need to understand where they are, so you can properly prepare them for that future space that they want to get into.

So understanding where you are, understanding all those aspects of it are going be the only ways to understand what you have to do to get you in a state. As was mentioned earlier, you know the metrics of measurement to track your progress. Are you realizing the value, the saving, the benefit to your company that you initially used or justified transformation?

Gardner: Mark, I had a thought when you were talking. We’re not just going from physical to physical. A lot of DCT projects now are making that leap from largely physical to increasingly virtual. And that is across many different aspects of virtualization, not just server virtualization.

Is there a specific requirement to know your physical landscape better to make that leap successfully? Is there anything about moving toward a more virtualized future that adds an added emphasis to this need to have a really strong sense of your present state?

Grindle: You're absolutely right on with that. A lot of people have server counts—I've got a thousand of these, a hundred of those, 50 of those types of things. But understanding the more detailed measurements around those, how much memory is being utilized by each server, how much CPU or processor is being utilized by each server, what do the I/Os look like, the network connectivity, are the kind of inventory items that are going to allow you to virtualize.

I talk to people and they say, "I've got a 5:1 or a 10:1 or a 15:1 virtualization ratio, meaning that you have 15 physical servers and then you’re able to talk to one. But if you really understand what your environment is today, how it runs, and the performance characteristics of your environment today, there are environments out there that are achieving much higher virtualization ratios; 30:1, 40:1, 50:1. We’ve seen a couple that are in the 60 and 70:1.

Of course, that just says that initially they weren’t really using their assets as well as they could have been. But again, it comes back to understanding your baseline, which allows you to plan out what your end state is going to look like.

If you don’t have that data, if you don’t have that information, naturally you've got to be a little more conservative in your solutions, as you don’t want to negatively impact the business of the customers. If you understand a little bit better, you can achieve greater savings, greater benefits.

Remember, this is all about freeing up money that your business can use elsewhere to help your business grow, to provide better service to those customers, and to make IT more of a partner, rather than just a service purely for the business organization.

Gardner: So it sounds as if measuring your current state isn’t just measuring what you have, but measuring some of the components and services you have physically in order to be able to move meaningfully and efficiently to virtualization. It’s really a different way to measure things, isn’t it?

Grindle: Absolutely. And it’s not a one-time event. To start out in the field—whether transformation is right for you and what your transformations look like—you can do that one-time inventory, that one-time collection of performance information. But it’s really going to be an ongoing process.

The more data you have, the better you’re going to be able to figure out your end-state solution, and the more benefit you’re going to achieve out of that end state. Plus, as I mentioned earlier, the environment changes, and you’ve got to constantly keep on top of it and track it.

You mentioned that a lot of people are going towards virtualization. That becomes an even bigger problem. At least when you’re standing up a physical server today, people complain about how long it takes in a lot of organizations, but there are a lot of checks and balances. You’ve got to order that physical hardware. You've got to install the hardware. You’ve got to justify it. It's got to be loaded up with software. It’s got to be connected to the network.

A virtualized environment can be stood up in minutes. So if you’re not tracking that on an ongoing basis, that's even worse.

Gardner: Bruce, you’ve been looking at the need for being flexible in order to be successful, even as you've got a long-term roadmap ahead of you. Perhaps you could fill us in on why it’s important to evaluate along the way and not be even blinded by long-term goals, but keep balancing and reassessing along the way?

Randall: That goes along with what Mark was just saying about the infrastructure components, how these things are constantly changing, and there has to be a process to account for all of the changes that occur.

If you’re looking at a transformation process, it really is a process. It's not a one-time event that occurs over a length of time. Just like any other big program or project that you may be managing you have to plan not only at the beginning of that transformation, but also in the middle and even sometimes in the end of these big transformation projects.

If you think about these things that may change throughout that transformation, one is people. You have people that come. You have people that are leaving for whatever reason. You have people that are reassigned to other roles or take roles that they wanted to do outside of the transformation project. The company strategy may even change, and in fact, in this economy, probably will most likely within the course of the transformation project.

The money situation will most likely change. Maybe you’ve had a certain amount of budget when you started the transformation. You counted on that budget to be able to use it all, and then things change. Maybe it goes up. Maybe it goes down, but most likely, things do change. The infrastructure that Mark pointed to is constantly in flux.

So even though you might have gotten a good steady state of what the infrastructure looked like when you started your transformation project, that does change as well. And then there's the application portfolio. As we continue to run the business, we continue to add or enhance existing applications. The application portfolio changes and therefore the needs within the transformation.

Because of all of these changes occurring around you, there's a need to plan not only for contingencies to occur at the beginning of the process, but also to continue the planning process and update it as things change fairly consistently. What I’ve found over time, Dana, with various customers, as they are doing these transformation projects and they try to plan, that planning stage is not just the beginning, not just at the middle, and not just the one point. In other words, it makes the planning process go a lot better and it becomes a lot easier.

In fact, I was speaking with a customer the other day. We went to a baseball game together. It was a customer event, and I was surprised to see this particular customer there, because I knew it was their yearly planning cycle that was going on. I asked them about that, and they talked about the way that they had used our tools. The HP tool sets that they used had allowed them to literally do planning all the time. So they could attend a baseball game instead of attend the planning fire-drill.

So it wasn’t a one-time event, and even if the business wanted a yearly planning view, they were able to produce that very, very easily, because they kept their current state and current plans up to date throughout the process.

Gardner: This reminds me that we've spoken in the past, Bruce, about software development. Successful software development for a lot of folks now involves agile principles. There are these things they call scrum meetings, where people get together and they're constantly reevaluating or adjusting, getting inputs from the team.

Having just a roadmap and then sticking to it turns out to not be just business as usual, but can actually be a path to disaster. Any thoughts about learning from how software is developed in terms of planning for a large project like a DCT.

Randall: Absolutely. There are a lot of similarities between the new agile methodologies and what I was just describing in terms of planning at the beginning, in the middle, and the end basically constantly. And when I say the word plan, I know that evokes in some people a thought of a lot of work, a big thing. In reality, what I am talking about is much smaller than that.

If you’re doing it frequently, the planning needs to be a lot smaller. It's not a huge, involved process. It's very much like the agile methodology, where you’re consistently doing little pieces of work, finishing up sub-segments of the entire thing that you needed to do, as opposed to all of it describing it all, having all your requirements written out at the beginning, then waiting for it to get done sometime later.

You’re actually adapting and changing, as things occur. What's important in the agile methodology, as well as in this transformation, like the planning process I talked about for transformation, is that you still have to give management visibility into what's going on.

Having a planning process and even a tool set to help you manage that planning process will also give management the visibility that they need into the status of that transformation project. The planning process, also like the agile, the development methodology allows collaboration. As you’re going back to the plan, readdressing it, thinking about the changes that have occurred, you’re collaborating between various groups in silos to make sure that you’re still in tune and that you’re still doing things that you need to be doing to make things happen.

One other thing that often is forgotten within the agile development methodology, but it’s still very important, particularly for transformation, is the ability to track the cost of that transformation at any given point in time. Maybe that's because the budget needs to be increased or maybe it's because you're getting some executive mandate that the budget will be decreased, but at least knowing what your costs are, how much you’ve spent, is very, very important.

Gardner: When you say that, it reminds me of something 20 years or more ago in manufacturing, the whole quality revolution, thought leaders like Deming and the Japanese Kaizen concept of constantly measuring, constantly evaluating, not letting things slip. Is there some relationship here to what you’re doing in project management to what we saw during this “quality revolution” several decades ago?

Randall: Absolutely. You see some of the tenets of project management that are number one. You're tracking what’s going on. You’re measuring what’s going on at every point in time, not only with the cost and the time frames, but also with the people who are involved. Who's doing what? Are they fulfilling the task we’ve asked them to do, so on and so forth. This produces, in the end, just as Deming and others have described, a much higher quality transformation than if you were to just haphazardly try to fulfill the transformation, without having a project management tool in place, for example.

One thing that I would start with is to use multiple resources from HP and others to help customers in their transformation process to both plan out initially what that transformation is going to look like and then give you a set of tools to automate and manage that program and the changes that occur to it throughout time.

That planning is important, as we’ve talked about, because it occurs at multiple stages throughout the cycle. If you have an automated system in place, it certainly it makes it easier to track the plan and changes to that plan over time.

We have a lot of tools. One of the ones I want to highlight is the Data Center Transformation Experience workshop. And the reason I want to highlight because it really ties into what we’ve been talking about today. It’s an interactive session involving large panels, very minimal presentation and very minimal speaking by the HP facilitators.

We walk people through all the aspects of transformation and this is targeted at a strategic level. We’re looking at the CIOs, CTOs, and the executive decision makers to understand why HP did what they did as far as transformation goes.

We discuss what we’ve seen out in the industry, what the current trends are, and pull out of the conversation with these people where their companies are today. At the end of a workshop, and it's a full-day workshop, there are a lot of materials that are delivered out of it that not only documents the discussions throughout the day, but really provides a step or steps of how to proceed.

So it’s a prioritization. You have facility, for example, that might be in great shape, but your data warehouses are not. That’s an area that you should go after fast, because there's a lot of value in changing it, and it’s going to take you a long time. Or there's a quick hit in your organization and the way you manage your operation, because we cover all the aspects of program management, governance, management of change. That’s the organizational change for a lot of people. As for the technology, we can help them understand not only where they are, but what the initial strategy and plan should be.

You brought up a little bit earlier, Dana, some of the quality people like Deming, etc. We’ve got to remember that transformation is really a journey. There's a lot you can accomplish very rapidly. We always say that the faster you can achieve transformation, the faster you can realize value and the business can get back to leveraging that value, but transformation never ends. There's always more to do. So it's very analogous to the continuous improvement that comes out of some of the quality people that you mentioned earlier.

The workshops are scheduled with companies individually. So a good touch point would be with your HP account manager. He or she can work with you to schedule a workshop and understand that how it can be done. They're scheduled as needed.

We do hold hundreds of them around the world every year. It’s been a great workshop. People find it very successful, because it really helps them understand how to approach this and how to get the right momentum within their company to achieve transformation, and there's also a lot of materials on our website.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 26th October 2011 Copyright Interarbor Solutions © 2011

The issue of success or failure in moving your company data, IT storage, servers or software to the cloud is often driven by technical issues, including performance, bandwidth, security and total-cost-of-ownership (TCO) considerations. While many of these factors are key criteria for selecting cloud solutions, they usually don’t align with the bigger picture that C-level executives must consider when adding new IT solutions.

How IT can help sustain or create a competitive advantage has never been more apparent than today through the use of cloud computing. This technology boasts benefits such as reduced costs and scalability, just to name a few, but many companies fail to find the right fit for cloud within their business. [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

This is because cloud computing is not 'one size fits all'. Performance, network bandwidth, security, and total cost concerns can be allayed through a better portfolio and investment approach that considers the multitude of options available.

How cloud computing fits
In industries where working capital bears a high price and is in short supply, businesses often have to make ends meet and have limited investment available. Therefore, being able to source the lowest cost and drive efficiencies even further is critical to growing business and market share.

For companies with limited working capital resource or cash flow funds, the use of on-demand services becomes an attractive option for consumers to avoid upfront costs or maintenance of services. Likewise, companies seeking to provide better profitability from their operation and vendors managing their cost center can leverage on-demand models to target areas of their portfolio to reduce cost and maximize return.

When adopting cloud computing, companies are often driven by cost effectiveness, rather than looking at the bigger picture and asking what cloud solution is the best fit for the business. Cost savings, longevity of product, and performance aren’t mutually exclusive, and all should be factored into the decision-making process when researching and purchasing a cloud solution.

Here are four questions, which include key metrics and drivers, to ask when researching cloud solutions that will maximize the value of cloud computing for your organization:

1. Why is investment being spent on areas of IT that are not differentiating your business and can be commoditized?
   Key Metric: The balance of percent of investment on non-core commodity IT
   Key drivers: TCO needs to consider where to focus IT investment
   
2. How can IT grow and adapt with the ever-increasing expansion of data storage and the growth of computing demands eclipsing on-premise facilities?
   Key Metric: The cost of storage and archiving , recovery and continuity
   Key drivers: Latency of network and storage costs can be targeted through considering the whole IT portfolio, not just niche use cases of cost-performance. Look at the bigger picture.
   
3. How can access to new markets and new channels be better served through extending networks and partnerships?
   Key Metric: Size of markets and effectiveness of sales channels, both internal sales and external direct sales and reselling
   Key drivers: Total cost of acquisition can include the creation or use of third-party distributed marketplaces and self-service portals and platforms
   
4. Is your own IT fast enough to beat your competition or drive the cost savings or revenue and margin growth plans you need?
   Key metric: Speed of IT delivery and its cost and quality of service.
   Key drivers: Performance can be offered through selected service provisioning. Question whether all knowledge needs to be in-house. Skills can be as-a-service too.

Open Group Cloud Computing Book
The Open Group recently published Cloud Computing for Business –The Open Group Guide to address many of these key questions. The book is intended for senior business executives and practicing architects responsible for defining corporate strategy, and it identifies how to select and buy cloud computing services to achieve the best business and technical outcomes.

You can purchase a copy of the book from The Open Group here or download preview sections here:

• Cloud Computing for Business – Preview – first 30 pages
• Cloud Computing for Business – Section 1.8
• Cloud Computing for Business – Section 4.1

This guest post comes from Mark Skilton, Global Director of Strategy and Global Infrastructure Services, Capgemini, and Co-Chair of the Cloud Computing Work Group, The Open Group.