By: Neil Ward-Dutton, Research Director, MWD Advisors Posted: 3rd February 2012 This work is licensed under a Creative Commons License

Over the years I’ve been part of many conversations that revolve around how ‘BPM’ is not the same as ‘BPMN’ (in the context of process automation). The point consistently made is that even when you’re tackling work improvement scenarios that are suitable for modelling with BPMN (i.e. scenarios where the structure of work can be largely designed upfront), there are lots and lots of other important considerations you need to address before you can create a deployable process application that will automate all or part of a business process and help people work more effectively.

At the risk of repeating what you may already know, the design elements commonly needed within a process application but not addressed at all by BPMN include:

• Logic associated with providing integration links to back-end systems and data sources
• Task form and other user interface definitions
• Logic to define task management (task assignment, delegation, escalation)
• Specifics of calculations and other important rules and algorithms that are separate from the conditions you specify in BPMN gateways
• Definitions of performance monitoring models, KPIs, reports and dashboards.

This isn’t an exhaustive list, but even the items above add up to a pretty comprehensive set of things you need to deal with to get to a deployable process application.

Most of what I’ve heard in discussion around this point focuses primarily on implications for the time to deliver projects: in other words, don’t think that once you’ve created a BPM and model you're even close to a finished application for real-world deployment. However there is a bigger issue at stake here, which is: exactly what kind of provision a given BPM technology platform makes for the specification of those items in the list above—and, specifically, to what degree you’re encouraged to design and (when necessary) code these items so that each kind of concern is kept separate from all the others.

The quality of this "separation of concerns" in design might not make a huge amount of difference when you first start in implementation, but it can become incredibly important over time. And support for it turns out to be one of the most important (to my mind) differentiating points between BPM technology platforms.

Of course, because almost all BPM technology platforms centre implementation work around a graphical process model there is always likely to be a clean separation between definition of process and all of the other important design elements I’ve listed. But whereas some platforms provide a rich, well structured asset repository and clean design tools that implement the principle of "a place for everything, and everything in its place", other platforms really provide quite weak facilities of this kind. With this latter group of platforms, it’s still theoretically possible to create process applications that are relatively easy to maintain; but designers and developers are going to be pushing against the tools available rather than working with them.

Easy process application maintainability is of course one of the key parts of the BPM technology value proposition! Without the right tools, the cost and risk of managing and improving business processes in an operational environment just aren’t as easy to control as they should be.

It’s interesting to me that there’s very little attention paid to this issue in BPM technology vendors’ marketing literature; instead, vendors prefer to focus on sexy things like support for mobile devices, integration with social collaboration capabilities, cloud-based deployments and so on. When we examine BPM technology offerings in our detailed assessment reports, though, the architecture and philosophy of the of the toolset and platform in relation to application maintainability is one of the main things we dig into.

If you’re interested in finding out more about our assessment approach, you can get access to our assessment guide reports for free here and here. You can also see overviews of our most recent versions of our in-depth reports here.

What’s your view – how important do you think the principle of "a place for everything, and everything in its place" is in BPM implementation?

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 3rd February 2012 Copyright Interarbor Solutions © 2012

This guest post comes courtesy of Tony Baer’s OnStrategies blog. Tony is a senior analyst at Ovum.

To date, Big Storage has been locked out of Big Data. It’s been all about direct attached storage for several reasons. First, Advanced SQL players have typically optimized architectures from data structure (using columnar), unique compression algorithms, and liberal usage of caching to juice response over hundreds of terabytes. For the NoSQL side, it’s been about cheap, cheap, cheap along the Internet data center model: have lots of commodity stuff and scale it out. Hadoop was engineered exactly for such an architecture; rather than speed, it was optimized for sheer linear scale.

Over the past year, most of the major platform players have planted their table stakes with Hadoop. Not surprisingly, IT household names are seeking to somehow tame Hadoop and make it safe for the enterprise.

Up ' til now, anybody with armies of the best software engineers that Internet firms could buy could brute force their way to scale out humungous clusters and, if necessary, invent their own technology then share and harvest from the open source community at will. Hardly a suitable scenario for the enterprise mainstream, the common thread behind the diverse strategies of IBM, EMC, Microsoft, and Oracle toward Hadoop has been to, not surprisingly, make Hadoop more approachable.

What’s been conspicuously absent so far was a play from Big Optimized Storage. The conventional wisdom is that SAN or NAS are premium, architected systems whose costs might be prohibitive when you talk petabytes of data.

Similarly, so far there has been a different operating philosophy behind the first generation implementations from the NoSQL world that assumed that parts would fail, and that five nines service levels were overkill. And anyway, the design of Hadoop brute forced the solution: replicate to have three unique copies of the data distributed around the cluster, as hardware is cheap.

As Big Data gains traction in the enterprise, some of it will certainly fit this pattern of something being better than nothing, as the result is unique insights that would not otherwise be possible. For instance, if your running analysis of Facebook or Twitter goes down, it probably won’t take the business with it. But as enterprises adopt Hadoop—and as pioneers stretch Hadoop to new operational use cases such as what Facebook is doing with its messaging system—those concepts of mission-criticality are being revisited.

And so, ever since EMC announced last spring that its Greenplum unit would start supporting and bundling different versions of Hadoop, we’ve been waiting for the other shoe to drop: When would EMC infuse its Big Data play with its core DNA, storage?

Today, EMC announced that its Isilon networked storage system was adding native support for Apache Hadoop’s HDFS file system. There were some interesting nuances to the rollout.

Big vendors feeling their way
It’s interesting to see how IT household names are cautiously navigating their way into unfamiliar territory. EMC becomes the latest, after Oracle and Microsoft, to calibrate their Hadoop strategy in public.

Oracle announced its Big Data appliance last fall before it lined up its Hadoop distribution. Microsoft ditched its Dryad project built around its HPC Server. Now EMC has recalibrated its Hadoop strategy; when it first unveiled its Hadoop strategy last spring, the spotlight was on the MapR proprietary alternatives to the HDFS file system of Apache Hadoop. It’s interesting that vendor initial announcements have either been vague, or have been tweaked as they’ve waded into the market. For EMC’s shift, more about that below.

For EMC, HDFS is the mainstream
MapR’s strategy (and IBM’s along with it, regarding GPFS) has prompted debate and concern in the Hadoop community about commercial vendors forking the technology. As we’ve ranted previously, Hadoop’s growth will be tied, not only to megaplatform vendors that support it, but the third party tools and solutions ecosystem that grows around it.

For such a thing to happen, ISVs and consulting firms need to have a common target to write against, and having forked versions of Hadoop won’t exactly grow large partner communities.

Regarding EMC, the original strategy was two Greenplum Hadoop editions: a Community Edition with a free Apache distro and an Enterprise Edition that bundled MapR, both under the Greenplum HD branding umbrella. At first blush, it looked like EMC was going to earn the bulk of its money from the proprietary side of the Hadoop business.

What’s significant is that the new announcement of Isilon support pertains on to the HDFS open source side. More to the point, EMC is rebranding and subtly repositioning its Greenplum Hadoop offerings: Greenplum HD is the Apache HDFS edition with the optional Isilon support, and Greenplum MR is the MapR version, which is niche targeted towards advanced Hadoop use cases that demand higher performance.

Coming atop recent announcements from Oracle and Microsoft that have come clearly out on the side of OEM’ing Apache rather than anything limited or proprietary, and this amounts to an unqualified endorsement of Apache Hadoop/HDFS as not only the formal, but also the de facto standard.

This reflects emerging conventional wisdom that the enterprise mainstream is leery about lock-in to anything that smells proprietary for technology where they still are in the learning curve. Other forks may emerge, but they will not be at the base file system layer. This leaves IBM and MapR pigeonholed—admittedly, there will be API compatibility, but clearly both are swimming upstream.

Central Storage is newest battleground
As noted earlier, Hadoop’s heritage has been the classic Internet data center scale-out model. The advantage is that, leveraging Hadoop’s highly linear scalability, organizations could expand their clusters quite easily by plucking more commodity server and disk. Pioneers or purists would scoff at the notion of an appliance approach because it was always simply scaling out inexpensive, commodity hardware, rather than paying premiums for big vendor boxes.

In blunt terms, the choice is whether you pay now or pay later. As mentioned before, do-it-yourself compute clusters require sweat equity—you need engineers who know how to design, deploy, and operate them. The flipside is that many, arguably most, corporate IT organizations either lack the skills or the capital. There are various solutions to what might otherwise appear a Hobson’s Choice:

• Go to a cloud service provider that has already created the infrastructure, such as what Microsoft is offering with its Hadoop-on-Azure services;
• Look for a happy, simpler medium such as Amazon’s Elastic MapReduce on its DynamoDB service;
• Subscribe to SaaS providers that offer Hadoop applications (e.g., social network analysis, smart grid as a service) as a service;
• Get a platform and have a systems integrator put it together for you (key to IBM’s BigInsights offering, and applicable to any SI that has a Hadoop practice)
• Go to an appliance or engineered systems approach that puts Hadoop and/or its subsystems in a box, such as with Oracle Big Data Appliance or EMC’s Greenplum DCA. The systems engineering is mostly done for you, but the increments for growing the system can be much larger than simply adding a few x86 servers here or there (Greenplum HD DCA can scale in groups of 4 server modules). Entry or expansion costs are not necessarily cheap, but then again, you have to balance capital cost against labor.
• Surrounding Hadoop infrastructure with solutions. This is not a mutually exclusive strategy; unless you’re Cloudera or Hortonworks, which make their business bundling and supporting the core Apache Hadoop platform, most of the household names will bundle frameworks, algorithms, and eventually solutions that in effect place Hadoop under the hood. For EMC, the strategy is their recent announcement of a Unified Analytics Platform (UAP) that provides collaborative development capabilities for Big Data applications. EMC is (or will be) hardly alone here.

With EMC’s new offering, the scale-up option tackles the next variable: storage. This is the natural progression of a market that will address many constituencies, and where there will be no single silver bullet that applies to all.

By: Louella Fernandes, Principal Analyst, Quocirca Posted: 2nd February 2012 Copyright Quocirca © 2012

Nuance is a company with a plethora of products that cover the gamut of voice recognition, document capture and print management. Nuance has largely grown through acquisition (about 50 in the last ten years) so it is probably better known by its product names which include established brands such as PaperPort (desktop productivity), OmniPage (OCR), Dragon Dictate (voice recognition), eCopy (document capture and workflow) and Equitrac (print management) – its most recent acquisition. Overall, Nuance’s 2011 revenue reached $1.318 billion in 2011 with 2012 sales expected to reach $1.6 billion.  Boosted by its eCopy and Equitrac acquisitions, its imaging division growth has been strong, revenue reaching $177m in 2011 and expected to exceed $200m in 2012.

At its first European analyst event in London, Nuance discussed its strategic priorities for 2012, which include integration of its scan and print products and expansion of mobile and cloud delivery platforms. Nuance’s goal is to become the “MFP software standard” through delivering integrated cross-platform document capture and print management products – eCopy and Equitrac. Today, both products are well established, and Equitrac is already widely used to control and monitor print usage and costs across many verticals, with a particularly strong presence in the legal market – Nuance estimates that, globally, over 3,000 law firms use Equitrac. Its strong MFP and printer partner alliances mean Equitrac has long been used by major printer and copier OEMs such as HP, Ricoh and Xerox to provide enhanced multivendor print management capabilities for tracking, monitoring and reporting on scan, copy and print usage to their managed print services (MPS) customers.

This broadens the already strong OEM relationships on the eCopy side, including Canon, Konica Minolta and others.  With Equitrac, eCopy and its desktop products, Nuance has business relationships with nearly all major MFP, printer and scanner manufacturers worldwide.

Capturing the MPS opportunity
Nuance sees MPS as a key driver for its growth in the coming year and views the Equitrac and Nuance document imaging solutions as important components of helping MPS providers to succeed. Indeed there is rapid adoption - Quocirca research shows that around 45% of large corporates now have some form of MPS as they seek to reduce the cost and complexity of operating previously unmanaged printer fleets, typically characterised by a patchwork of devices from different manufacturers, with different consumables, paper, supplier and service requirements. Few organisations have the tools to track and monitor usage leading to spiralling print costs – both financial and environmental. Security is also an issue as all too often documents are left in output trays exposed to prying eyes.

MPS addresses these issues through three major phases – assessment, optimisation and on-going continuous management. Nuance’s Equitrac products have a strong part to play in all phases, helping organisations to not only reduce print wastage through tracking and reporting, but also enhance security, promote user mobility and reduce environmental impact. Key to this is Equitrac’s “Follow-You” or pull-printing which releases documents only upon user authentication – through either user PIN or smart card authentication. The results are compelling - Liverpool John Moores University discussed how they had saved £100,000 and reduced page volumes by 4.5 million per year through implementing Equitrac.

Nuance is also looking to address the largely untapped opportunity for MPS in the SMB market, via the reseller channel. Many resellers lack the resources or skills to deliver their own MPS, and are looking for a low-cost approach based on 3rd party platforms. Nuance intends to participate in this market which is seeing the emergence of cloud-based MPS offerings from vendors such as HP and Xerox. To capitalize on the emergence of cloud-based technologies and to support its partners’ Managed Services initiatives, Nuance will continue to expand its product portfolio (print management, capture and OCR) from on-premise deployments to off-premise (cloud) models. This will provide a set of cloud-based print management, document capture and OCR technology services to partners who wish to include them as part of their own managed services offerings. 

With the likes of HP and Xerox already having established cloud MPS platforms, Quocirca believes that Nuance will need to get these solutions to market quickly, particularly if it wishes to target the emerging ecosystem of independent MPS providers who will be looking for multivendor supported cloud-based services.

Quocirca believes that Nuance has product breadth, technical resources and channel reach to create a compelling set of enterprise cloud services around its eCopy and Equitrac products. However, given that both eCopy and Equitrac platforms have been gained through acquisition, Nuance still has some work to integrate them.

Talking to printers?
Given its heritage in speech recognition consumer technology, Nuance is uniquely positioned to apply this technology to enhance the printer and MFP user experience. The printer industry is far from immune from IT consumerisation, which continues to influence user expectations in the workplace. Whilst employees are used to the convenience, elegance and usability of tablets and smartphones, MFPs, in comparison, are in danger of becoming the elephant in the room.

Whilst most people are familiar with how to press print or copy, few users bother navigating complex nested menus to access finishing options or scan features. Businesses may therefore miss opportunities to minimise paper wastage through using features as duplex or booklet printing instead of single side printing. 

One technology that could improve the use of MFPs is voice recognition. Nuance has long been a leader in this field, and quietly provides back-end voice recognition functionality for Apple’s Siri. Could we in the future be telling our printers to print and staple 5 copies of a document – or scan and document and email it to a colleague? Yes - according to Nuance, the technology is already here to make it possible. It remains to be seen whether hardware vendors will embrace this opportunity to bring printers and MFPs into the 21st century.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 2nd February 2012 Copyright Interarbor Solutions © 2012

Good data management, analytics, and helping to shape the goals of the business are keys to transforming the enterprise through impactful enterprise architecture (EA). That was the theme, from different perspectives, presented by a series of plenary speakers this week at The Open Group Conference in San Francisco.

Jeanne Ross, Director and Principal Research Scientist at MIT's Center for Information System Research, opened Monday's plenary session, telling the attendees that the stakes are high for EA, which needs to show swift success in the new digital economy. Enterprise architects also now need to help their organizations better use new services and instill a "value cycle." [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

Coming from the siloed past in IT, companies are now moving to business service-driven processes across various resources, Ross said. But they need to recognize the forces around consumption of such services, not just the implementation.

Making good data management a priority, a "single source of truth" is also at the heart of making EA valuable, said Ross. Ensuring the quality of data and the speed of data refresh will help enterprise architects rise in performance appreciation more than just about anything else, she said. Ross studies how firms develop competitive advantage through the implementation and reuse of digitized platforms.

She is also the co-author of three books: IT Governance: How Top Performers Manage IT Decision Rights for Superior Results, Enterprise Architecture As Strategy: Creating a Foundation for Business Execution, and IT Savvy: What Top Executives Must Know to Go from Pain to Gain.

I also interviewed Ross on enterprise transformation issues before the conference.

IT-enablement isn't enough, Ross said, because companies typically under-utilize new systems and applications. It's not that we can't build them, she said of systems, but that companies aren't using them to their potential. Architects need to consider this and then market and evangelize solutions.

And EAs need to be more involved with making quality data center stage in their companies. "You don't get good analytics with bad data," Ross said, "The secret to good EA is to put information in every person's hands so they can use data better." And that in turn will help transform the business and spur added innovation using IT systems and good architecture principles.

Most senior executives aren't very good at combining business and technology strategies, Ross said, and she outlined the architect's elevated role in helping their bosses deliver increased business value:

• Help senior execs clarify business goals
• Identify architectural capabilities that can be readily exploited
• Present options and their implications for business goals
• Build capabilities incrementally

She closed out, getting applause from the audience, by predicting, "Some day CIOs are going to report to the enterprise architect, because that's the way it ought to be."

Impressive cost reduction
The second plenary speaker, Celso Guiotoko, Corporate Vice President and CIO of Nissan Motor Co, Ltd., told how business value is at the top of IT principles for Nissan, information as an asset comes next, and then reducing complexity.

Using these principles, Nissan in 2005 developed "BEST" as an IT mid-term plan and significantly improved the efficiency of its information systems. BEST is an acronym for business alignment, EA, selective sourcing, and technology simplifications.

This was followed in 2009 with the development of the "Change" program, which provided the basis for further advances by changing people, technology, and "process." And, in 2011, the next IT mid-term plan "VITESSE" was launched, designed to bring direct profit to the company. VITESSE encompasses value, innovation, technology, simplification, and service excellence. Through the various initiatives, Nissan has reduced IT cost by over 40 percent, going from a cost per user of $1.09 to $0.63.

The transformed enterprise
Andy Mulholland, Global Chief Technology Officer and Corporate Vice President at Capgemini, focused on the transformed enterprise and cloud trends, as well as the effect of new devices and social networking. Forty million tablets and 70 million smartphones are having a huge impact on how workers and consumers expect to work and shop.

The "bring your own device" phenomenon is forcing a change in thinking for enterprises, Mulholland said, as two environments are developing—inside IT and outside IT. Typically back-end activities operate inside the firewall, while front-end people and activities operate outside the firewall, yet people nowadays want to be able to use smartphones and tablets for both personal and work tasks.

This has led to a situation in which workers are increasingly going outside IT to buy services. Mulholland quoted a Gartner prediction that up to 35 percent of IT expenditures will be outside the IT department by 2015. Other industry analysts like IDC have placed the figure higher.

Because of this, IT faces a huge “re-integration project” to bring together the inside and outside services in a rational way, Mulholland said, adding that the transformed enterprise needs to focus on the productivity of people and innovative business models.

I interviewed Mulholland a few weeks ago and we delved even deeper into the cloud duality issues now coming to the fore of enterprise technology issues and planning. I was also intrigued by a Wall Street Journal piece today on how the US faces a new tech boom. It was aligned with much of what Mulholland was saying.

The key to doing this “re-integration project,” according to Mulholland, is governance, and the industry really lacks a good cloud governance model, meaning that many businesses are already in trouble. However, enterprises shouldn't let that get in the way of progress. Mulholland advised, "If business wants something radically different from you, don't try to stop it. Try to understand it and take control of it."

Driving IT transformation
Lauren States, Vice President and Chief Technology Officer, Cloud Computing and Growth Initiatives, IBM, emphasized that transforming the enterprise requires a huge emphasis on analytics, and a successful integration of analytics and IT.

States drew on IBM's decades-long journey of constant transformation, relying on business process excellence, values-based culture, and IT-enablement. This has led to $1.5 billion in IT savings since 2005 as well as avoiding over $20 million in expenses over five years with a private analytics cloud, she said.

According to States, CMOs are overwhelmingly underprepared for the data explosion and recognize the need to invest in and integrate technology and analysis and consider analytics as business differentiators.

CEOs and CIOs are both highly focused on insights, clients, and people skills, States said, feeding into what she called the "new reality," the need to harvest and pass insights and build trusted relationships.

States' takeaway: We're at the beginning of a major change, much like the PC revolution three decades ago. The cloud's sweet spot now, she says, is in bringing new innovation and insights to marketing, sales and customer service.

No need to wait
Speaker Bill Rouse, executive director, Tennenbaum Institute at Georgia Tech, said that many enterprises wait too long to change, with the decision to transform dragging on until the damage is beyond repair. As evidence, he said that in the past 25 years, 1000 companies have dropped from the Fortune 500 list—showing enterprise transformation has high failure rate, and that waiting for the right time change is a risky business plan.

Moreover, for those enterprises seeking transformation, they need to look at the full ecosystem that a business operates in to effectively transform, says Rouse. Business ecosystems are co-creating high-value services, expanding transformation across supply chains, says Rouse. This is an important nee dimension, he added.

Using analytics better to support evidence-based decision making is transformative and should be a priority, says Rouse. And architecture-oriented thinking can be transformative in itself, he said.

Cyber security threats
On the topic of cyber security, plenary speaker Joseph Menn, cyber security correspondent for the Financial Times and author of Fatal System Error: The Hunt for the New Crime Lords Who are Bringing Down the Internet, made it clear that business as usual won't do.

Joe has covered security since 1999 for both the Financial Times and then before that, for the Los Angeles Times. Fatal System Error is his third book, he also wrote All the Rave: The Rise and Fall of Shawn Fanning's Napster. I also recently interviewed him.

"It's in no one's interest to tell us how bad it really is" when it comes to cyber crime and security, said Menn. And the Stuxnet affair is huge as a harbinger of things to come, he said.

As a result, more taxpayer money will be needed for effective government-level defenses against cyber attacks, he suggested. But government intervention won't do the job alone. Increasingly, corporations will need to play more than just defense on attacks, many of which come from Russia and China and from groups that blend state and criminal interests.

Counter attacks may be a strong defense when it comes to cyber risks, and US government may "turn a blind eye", says Menn. We may even see cyber crime bounty hunters that corporations hire on the QT to go after those that attack them, he said.

Meanwhile, IT groups and enterprise architects can play a bigger role. Knowing what you have helps you know when something has been taken, so improve tracking of assets, Menn told them. He also suggusted that companies keep their most critical data offline, and protect their intellectual property by burying it in and among fake data.

Allen Brown, President and CEO of The Open Group, said that more than 400 corporations are now members of The Open Group, showing strong growth over past 12 years since its founding. TOGAF 9 certification rates growing rapidly worldwide, he said.

FACE standard
In other news from The Open Group on Monday, The Future Airborne Capability Environment (FACE) Consortium, announced the official release of the FACE Technical Standard, which provides guidelines for creating a common operating environment to support applications across multiple Department of Defense avionics systems. See my interview on FACE as it was just getting under way.

The standard is designed to enhance the U.S. military aviation community’s ability to address issues of limited software reuse and accelerate and enhance warfighter capabilities, as well as enabling the community to take advantage of new technologies more rapidly and affordably.The FACE technical standard will enable developers to create and deploy a wide catalog of applications for use across the spectrum of military aviation systems through a common operating environment. Product development efforts by industry and procurements by government customer organizations are already underway based on the FACE standard.

“The introduction of the FACE Technical Standard is an important milestone in extending interoperability among the armed forces and creating a common platform for avionics that enables systems to work together across each of the branches of the U.S. military,” said Brown.

And on Tuesday, The Open Group announced the arrival of ArchiMate 2.0, the latest version of the organization's open and independent modeling language for enterprise architecture. This version is more tightly aligned to TOGAF, so enterprise architects using the language can improve the way key business and IT stakeholders collaborate and adapt to change.

ArchiMate 2.0 improves collaboration through clearer understanding across multiple functions, including business executives, enterprise architects, systems analysts, software engineers, business process consultants and infrastructure engineers, according to the release. The new standard enables the creation of fully integrated models of an organization's Enterprise Architecture, the motivation behind it, and the programs, projects and migration paths to implement it.

"By combining TOGAF and ArchiMate, TOGAF becomes more easy to apply in any organization," said Harmen van den Berg, partner and co-founder at BiZZdesign. "Having a reference model makes them both easier to apply in any industry or vertical."

He added: "Architects like to make models, and this now helps them to use those models to create change in the organization, for something that means more to the business."

Making the EA function a chief weapon of enterprise transformation in a time of roiling change and complexity, that's the main message from the conference. No time to wait.

By: Dr Fern Halper, Partner, Hurwitz & Associates Posted: 31st January 2012 Copyright Hurwitz & Associates © 2012

Next in my discussion of big data providers is IBM.  Big data plays right into IBM’s portfolio of solutions in the information management space. It also dove tails very nicely with the company’s Smarter Planet strategy. Smarter Planet holds the vision of the world as a more interconnected, instrumented, and intelligent place. IBM’s Smarter Cities and Smarter Industries are all part of its solutions portfolio. For companies to be successful in this type of environment requires a new emphasis on big data and big data analytics.

Here’s a quick look at how IBM is positioning around big data, some of its product offerings, and use cases for big data analytics.

IBM
According to IBM, big data has three characteristics. These are volume, velocity, and variety. IBM is talking about large volumes of both structured and unstructured data. This can include audio and video together with text and traditional structured data. It can be gathered and analyzed in real time.

IBM has both hardware and software products to support both big data and big data analytics. These products include:

• Infosphere Streams – a platform that can be used to perform deep analysis of massive volumes of relational and non-relational data types with sub-millisecond response times.  Cognos Real-time Monitoring can also be used with Infosphere Streams for dashboarding capabilities.
• Infosphere BigInsights – a product that consists of IBM research technologies on top of open source Apache Hadoop. BigInsights provides core installation, development tools, web-based UIs, connectors for integration, integrated text analytics, and BigSheets for end-user visualization.
• IBM Netezza – a high capacity appliance that allows companies to analyze pedabytes of data in minutes.
• Cognos Consumer Insights- Leverages BigInsights and text analytics capabilities to perform social media sentiment analysis.
• IBM SPSS- IBM’s predictive and advanced analytics platform that can read data from various data sources such as Netezza and be integrated with Infosphere Streams to perform advanced analysis.
• IBM Content Analytics – uses text analytics to analyze unstructured data. This can sit on top of Infosphere BigInsights.

At the Information on Demand (IOD) conference a few months ago, IBM and its customers presented many use cases around big data and big data analytics. Here is what some of the early adopters are doing:

• Engineering: Analyzing hourly wind data, radiation, heat and 78 other attributes to determine where to locate the next wind power plant.
• Business:
  • Analyzing social media data, for example to understand what fans are saying about a sports game in real time.
  • Analyzing customer activity at a zoo to understand guest spending habits, likes and dislikes.
• Analyzing healthcare data:
  • Analyzing streams of data from medical devices in neonatal units.
  • Healthcare Predictive Analytics. One hospital is using a product called Content and Predictive analytics to understand limit early hospital discharges which would result in re-admittance to the hospital

IBM is working with its clients and prospects to implement big data initiatives. These initiatives generally involve a services component given the range of product offerings IBM has in the space and the newness of the market. IBM is making significant investments in tools, integrated analytic accelerators, and solution accelerators to reduce deployment time and cost to deploy these kinds of solutions.

At IBM, big data is about the “the art of the possible.” According to the company, price points on products that may have been too expensive five years ago are coming down. IBM is a good example of a vendor that is both working with customers to push the envelope in terms of what is possible with big data and, at the same time, educating the market about big data. The company believes that big data can change the way companies do business. It’s still early in the game, but IBM has a well-articulated vision around big data. And, the solutions its clients discussed were big, bold, and very exciting. The company is certainly a leader in this space.

By: Andy Tudor, Head of Multichannel Solutions, Retail Assist Posted: 31st January 2012 Copyright Retail Assist © 2012

ME commerce is a term recently coined to describe the coming together of mobile and e-commerce. The ethos is simple: virtual shopping is no longer limited to sitting in front of a PC or laptop, browsing and buying from retail sites, an activity traditionally referred to as e-commerce. With the advances of mobile/smartphone technology, ME commerce empowers consumers to make decisions to buy from multichannel retailers, wherever and however they choose. The retailer’s challenge - and opportunity - is to recognise that ME commerce is about ME, the consumer, and MY ability to be more demanding and shop ‘smarter’.

We live in a world where channels are multiplying and integrating at an impressive rate, so the retailer’s approach to underpinning an ME commerce offer is important. It is predicted that by 2014 there will be more mobile users than PC users. Mobility opens up a new dimension to retail, one which will fast become the norm if the 2014 statistic holds true. So how does the retailer adapt to this omnipresent channel of mobile commerce?

Most users carry mobile phones wherever they go. In fact, according to a Pew study conducted last year, two thirds of adults sleep with their mobile phone next to their bed. Among ‘millennials’ (young people aged 18 to 29), this figure rises to 90%. The Pew research further reveals that adults who use text messaging receive an average of 10 texts a day, rising to 50 a day in the case of most teenagers.

The potential for retailers to connect with consumers through mobile devices is significant. However, "with great power comes great responsibility". Retailers would be well advised not to annoy consumers with irrelevant marketing to their mobile phones or attempt to interact with them at inconvenient times. When used correctly, the immediacy of the mobile channel brings benefits to seller and purchaser alike; consumers gain access to the goods they want when they want, and retailers enjoy sales they would not have generated in the past.

There has been much debate about whether mobile commerce generates incremental sales or simply cannibalises existing sales. For consumers who are ‘glued’ to their mobile devices, facilities such as free Wi-Fi instore can encourage them to shop there and then. This may sound strange – after all, why would you want to order something online if it was there in front of you? Take the case of an item of clothing not being available in the required size instore; how much better it is to order it online without leaving the store than go home and perhaps forget to make the purchase.

It’s important to note too that a mobile phone is not only a tool for shopping ‘on the go’. It’s also a useful means for consumer and retailer alike to learn more about a product, whether that be scanning a barcode in store to get rich media product content sent to a phone, or searching the internet by phone whilst on the high street to find out which shop is holding stock.

Many consumers have preferred shopping channels. I, for example, prefer not to shop in-store but to order online from the comfort of my own home. My wife has a different approach. She enjoys the high street experience, browsing in several shops before making a purchasing decision.

Retailers need to accept that, whilst my wife and I are by no means exceptions to the rule, there is an important subset of consumers who are channel-agnostic. These customers are indifferent to the channel they use but do want to engage with the product and the service the retailer provides. In 2005, 97% of the consumers surveyed by the eCommerce Times said that they expected a channel-agnostic customer experience. I wager this percentage will have increased over the past seven years.

Even stronger than our preference for certain channels is our emotional attachment to brands, particularly those in the fashion sector. To capitalise on this, retailers should enable consumers to engage positively with their brands, irrespective of the channels they decide to use, in a way that is well managed and consistent throughout. Even though it is calculated that a consumer who shops across multiple channels is up to four times more valuable in monetary terms than one who shops via a single channel, creating a universal consumer experience remains of paramount importance to all retailers.

As retailers introduce multiple shopping channels, they should factor in the consumer’s ability to influence brand identity positively and negatively.  They have much to gain from mapping consumer interaction with each channel to market, putting in place support functions that give real-time visibility of cross-channel consumer interaction, and building a mechanism to capture feedback at the point of purchase.

At the end of the day, smart retailers will entice us to interact with their brands in many new ways. They may have already created a profile of your previous brand contact and purchasing habits. Having made a purchase in a ‘bricks and mortar store’, you may now have received an invitation on your mobile phone to download the retailer’s smartphone app or been directed to their mobile-optimised website. A friend’s Facebook profile may have prompted you to click through to a retailer’s profile page, where you found yourself able to purchase from within the social media site. In short, you may, without knowing it, already be part of an online community which has a common loyalty to certain brands.

To win our custom, retailers must evolve their IT strategy from 'multiple channels' to 'multichannel', by providing consumers with the same destination, no matter what vehicle the consumer uses to reach it.

Andy Tudor is the Head of Multichannel Solutions in the Aurora Client Services division of retail-only IT services and solutions provider, Retail Assist. Contact Andy at andy.tudor@retail-assist.co.uk.

By: Mark McGregor, Research Director, Bloor Research Posted: 31st January 2012 Copyright Bloor Research © 2012

In my recent article The Game of Process Improvement, I referred to a book called "Innovation Games". The book is packed with details on how any of us can leverage Innovation Games to gain greater insight into our customers and users. Something that is critical to the success of BPM and Process projects, but also can be applied to the vendors of these, and other products too. Last week I spend some time talking with Luke, the author, about the book, the games and his company.

When it comes to the use of games in business I am a firm believer that they should enable people to learn. As I mentioned in my previous article, we learn more by play than by analysis. Luke, though, is concerned about making sure people understand that his games are not seen as simply a learning tool, but a business tool directed towards delivering specific outcomes. Of course he still believes that they can and should be fun!

Luke has undertaken a lot of research into the linkages between the brain and productivity. In his words: "Productivity Games are not simply 'more fun' - they are literally more effective. This is due to the fact that the concept of play is deeply integrated into human beings' mental development."

"Studies tell us that there are parts of the brain that we do not access when we are simply discussing our views, or trying to think through a complicated situation. However, when we play a well structured game with other interested players, our actions, interactions with other players, and explanations of our behaviour can provide a better, more comprehensive view of how and why we make certain decisions."

Innovation Games, although a relatively young company, boasts an extremely impressive customer list. Companies, including Adobe, SAP, Aladdin, Wyse, Google and Qualcomm, have all leveraged Innovation Games to to improve holistic design thinking, discover new business opportunities, drive strategy and product road map decisions, improve the effectiveness of sales and service organisations, fine tune marketing messages, and create more intimate, durable relationships with customers.

One of the challenges that Luke has faced over the years is the stigma associated with the idea of using games in business. In part this is due to the mistaken understanding that games do not equate to work. This has led to him and others using the term "serious games", although he (and I) prefer the term that he also uses - "Productivity Games" - to try and overcome these obstacles.

I agree totally with Luke that the objective has to be to deliver objective, useable business outcomes, and Innovation Games amply delivers on this front. I also come from the perspective that, for effective change to take hold, then people need not just outcomes but the learning. The ability to come to their own "Aha!" or "Light Bulb" moment. So for me it is also about going back and seeing how people learn most effectively and, as we say, this is through structured play.

What I can see is that there is a need for people to be able to understand how to differentiate between unstructured and structured play. I can also see that even the most boring of analysis tasks can be made to be more fun through games. So perhaps we could use terms like "Strategy Games" or "Objective Gaming" to make it clear that in board game terms it is more like 'Risk' or 'Diplomacy' rather than 'Ludo' or 'Snakes & Ladders' - e.g. it is a game, should be fun, but is directed toward a targeted outcome. As mentioned in the previous article, there are many business leaders who have successfully grown businesses using their love of, and skill at, the game of chess to succeed.

Truly successful games need to deliver both concrete outcomes and learning. The outcomes ensure that you are making good use of your time and getting business value, while the learning ensures that your people continue to grow and develop. The great thing about the Innovation Games concept, as developed and promoted by Luke, is that it delivers on both counts. One only has to take a look in more depth at the success stories to see how much has been saved/made/changed to understand that the results are definitely there. If you take the time to talk with people who have been involved in those projects, you will hear them enthuse about learning things that they did not even realise were important.

Next month, during my trip to California, I hope to meet with Luke and learn first hand more about the way he and the team leverage Innovation games.

By: Fran Howarth, Practice Leader, Bloor Research Posted: 30th January 2012 Copyright Bloor Research © 2012

Cloud-based computing is growing faster than the IT sector as a whole. There are plenty of analysts throwing numbers about regarding cloud spending. Here are some from Forrester Research: in 2011, US$40.7 billion was spent on public, private and virtual private cloud IT services and that will expand to US$241 billion by 2020. Of that spend, US$21.2 billion was spent on software as a service, which will expand to US$92.8 billion by 2016 - 26% of all sales of packaged software applications.

One area that is showing particular growth is the market for email management and archiving. It is estimated that around 60% of business-critical data is transmitted via email, either in the body of the text or as attachments, and that information forms the basis of vital business records. All organisations are subject to regulations of some form or another and many of those regulations demand that business records be maintained. Those regulations vary widely from those applicable to specific industries, such as financial services or pharmaceuticals, to those affecting any organisation, such as employment and data protection regulations.

Being able to retrieve those business records when needed is not only vital for regulatory compliance, but also aids greatly in the productivity of workers and in responding to internal or external regulations. According to Osterman Research, 66% of IT organisations that it surveyed referred to email or instant message archives or backup tapes to support their organisation's case in litigation in 2010 and 63% were ordered by a court or regulatory body to product email or instant message records.

However, whilst the need to maintain business records is stark, technology vendor Proofpoint found in a recent survey that just 54% of large organisations in the US had deployed a technology solution for email archiving in 2010 and another survey, by GFI Software, found that that proportion fell to just over one-third of small and medium-sized companies.

Early email archiving technology tended to focus on the needs of specific types of companies, with financial services particularly well served. And, as email volumes continue to grow and grow, many felt that centrally archiving all emails was too complex a challenge. For large organisations in particular, scalability was considered to be an issue, and many others brushed the issue under a carpet. As a result, many organisations continue to rely on users storing emails on their own hard drives or using the email system itself as a storage repository. Neither is a good option as email records can be hard to find or even lost forever - especially if stored on a piece of equipment that is lost or stolen, which is a common problem with laptops and other portable media. A further issue to be considered is that many employees regularly use their smartphones for sending and receiving emails and those emails need to be captured for future use as well.

However, there is an alternative available that is suitable for any organisation, no matter its size or the regulatory burden that it faces - subscribing to cloud-based email management and archiving services. Such services take the cost and complexity out of managing email storage and provide ancillary services as well, such as business continuity and security. They are also highly scalable and suited to the demands of the mobile workforce.

According to Orlando Scott-Cowley, product marketing manager at cloud archiving vendor Mimecast, "Email archiving is going through the phases of its lifecycle. On-premise solutions are no longer scalable, have become too complex and don't really solve the email retention or litigation readiness problems that organisations have. Companies, whether regulated or not, are now turning to the cloud for their email archiving needs. Those that chose to deploy on-premise archives all those years ago are now finding they have the added complexity of migrating those solutions to more flexible and scalable cloud offerings. Setting their data free has become a bit of a nightmare, but their current on-premise vendors do not appear to be keen to wake them up from their bad dream."

Jon Pilkington, VP marketing and product management at cloud archiving vendor Sonian agrees, stating "Cloud-powered archiving provides a cost-effective, highly scalable solution for SMEs and enterprises alike. We view the cloud as a transformation service that is challenging the capital-intensive, on-premise models in use today, making email archiving accessible to companies of all sizes and verticals."

Email management and archiving are considered by many organisations to be among the most suitable applications for using cloud-based services as they are relatively uncomplicated and uniform. In December 2010, the US government unveiled its "Cloud first" policy, under which federal agencies must consider the option of using cloud-based services when planning new IT projects. In April 2011, the White House CIO stated that 15 agencies had announced that they intended to move their email management and archiving applications into the cloud. Two agencies - the General Services Administration and the Department of Agriculture - claim to have saved some US$40 million by abandoning in-house email. Building on this, the US government announced in November 2011 that all federal agencies have until May 2012 to report on how they intend to improve the way that they store and manage electronic records including emails, blog posts and social media activity, and the White House, in conjunction with the National Archives and Records Administration, is currently drafting a new records management directive. Using cloud-based services is considered by many to be the best option.

Other governments are following this lead. The UK government has stated that cloud computing should account for half of its IT spend by 2015 and it is hoped that this will reduce its annual IT expenditure of £16 billion by £3.2 billion.

Organisations that follow suit and embrace the cloud for email management and archiving will find that there are many benefits from doing so, not least of which is the peace of mind that business records will be securely preserved and can be easily retrieved as and when necessary. Emails are among the most requested documents as evidence in lawsuits and the courts no longer accept the argument of technical difficulty when dealing with legal issues surrounding email management and archiving. With cloud-based services, the burden and cost is taken out of the hands of the organisation and placed in those of specialists. For a competitive overview of some of the main players, click to download this document: Email archiving best practices.

By: Simon Holloway, Practice Leader - Process Management & RFID, Bloor Research Posted: 27th January 2012 Copyright Bloor Research © 2012

When I first wrote the RFID market overview, one of the key things I identified was that RFID hardware couldn't exist without RFID middleware and applications, and neither could RFID middleware and applications exist without RFID hardware. What has also become clear is that no longer are organisations just looking at passive or active tags, what they want is for their RFID middleware and applications to be able to work with a mix of different tags, both active and passive, and even at different frequencies. It is a case of choosing the right horse for the course!

On January 12th, Zebra announced they have entered into a "cooperative relationship and licensing agreement" with Checkpoint Systems. This relationship brings together Zebra's active location solutions with the passive RFID, auto-ID, Wi-Fi and sensor capabilities of Checkpoint division OATSystems' OATxpress middleware. The objective is to provide increased visibility of assets across an enterprise. The agreement is a non-exclusive contract and provides Zebra with an OEM software license for OATxpress

A reminder for those of you who are not sure about the two organisations involved. Zebra is one of the leading suppliers of bar code, receipt, card, kiosk and RFID printers and supplies, as well as real-time location solutions. Over the last year or so they have also developed a real-time location solution (RTLS), WhereNet ISO/IEC 24730-2. This provides robust location performance both indoors and outdoors with a long tag to sensor range. WhereLAN III RTLS tag delivers 1 meter locating accuracy, lower deployment and ownership costs, lower power consumption, and 802.11 b/g Wi-Fi backhaul.

Checkpoint Systems is a leader in shrink management, merchandise visibility, apparel labeling and asset tracking solutions. Checkpoint has some 40 years of experience of RF technology and shrink management requirements. In 2008, Checkpoint Systems acquired one of the leading RFID middleware companies, OATSystems (see OATSystems acquired by Checkpoint). This strengthened their RF capability and RFID customer base and has allowed OATSystems, as a division of Checkpoint, to further develop supply chain, manufacturing and inventory management applications on top of their RFID middleware for a number of verticals ranging from Apparel to Aerospace.

So what we have with this agreement is that Zebra can now offer Checkpoint's OATxpress device and data management capabilities in conjunction with their WhereNet RTLS solution. This makes it easier for a potential customer to purchase a complete solution from one point. From Checkpoint's viewpoint it gives access to Zebra customers and to the Zebra partner network thus providing further global access. From Zebra's viewpoint it can be summed up by a quote from Phil Gerskovich, senior vice president, new growth platforms at Zebra Technologies, "The addition of OAT's passive RFID and other auto-ID technologies capabilities will enable Zebra to play a larger and more meaningful role in helping organizations to make smarter decisions in managing their operations." Zebra has stated that they will announce details around its first product with the capability to implement applications that combine both active and passive RFID in the coming months, so watch this space!

In my view this relationship makes perfect sense to everyone and, most importantly, to potential and existing customers of Zebra and Checkpoint Systems.

By: Peter Williams, Practice Leader - IT Infrastructure Mgmt., Bloor Research Posted: 26th January 2012 Copyright Bloor Research © 2012

Thanks to x86 and other "commodity" components, users usually get good value for money from their server systems. However, storage subsystems and their connectivity remain stubbornly complex and costly. This has got to change - and for some this has started to happen.

When I wrote about Coraid last summer, I wondered about storage users' familiarity with ATA over Ethernet (AoE), especially when running Windows for which is fairly new. Coraid is belatedly betting its shirt on AoE which can be blisteringly fast. It developed the "protocol/compute layer" as far back as 2004 then donated it as open source to the Linux community, not then realising what it had; AoE has been part of the Linux kernel since 2005.

Coraid's progress in the past year has been solid - a user base of around 1,500 customers compared with 1,000 a year ago - and expansion in Europe from its UK office as well as a small start in APAC. In its storage arrays, it has now added flash to mix and match with SAS and various SATA drives (in a choice of RAID levels or JBOD) in the SAN.

But CEO Kevin Brown has a bigger vision for its AoE approach - which is to drive a commodity pricing revolution for storage. "Storage is like a mainframe when rest of system has become commoditised" Brown told me. This contrasted with servers long commoditised with x86 and, for instance, a 10GbE port could cost under $500. (EMC and NetApp may not take this too seriously but could perhaps be caught short, especially at the low-end.)

Using Coraid's AoE technique, the SAN (theirs or another's), appears to VMware of Hyper-V as a single direct-attached storage (DAS) drive - as a single SCSI controller "except that it has latency under one second" Brown added. Then the Coraid architecture covers the whole network, both initiating and receiving the storage data requests that cross the Ethernet wires. This, he said, removed several layers of complexity and was simple to set up, not least because everyone's server has long provided DAS connectivity.

The potential from this is manifesting itself in several ways that also help define Coraid's roadmap for the coming months. For instance, Brown wants to complement AoE by making everything simple. So Coraid is aiming for an end-user on-screen Q&A to define the storage need without concern for LUNs or HBAs - from which the software automatically tailors all the configuring and manages day-to-day operation. This helps explain the company's purchase of cloud orchestration software vendor Yunteq late last year as, for instance - it brought in a policy engine; the first fruits of this may appear mid-year.

In turn, this reflects another trend Brown told me he was witnessing. More and more internal and external clouds are appearing - and VARs were realising that, as well as reselling AoE, they could also start to offer cloud services of their own. However, that, in turn, put pressure on Coraid to automate its storage tiering management and to make sure security was sufficiently granular and robust (although the VARs can already provide complementary third party solutions). On the positive side, its own operating system and architecture was designed from scratch as distributed and scale-out, allowing very rapid expansion, as may be needed in clouds.

This week, Coraid announced a technology alliance partnership with Veeam. Veeam complements Coraid's EtherDrive and EtherFlash with innovative data protection, recovery, DR and management for virtual data centres - that is designed from the ground up for virtualised server environments. Recovery can be for a whole VM or individual file or application item.

It has also not been lost on Coraid that, by controlling storage data traffic end-to-end across the network, it is well placed to provide useful performance measurement software. That could be another nice little earner (but no date has been set for that yet).

Brown sees little value in chasing after conservative enterprises with wall-to-wall Fibre Channel (FC) storage connectivity but everyone else is ripe for this AoE-based storage commoditisation to cut costs and boost performance. The coming year should be enough to confirm whether AoE is likely to break through to make commoditisation a reality.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 25th January 2012 Copyright Bloor Research © 2012

There have been a couple of things floating around in the ether about CEP (complex event processing) recently. The first is the question, supposedly credited to Curt Monash, of whether it should be called something different.

I've been going back through my records. When I first wrote a product evaluation of what is now Progress Apama in 2002, I stated that "the company's contention is that conventional approaches to real-time queries are only suitable for small scale environments or those in which limited numbers of data feeds are being monitored. In particular, its view is that these solutions cannot cope with environments where large numbers of data feeds need to be combined in a complex and dynamic fashion." That was the only use of the word "complex" in a total of nearly 3,000 words. Events were mentioned several times, streams not at all.

As an aside, look at Wikipedia and other sources about the development of CEP and you'll see lots of mentions of David Luckham, who coined the CEP term in his book "The Power of Events", published in 2001. You will also see various other attributions to American scholarship but no mention at all to Cambridge (that's UK not Harvard), which is where Apama came from. I guess all the writers are American.

Anyway to get back to the subject, I wrote the following in our report on CEP, published in 2006: "the subject under discussion is frequently referred to as either complex event processing (CEP) or as event stream processing (ESP). We believe that both of these names are misleading: the former suggests that the technology is not also suitable for processing simple events, while you could infer from the latter (processing streams) that this was only about high performance. We prefer event processing as a neutral term to cover all of these possibilities." Frankly, I gave up this argument years ago.

The second piece of discussion that has recently hit the blogosphere is from Chris Carlson at Informatica. He is suggesting, quite rightly, that CEP isn't simply about real-time processing and that, in fact, it is misleading to refer to it as such. Again, from our 2006 report, "event processing is suitable for use in a wide range of diverse environments. Some of these are more about event streaming (low latency) and some are more about complex processing and some potentially both" and "what event processing does is to reduce the data latency, insight latency and, sometimes, the decision latency involved in taking action when compared to traditional approaches." In other words, CEP is about processing real-time data - it isn't necessarily about making real-time decisions.

Finally, I want to make a point about big data and CEP. Back before Christmas I wrote about how only StreamBase and IBM, of the major vendors, are currently, as far as I can tell, targeting their products at general-purpose operational intelligence environments as opposed to specific areas such as capital markets and security services or applications environments such as SOA. I did mention that SAS will be bringing out a product later this year and that there is also Darkstar from Cloud Event Processing (CEP - ycch!) which, naturally, runs in the Cloud. What I didn't mention was that there are a number of companies/products that have been specifically designed to work in conjunction with Hadoop, namely HStreaming, S4 (from Yahoo!) and Storm (from Twitter). I haven't looked at any of these in detail so can't comment on them but that is definitely an area that is heating up.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 25th January 2012 Copyright Bloor Research © 2012

I am continuing to investigate Hadoop storage options as I get briefed by more vendors and as new products get released. In this article I want to focus on Cassandra.

DataStax is the leading commercial provider for distributions of Cassandra, which is a BDDB (big data database). However, unlike HDFS (the standard storage mechanism for Hadoop) or GPFS (IBM's alternative) Cassandra is not a key-value store but a column-family store. This is not to be confused with a column-based relational database such as HP Vertica or ParAccel. In fact, it is unfortunate that whoever thought of the name "column-family" didn't think of something else. The point is that while Infobright and Sensage (more columnar relational databases) and Cassandra all use columns, this is the limit of their similarity: the former two are relational and Cassandra isn't.

I don't intend to go into the details of column-family databases and how they are architected. At least not now. But the main difference between a column-family database such as Cassandra and a key-value data store such as HDFS is that the latter stores just a key and a value while the former stores tuples that consist of a name, a value and a time stamp. It is this last that makes a big difference: there are lots of environments - smart metering, security logs and so on - where understanding time series is important and this means that Cassandra can support applications that Hadoop cannot. Not surprisingly, DataStax is exploiting this capability. Thus, for example, you can either store timestamps as the order in which they arrive in the database or as the order in which the events actually occurred (which may not be the same thing). You can also index against the timestamps and, indeed, the software supports secondary indexes as well. One further notable feature is that DataStax has introduced CQL as a query language, which is a subset of SQL, although you can't do such things as joins, because there are no tables.

In so far as Hadoop is concerned you can implement Hadoop and Cassandra on the same cluster. This means that you can have your time-based and real-time applications (real-time being a strength of Cassandra) running under Cassandra while batch-based analytics and queries that do not require a timestamp can run on Hadoop. In practice, in this environment, Cassandra replaces HDFS under the covers but this is invisible to the developer. You can reassign (dynamically where appropriate) nodes between the Cassandra and Hadoop environments as is appropriate for your workload. The other major upside is that using Cassandra removes the single points of failure that are associated with HDFS, namely the NameNode and JobTracker, which I have discussed in previous articles.

One final point is that Cassandra has a reputation for being difficult to get started. In order to simplify this process, DataStax is providing installers, examples and so forth within its Community Edition, while the Enterprise Edition, amongst other things, includes a visual point-and-click, web-based management environment that integrates with third party environments such as Tivoli and OpenView.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 24th January 2012 Copyright Interarbor Solutions © 2012

Scribe Software, a customer relationship management (CRM) data integration provider, will launch next week Scribe Online Synchronization Services (SYS), the second major service delivered on the Scribe Online cloud integration platform.

According to the Manchester, NH-based company, Scribe Online provides a cloud-based alternative to integration middleware, and simplifies the integration experience without sacrificing performance or functionality. The goal is to allow companies to reap the benefits of integrated CRM data from a variety of sources and technologies in days, rather than months.

The timing is more than pretty good because CRM as a category is expanding, driven by businesses' recognition that rich data on customers (and partners) is essential for better productivity, and for leveraging cloud-enabled business innovation outside the company.

Many companies I speak with are looking to pull appropriate and relevant data in near real-time from many internal systems of record to augment the full picture of customers. They are looking to their CRM systems as the meta data repository of such integrated views. And now they want to bring in more data from more sources, including those outside their four walls.

And, of course, the power of knowing the most about customers—and making the analysis from such data widely available to business units and functions across the enterprise—can make or break a company. Across the full business cycle, relevant and insightful data on customers drives success, from product development to effective marketing, to help desk and support, to entering new markets.

Scribe then, has developed its cloud offerings, built on Microsoft Azure and released last year, to make the instantiation of CRM data from as many sources as makes sense a function of the cloud, as well as on-premises. Such a hybrid approach to data integration makes even more sense than a hybrid approach to IT infrastructure services, if you ask me. You really need to be in the cloud to leverage the hybrid data integration benefits.

Now, Scribe has made it easier to leverage that cloud by offering synchronization services for CRM data integration a drag-and-drop affair that many business users can accomplish. Furthermore, Scribe is releasing SPARK, a developer program to help foster a community effort around making more connections to more types of data available to more synchronization efforts.

“Synchronization Services builds on our commitment to deliver superior CRM integration to customers and partners in the cloud. SYS fills a void in the market for an integration tool that is affordable and easy to use,” said Lou Guercia, president and CEO of Scribe. “Until now, integration products have been either too basic or too complex.”

Developer program
Scribe, with the SPARK Solution Developer Program, is targeting software-as-a-service (SaaS) providers, channel partners, systems integrators, VARS, and other business technology consultants. This means that while enterprise IT departments are gearing up for hybrid cloud-based CRM integrations, the community of ISVs and VARs needs to move more quickly, to innovate and expand into new models.

The SPARK Solution Developer Program is designed to help solution providers quickly build data integration capabilities between their solutions and CRM, as well as any other application or endpoint on Scribe Online. This will fit very well, too, into the Salesforce.com ecosystem, and the Microsoft Dynamics one, as well.

Scribe expects that partner networks will share and extend customer data—and value-added services on top of that joined and integrated data—for a variety of additional business services, said Guercia. Integrated and automated marketing services providers like HubSpot, Marketo, and Eloqua, certainly come to mind, too.

“CRM is no longer just a contact management system. It’s a critical revenue enabler for the business. Companies that integrate customer data from all areas of the business benefit with increased sales and satisfied customers,” said Roger Hodskins, vice president of strategic alliances at Scribe.

Using Scribe's latest offering, SaaS independent software vendors (ISVs) who offer integration to more than one CRM vendor can extend their presence in multiple CRM markets. As customers expand the scope of CRM in their businesses, integration can readily incorporate the SaaS ISVs’ offerings with connections both to CRM and to other complementary applications, said Scribe.

For more information on Scribe SYS, sign up for live weekly webinars, or to watch a four-minute demo video at scribesoft.com/online. Scribe Online SYS is available, too, free for 15 days at scribesoft.com/Free-Trials.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 23rd January 2012 Copyright Interarbor Solutions © 2012

Advanced and pervasive virtualization and cloud computing trends are driving the need for a better, holistic approach to IT support and remediation.

And while the technology to support and fix virtualized environments is essential, it’s the people, skills, and knowledge to manage these systems that provide the most decisive determinants of ongoing performance success.

In a special BriefingsDirect sponsored podcast, created from a recent HP Expert Chat discussion on best practices for VMware environment support, HP experts explain how they have made the service and support of global virtualization market leader VMware a top priority.

For example, Cindy Manderson, Technical Solutions Consultant for Complex Problem Resolution and Quality for VMware Products at HP, provides case studies for how managed escalation and multi-vendor support around the globe can reduce downtime by 70 percent, with large ROI benefits as well.

Other HP experts in the discussion include Pat Lampert, Critical Service Senior Technical Account Manager and Team Leader, as well as Sumithra Reddy, HP Virtualization Engineer. The discussion is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: HP and VMware are both sponsors of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: Virtualization isn’t just server-by-server, but really impacts the entire data center. You need to think about it more holistically, particularly in regard to things like security, performance and how your brands and businesses are perceived across the globe. Many of the companies that I deal with day in and day out are up at 80 percent and even 90 percent virtualized.

When they think about virtualization, they go beyond just server virtualization. It’s really now looking at storage, applications, networks and even the end-user desktop experience, or desktop as a service (VDI).

Successful virtualization is no longer just about servers, it’s about managing complexity when you get beyond the 20 percent or 30 percent level and expand into converged infrastructure virtualization without failures.

So how to take advantage of the best things about virtualization? Part of that means allowing your IT team to have access to other experienced support teams, from HP and VMware, around the world, 24x7, to help keep systems up and running. Such support also allows your IT team to progress, to learn as they go, and to be able to take advantage of more virtualization benefits over time.

So how do you go about attaining such benefits? How do you keep the positive side of virtualization on track? And how do you put in place an insurance policy around service and support?

Manderson: We have several different packages. Our highest level is the mission-critical. In this particular process, you're assigned a team that are across the technology that you have in your environment. But you also get a set of folks who would actually look at not just the reactive support and even some of the proactive, but how actually your entire business is running according to the ITIL standard.

That is coupled with keeping you up and running, and we also can work with you on a type that would be best suited for your environment.

Our critical and independent support includes onsite resources from HP that also include a lot of proactive support. In addition, they're more focused on specific management, but that would be more of an ITSM technology. We can look at that for you.

... We also have the hardware and software support. One of the cool things we have with our hardware support is support automation, our Insight for remote support. That can notify HP that you're having a disk drive failure. Or we will call you and say that we know that disk drive is failing, or something on a buffer server and storage is about to.

You can even take that a step further to look inside at the Windows operating system. We're hardware agnostic on that operating system. We don't care about the vendor—and I believe we are looking at expanding that automation to other operating systems. We have installation and startup services that we can actually go out and set up and configure the hardware and software at a site.

So we definitely integrate across all the multi-vendor services. We run the gamut between all the x86 operating systems, as well as our proprietary operating systems, our servers and storage. Again, we're no stranger to multi-vendor support and keeping the entire environment up and running.

... One of our most creative services would be Proactive Select, a core product series of credits. You can use these credits for maybe planning on migration and upgrade. You can say you need some consulting time. You can use these credits and work with upgrade and migration. You may need some performance or you may need some type of environmental assessment, and these credits can be used for that.

Gardner: When people do employ these services, how do they measure what the payoff is, the value of these services?

Manderson: In 2010, IDC did a study. They went out and looked at the methodology, and this is out on our website. They saw that the customers who have the mission-critical services, reduce their downtime by over 70 percent, and increase their return on investment (ROI) quite high, over 400 percent. The main benefit was in problem management as well as help desk calls, because these were alleviated due to the proactive nature, a lot of looking at the entire environment, and looking at the business processes.

So take a look at the study. It shows IDC's methodology. So looking at things proactively and these support processes can certainly help you reduce that downtime.

... I've been in the multi-vendor space for many, many years—from applications to operating systems—all with HP.

In 2002, when VMware came on the scene, HP actually became alliance partners with them. In 2003, we became a reseller, and thus began our support partnership with them. It would only extend recent in 2005, we also became an OEM. We have thousands of trained and certified Microsoft engineers and Linux professionals, too.

But we have the largest number of VMware-certified professionals. We also have the largest global VMware off-site training center. So HP also does education on these technologies as well. We’ve trained over 20,000 students in the VMware space alone.

And we have had this very strong collaboration with VMware for many years and have support teams around the globe. In addition, we also offer the same level of training that VMware support engineers do. We actually go to their facilities and train right alongside them, too.

We further do this training virtually. The training is then recorded and made available on demand for reference, for folks who are not able to attend a scheduled course. There's definitely a very strong partnership, and as you see from our history with the other vendors as well as VMware, we are no strangers to multi-vendor support.

With all of the VMware products that HP sells, we do provide support across them all. It runs the gamut from the vSphere operating system that will install on the x86 server, through the enterprise management to the vCenter, and virtual desktop infrastructure products like VMware ThinApp. We also support the converter product getting into vCloud Director.

In addition to that, we have the ability to access our peers on the other teams across HP hardware support. This includes servers and storage, and our networking chain. We are quickly able to collaborate with them and pull together a virtual team in to focus on the customer's whole environment, to provide a one-stop shop.

Additionally, you saw that we’ve been in this multi-vendor support business for so many years, with many experts across the other technologies, such as Microsoft and Linux. Of course, the virtual machines (VMs) are running these operating systems. So if the contract is also with them, we can easily pull them in to help us work an end-to-end solution and support it.

Gardner: Let’s think about what happens when there are different levels of support at work. How does that shake-out?

Manderson: We're in a reactive support business. If the customer has a problem, they can either call in at their local region telephone number—whether they are in America, Europe, or Asia Pacific. There are different phone numbers for them to call.

They can also log in via the web, and they'll get to our next developer Level 1 engineer. They're a great organization and have solved over 85 percent of their cases.

If they have issues where they have to escalate, first they will be collaborating with us. We also have an online chat tool, where we are all in a virtual room, the Level 1 engineers, Level 2 engineers, etc. So we’ll be consulting and collaborating with them before they even get to a point of escalation.

If the case does end up needing escalation, chances are they're already collaborating with the first person, and will then end up taking the case. That saves a lot of information transfer, as far as what type of server you have, what’s the firmware, what build level, and what’s the problem there, etc.

Once it reaches Level 2 support, as far as we can continue to collaborate, we can reach our teammates and the hardware teams, too, so we can look at the server and make sure that the environment is what we need it to be. If we can't resolve it, we can also go to Level 3 with VMware at an offline service-partner level.

We have a great relationship with the folks that we work alongside with and would escalate calls to at VMware. We’re obviously not going into Level 1 at VMware because we’ve already done all that work, and we are a service partner. They'll go right up to our peers over at VMware and then we work together, while always owning the solution that we provide back to the customer.

Another part of our infrastructure-as-a-support-organization is that we have a single customer database. I can give an example. A call came into our Level 1 French engineer. When this call came in, for the European folks, it was already the end of their day, and the French engineer could not speak English. It was a critical down, their VMs were offline.

So we worked in a virtual room and they talked to us, and brought the case to us here in America’s time zone. We worked with this case and another tool called HP Virtual Room, where we could actually all look at the customers' desktops in real time. They happened to have EVA storage, and we quickly got an EVA engineer engaged. Of course, we had to find a resource in the Americas because the European folks had already left. So we're all looking in real-time at the customer’s environment and found out that they had locked the storage.

The EVA engineer helped to get back online, while we all watched and the French engineer was translating in French for the customer in order to get it all resolved. We got it back online, and the customers were ready to go home.

We gave instructions on getting log files and we placed a call for follow-up for the daytime hours in Europe the next day. So our counterparts in European support teams picked that up and worked with the customers to resolution, to analyze exactly what happened and prevent it in the future.

We have another process in HP that can actually go with top organizations, our escalation manager process. I was lead source for a particular case where we had a field team assisting a customer deploying a virtual desktop infrastructure (VDI) design. They had a third-party VDI vendor. They had HP hardware, servers, and virtual connects. They had our storage, and we didn’t quite know where the bottleneck was. They were having performance issues by trying to have this VDI at two different locations with the hardware at one site.

The escalation manager was able to get the local office to borrow equipment, and then try to get performance and network traces. They had the Engineering Problem Management Resource (EPMR) lab in Houston trying to duplicate the problems.

Our escalation manager was able to drive the issue to completion across not only the solution standards, but the local office, to owning the actual escalation with all the action items to keep this all on track. We knew where we were going to go. That was about a six-month case, but we did finally find was that the customer was on the technological edge, and the "pipe" to have that performance just did not exist.

Pat Lampert is a technical account manager and does site visits. The technical account managers do go out on site. So we’re aware of the environment. We have the information of your environment documented into the database. When you call, we’re not saying, "Now what kind of server is this? What’s the firmware?" We know this because we already have it documented. We could be calling them to say, "Server 3 is running a little off." We already know which VMware version this is on, because we have that information.

And because we have that, we can also offer proactive advice. We can know that there's a new firmware update, or VMware just came out with a new build, and we have a place where you can go find the latest that's specific to your environment. So this helps to reduce further incidents, because we can be more proactive to help you maintain your business.

Gardner: What are some of the the most frequent questions you receive from the field?

Reddy: I'll address two questions that are frequently showing up. One is, what is the difference between the VMware ESXi image and an HP ESXi image?

Basically, HP takes the same ESXi image that VMware provides to the customers. It then adds HP thin components for hardware management, and it also adds any latest fibre channel and network drivers. Once it's tested and certified, it's available for download both from HP and VMware websites.

And one of the major difference between the two images is that VMware image is disk installable only, whereas HP image can be installed on a disk, USB key, or a SD card.

The other question we're getting nowadays is how to upgrade from VCA4 to VCA5. As with any major upgrades, planning helps. The first thing I would do is understand the difference between ESX 4 and ESX 5, because starting with ESX 5, we have no service console. So we need to understand what the architectural differences are.

Also learn about the new licensing policies. Then, use the System Analyzer that VMware provides to evaluate the current environments, and download, check, and complete the checklist. Once this is done, hopefully the upgrade will go smoothly.

Lampert: Another question that has come up from customers has to do with the added value of getting support directly from HP. It was partly addressed during the presentation we just gave. First of all, VMware does have a fine support organization. I have a couple of friends who work in VMware Support, and they do a good job of supporting their product.

HP, in addition to a similar level of expertise in the product, also offers our expertise in HP hardware, especially if you have systems based on HP Blades. The infrastructure behind that often is tied very closely to the performance and availability of your ESX host. So when you call us, you will have not only someone who is very familiar with the VMware product, but also is familiar with the HP hardware and able to pull in the proper resourced results, problems you might encounter with running vSphere on HP hardware especially.

In addition to that, we have a partnership agreement with VMware, and when you call in for support through HP, you're getting that same level of service when we have to go to VMware to get answers to questions or fixes.

One other question that has come up is about our lab ability to reproduce problems. We have two global labs, one in India and one in the United States. We have several static vSphere cluster configurations with a number of different types of servers already in those configurations, and the ability, when needed, to add specific models, if there is a problem that’s specific to a particular Blade or rack-mounted server model, or a particular card or something like that. So we're quite able to reproduce most problems that come in. We even have some Dell and IBM equipment in our lab also.

Gardner: What other issues are users grappling with?

Reddy: One question I can answer is how to troubleshoot server crashes. When something goes wrong in ESX, we call it the "Purple Screen of Death." Often, these are results of hardware failure, but we still need to rule out the software. So we collect all the logs, and look at it to see if it's a software issue. If it's not a software issue, then we engage the hardware team to see how we can get to the root cause and fix the issue.

Lampert: To dovetail with Sumithra’s comment there, one of the questions I get frequently is what to do if you don’t have a dump. Say the host hangs, and that seems to be almost more common than the Purple Screen of Death. Some customers are't aware that through HP’s Integrated Lights-Out Management, there is the ability to generate a non-maskable interrupt (NMI) just by pressing a button, and by saving a certain environment variable ahead of time in your ESX host.

There is a KB article on this, by the way, if you just search on NMI and core dumping in VMware. But with that setup, you can force a dump while a system is in a hung state, and that will assist us usually in troubleshooting and isolating what caused the hang, whether it be hardware or a problem with the ESX host software.

One question that came up ahead of time is what HP suggests as far as getting a handle on our inventory of VMs? I happened to be involved in field testing some new tools from HP that will be available in January and February regarding vSphere.

One of them is a Holistic Blade and Firmware Analysis that takes into account the VMware environment on our Blade systems which we are working on having ready soon. We have just completed field tests.

And the second is a really nifty Inventory Report HP has just put together. We're just completing field tests on that now. It will be available soon. Basically, we install a small Perl script in the customer environment on any machine that has access to the vCenter host and has a vSphere CLI installed.

This Perl Script crawls through the VMware environment and builds an XML file, which we then feed into a report generator here at HP. This can be used for us to gather information on customers, so we have ahead of time a clear picture of the environment. But also it will be sold as a service to customers.

The report is really quite nice, with all sorts of charts and showing availability of machines and availability of memory and also disk space. It's a very nice report.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 23rd January 2012 Copyright Bloor Research © 2012

"It was the best of times. It was the worst of times." For Dickens, in a Tale of Two Cities, this was how he saw the era of the French Revolution. I think that we live in a time that could be described as being both the worst of times, and the most exciting of times. We face unprecedented economic challenges; challenges that will only be met by business people making more, faster and better decisions about every aspect of their trade. In response to that challenge we are seeing technologies that are just staggeringly exciting with the potential they offer, and one such is Alteryx.

I see two major trends in the BI and analytics space. The two are closely linked to the market challenge. Firstly we have the emergence of what is being described as Agile BI; for me what Agile BI is about is providing the most sophisticated of analytics power to the business user on their desktop, with tools that do not require the skills of a statistician or an IT technician to understand and master, capable of delivering results that are rapid, reliable and readily understood at an affordable price. Then we have the rise of Big Data, which has been written about many times by my colleagues in Bloor, and is not so much about Big as in massive, as being big as a consequence of the range and diversity of data that should, and now can, be included in the decision making process for all but the most trivial of business decisions. It is in this sweet spot that Alteryx sit.

It should be noted that in the big study that McKinsey undertook into Big Data and its implications that they identified that one of the biggest barriers to successful roll out was the shortage of skilled knowledge workers to fill the management posts to use the information wisely. Unfortunately we do not have the time to set up the University courses to fill that void in the near term, but what we do have is products like Alteryx.

Alteryx talk a lot about Data Artisans: by this they mean the people in the business who are tasked with improving marketing responses, improving merchandising, deciding where to locate the next store, reducing churn etc. This is their day job and they do not want an IT project to give them a one-off solution, they need a way to create an ongoing and evolving answer to their needs. Those needs are complex, but they do not want a dumbed down solution, they want one that retains integrity, power and reliability, but which allows them to merge transactional data with operational data, with demographic data, with location intelligence, with social media trends, with market data etc etc. to make an informed decision.

By desktop to cloud they are saying that the whole process of identifying, gathering and loading data - then analysing that data to turn data into information - must be made available on the desktop, with the results then being broadcast via the cloud, be that public or private.

From such descriptions it would be easy to dismiss this as just marketing. But this is not an immature product, it has been going for several years in North America, and is has a very prestigious looking client list with household names like Wal-Mart, and Ford and even Apple. It has a number of vertical apps for groups like Telcos and Retailers that have prebuilt a lot of the functions that people need to do their everyday jobs. These are clearly very successful as 7 out of the 10 North American mobile operators have it deployed because it tackles things like churn, integrating operational data off the network with customer data coming in from the call centre.

As it is sold as a subscription service, these are not just one-off evaluation purchases but renewed deployments that are significant investments by the users. And although they do not a have a direct sales force in EMEA at present (it is planned) they have a channel partner, which is Experian, and Experian themselves use the product as a delivery channel for their data. If you go onto the web site you can see many endorsements by their customers, which indicates how happy they have been with the product.

Because this product essentially has the whole BI stack deployed in the desktop there is a danger that IT people will see it as a threat to the traditional tools, and try to discourage its use. This would be wrong. I do not see this as replacing the existing BI technologies, but it is highly complementary, and it does point to the way that all tools must go in the future. Alteryx clearly do not see their platform as an all-encompassing solution, because they are planning to build connectors to provide Hadoop support via ODBC drivers to Hive, giving them read/write into Hadoop clusters. This illustrates the extremely complex nature of full enterprise BI solutions, as they are now emerging from the era when we saw a single vendor suite on top of a data warehouse as the solution.

I am extremely impressed by what I have seen and will be following the progress of Alteryx with keen interest, and expect to see it putting established analytics players, like IBM and SAS, under pressure to improve their usability and responsiveness in the hands of people outside of the Analytics teams. This is another product to watch in 2012. it should make giant strides.

By: Roger Whitehead, Director, Office Futures Posted: 22nd January 2012 Copyright Office Futures © 2012

Newspapers and blogs have been full of the news that Eastman Kodak and its US subsidiaries have filed for Chapter 11 protection from their creditors. You can get a copy of the filing here. Kodak has created a special Web site about the matter, called Kodak Transforms. (Itself, one assumes.)

The company has long been a husk of its former dominant self and its financial problems have been evident for years. Kodak is pinning its hopes on digital printing, the sale or licensing of its numerous patents and the forbearance of its creditors (which are listed from page 10 of the petition).

Chapter 11 rules give Kodak time to reorganise itself. The bankruptcy court has approved a $950 million line of credit from Citibank, which will help.

What kind of company will result is anyone’s guess but it’s unlikely to end up a major player in many of its present markets. Its last Securities and Exchange Commission 10-K annual filing lists 16 of these, which looks over-ambitious.

(That section of the 10-K says Kodak is in business to help people “unleash the power of pictures and printing to enrich their lives”. I didn’t know the SEC permitted such bullshit in official documents.)

Kodak is still a sizeable company, with a turnover last calendar year of $7.2 billion but this was down from $9.4 two years before. Over the last 5 years, its turnover has shrunk by an average of nearly 9% each year. Debts total nearly $6.8 billion, with assets at the same date (September 2011) of just over $5.0 billion. Shrinkage is inevitable.

The Economist has published this analysis of Kodak’s recent history. Its authors compare it with the rise of Fujifilm of Japan, a once upstart competitor that has managed the transition to digital with greater imagination and clarity of purpose. Some of the comments are worth reading.

Michael Johnston’s blog, The Online Photographer, carries a more personal response to the news. The comments to this, too, repay reading, nearly all being informed and informative.

Among the comments is a reference from me to a paper about Kodak’s future that I wrote in 1998. Because of its length, I have placed it a new area of the Office Jotter site called Tracts. Here it is.

How stumbling the steps were to the all-digital world that many people take for granted today and how slow and costly it then was to produce and disseminate images. One forgets.

By: Helena Schwenk, Principal Analyst, MWD Advisors Posted: 20th January 2012 This work is licensed under a Creative Commons License

Of all the conversations and discussions held at Salesforce’s customer day in Egham this week, one in particular struck a chord with me. Cloudapps, an ISV and partner of Salesforce, spoke briefly about the challenges of capturing, storing and analysing Big Data, in particular telecom mast sensor data, on the Force.com platform.  Although there wasn’t time to drill into detail about Force.com’s Big Data support – it got me thinking generally about the wider options for exploiting vast amounts of digital data within the cloud.

As a lot of the Big Data that people are interested in analysing – such as sensor data, social media data or weblog file data – already lives in the cloud, it makes sense to also use it as a platform for hosting and analysing this data too. Rather than the often more inefficient method of pushing it to an on-premise enterprise data warehouse for example.  At the same time one set of technologies in particular – Hadoop – is also becoming synonymous as a lower cost approach for storing and processing these large-scale datasets in the cloud.

Given the proliferation of digital data and the desire to harness it for better business effect, it’s not surprising that Big Data and the Cloud are on a natural collision course. With its elastic processing capacity, lower cost and lower risk approach, the cloud provides a powerful platform for storing, processing and crunching this data – whether using Hadoop or not. Similarly the emergence of cloud based offerings that mix analytics and Big Data in the public cloud are also helping to circumnavigate some of the skills shortage issues relating to advanced analytics techniques and Big Data technologies.

However that’s not to say that the cloud should be seen as the answer to all your Big Data needs. On-premise data warehouses that employ technologies such as MPP analytic databases, in-memory computing, columnar databases or packaged appliances provide equally valid alternatives. In fact in many cases these Big Data approaches should be seen as complementary to each other, as each brings different strengths to the table.  The challenge however for organisations that haven’t put their eggs in any one particular Big Data basket, is how to mesh these approaches together and equally how to do this across the on-premise and cloud divide.  The benefits of  integrating data to support a more consolidated, complete and accurate view of your business are well known after all.

As we outlined in our recent report on Analytics in the Cloud, a hybrid cloud is one plausible approach to using both public clouds and private or on-premise IT to deliver a more integrated Big Data analytic system. This can provide a more pragmatic and blended approach for balancing the strengths and pitfalls of both cloud and on-premise implementations but it also comes with its own set of challenges. Apart from managing the environment there are also factors relating to the immaturity of certain Big Data technologies, lack of best practice and interoperability across platforms.  For example, of those looking at Big Data Hadoop projects a significant proportion are still in experimentation (rather than production) mode, testing out the concepts, design and technology -although we do expect this to change over the next 12-18 months as the market evolves.

Given the great many opportunities for leveraging Big Data in the cloud, it’s surprising to see that aside from its social media monitoring platform Radian6, Salesforce doesn’t have a stronger message or story about its Big Data hosting capabilities. It appears from the conversations at the customer event the company still has something to prove when it comes to supporting and helping ISVs and developers work with Big Data especially on the Force.com platform. But as a company used to pioneering cloud based offerings we don’t expect this to be the situation for very long.

By: Angela Ashenden, Principal Analyst, MWD Advisors Posted: 20th January 2012 This work is licensed under a Creative Commons License

This week, I attended IBM’s Lotusphere 2012, IBM’s annual customer and partner event which showcases the latest products and strategy in the area of collaboration. Like last year, “social business” was squarely at the centre of things, although there was a marked difference between the way the company positioned its social message this time when compared to last year. While Lotusphere 2011′s “Get Social. Do Business.” strapline and general sentiment was very much a call to action, this year I felt an interesting undercurrent of confidence and a sense of “fait accompli” in IBM’s message, reinforced by its new 2012 strapline “Business. Made Social.”

In parallel with the Lotusphere event, IBM hosted IBM Connect 2012, which comprised of a wealth of social business case studies, and was positioned as the event for the senior execs who wanted both ideas for how to take advantage of social business, and tips on best approaches. My personal favourite was Asian Paints, which has leveraged IBM Connections to support both internal collaboration and external customer engagement. Frankly, I was astonished by the sheer number of customers who stood up to share their experiences, although I do think that IBM missed an opportunity by not building in some sort of workshop element to the program to help attendees talk through their ideas and plans in a more structured way than simply over coffee at the event.

As usual, IBM used the Lotusphere platform to announce a number of new product capabilities and offerings, the most notable being these:

• IBM SmartCloud for Social Business – in line with other branding changes, IBM announced that their cloud-based collaboration offering, LotusLive, is being rebranded as IBM SmartCloud for Social Business, leaving  Notes/Domino as the only product retaining the Lotus branding. I think the rebranding is a sensible decision; despite efforts to expand the Lotus brand to include IBM’s other collaboration solutions, it is still limited to Notes/Domino in the minds of the majority. That said, the new name doesn’t exactly roll off the tongue, and is likely to be abbreviated in some way which will inherently remove the value of it as a name anyway.
• Connections Mail – a relatively small feature which has had a significant reaction already, Connections Mail allows Connections users to access their Notes or Exchange mail and calendar directly from within Connections. At present, the functionality is deliberately limited, allowing viewing and searching of the inbox, as well as the ability to compose messages. There are no mail handling features (apart from the star or flag feature), but IBM figures that users who leverage the more advanced capabilities of Notes or Exchange will continue to use their mail client for that purpose anyway. I think there is room for a little more functionality than is currently provided though, such as assigning folders to a message and providing access to messages in folders, but to be fair IBM is waiting to see how people use this feature rather than trying to second guess its use. The purpose of Connections Mail is more about enabling Connections to be a hub for accessing all your information, be that mail, social or documents. Kinda in the same way that the Notes client is getting activity streams in its next version (oh yes, that was another announcement – it will be called Notes Social Edition, and is expected later this year).
• IBM Docs - also this week we saw the launch of the public beta of IBM’s cloud-based collaborative editing tool, IBM Docs. The tool, which was previously called LotusLive Symphony and codenamed Project Concorde, is now available through IBM’s hosted trial platform, Lotus Greenhouse, and is expected to finally become generally available later in 2012.

Aside from the key product announcements, what was particularly noticeable this year was the omnipresence of analytics. First introduced at last year’s Lotusphere through the integration of Cognos with Connections, this year pretty much every session referenced analytics in some way or another, emphasizing its important role in deriving ROI from these types of social investments. (While we’re on the topic of analytics, don’t forget to visit our online event focused on social analytics which launched this week – it’s free with no registration required, and it’s on-demand so you can dip in and out as you want.)

All in all, it’s fair to say that Lotusphere has come a long way in just a couple of years: what was once an intensely techy event, focused on product demos and unquestionably the territory of developers, has in the space of a couple of years transformed smoothly into a business-focused event, aimed at engaging with customers and partners at a strategic, enterprise-wide level, to help and support them in bringing about business change enabled by the social revolution. This is a very different approach to IBM’s biggest competitors in this space – Microsoft and Google – which both continue to position at a more technical level. There is still work to be done to tie up IBM’s top level social strategy with its product portfolio, but the company is investing significant resources in this strategy, and is benefiting from its incredible traction with IBM Connections.  Big ambitions, but IBM’s looking in good shape to succeed here.

Advisory clients can read our recently published Email strategy profile of IBM here.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 20th January 2012 Copyright Interarbor Solutions © 2012

The Open Group has announced this week the availability of two new industry standards to integrate fundamental elements of service oriented architecture (SOA) and cloud computing into a solution for enterprise architecture (EA). The new standards are: SOA Reference Architecture (SOA RA) and the Service-Oriented Cloud Computing Infrastructure Framework (SOCCI).

The Open Group has released updates to The Open Group Service Integration Maturity Model (OSIMM), which has now been ratified as an ISO and IEC (ISO/IEC 166880) International Standard. OSIMM gives organizations a common model for developing a roadmap for achieving the right level of service adoption to meet business objectives. [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

SOA RA is a blueprint for creating and evaluating SOA solutions, while SOCCI is the first Open Group cloud standard that outlines the concepts and architectural building blocks necessary for infrastructures to support SOA and cloud initiatives.

"In today's global competitive marketplace it is imperative that business and IT drivers are aligned," said Chris Harding, Director for Interoperability, The Open Group. "Each of the three standards is vendor-neutral and helps an organization of any size to design and implement the proper SOA and cloud solutions for its business objectives."

SOA RA is an industry standard reference architecture for the development of SOA solutions. Utilizing the SOA RA Standard, enterprise architects will have a common language and approach for creating SOA solutions that meet different organizational needs and bridge the gap between business and IT.

SOCCI is the industry's first cloud standard for enterprises that wish to provide infrastructure as a service in the cloud and SOA. Developed by The Open Group SOA and Cloud Work Groups, SOCCI is the realization of an enabling framework of service-oriented components for infrastructure to be provided as a service in SOA solutions and the cloud.

The standard details a set of common SOCCI elements and management building blocks for organizations to consider and identifies the synergies that can be realized through cohesive application of SOA and cloud-based principles. Using SOCCI, organizations can incorporate cloud-based resources and services into their infrastructure for increased agility and scale, and lower maintenance costs.

Proven best practices
OSIMM leverages proven best practices to allow consultants and IT practitioners to assess an organization's readiness and maturity level for adopting services in SOA solutions. By aligning business goals and assessing associated SOA services IT practitioners can create a detailed roadmap for integrating services for SOA and cloud computing solutions into enterprises. With the recent ratification of OSIMM 2.0 by ISO and IEC, organizations worldwide have an extensible framework for understanding the value of implementing a service model, as well as a comprehensive guide for achieving their desired level of service maturity.

The SOA RA technical standard, SOCCI framework, and OSIMM 2.0 International standard are available for download from The Open Group Bookstore. These new standards can also be viewed online at: SOA Reference Architecture, Service-oriented Cloud Computing Infrastructure, Open Group Service Integration Maturity Model.

In addition to the standards news, The Open Group on Jan. 30 will begin its San Francisco conference to focus on the role played by IT and EA within enterprise transformation. Among the topics to be explored:

• The differences between EA and enterprise transformation, and how they relate to one another
• The use of EA to facilitate enterprise transformation
• How EA can be used to create a foundation for enterprise transformation that the board and business-line managers can understand and use to their advantage
• How EA facilitates transformation within IT, and how does such transformation support the transformation of the enterprise as whole
• How EA can help the enterprise successfully adapt to "disruptive technologies" like cloud computing and ubiquitous mobile access.

Among the speakers at the conference will be Andy Mulholland, the Global Chief Technology Officer and Corporate Vice President at Capgemini. In 2009, Andy was voted one of the top 25 most influential CTOs in the world by InfoWorld. And in 2010, his CTO Blog was voted best blog for business managers and CIOs for the third year running by Computer Weekly.

Andy recently participated in a BriefingsDirect podcast, in which he spoke about an upcoming Capgemini whitepaper, which draws distinctions between what cloud means to IT, and what it means to business -- while examining the complex dual relationship between the two.

Also, Jeanne Ross, Director and Principal Research Scientist at the MIT Center for Information Systems Research. Jeanne studies how firms develop competitive advantage through the implementation and reuse of digitized platforms.

Jeanne recently spoke with me about how adoption of EA leads to greater efficiencies and better business agility and explained how enterprise architects have helped lead the way to successful business transformations.

Also speaking is Joseph Menn, Cyber Security Correspondent for the Financial Times and author of Fatal System Error: The Hunt for the New Crime Lords Who are Bringing Down the Internet.

Joe has covered security since 1999 for both the Financial Times and then before that, for the Los Angeles Times. Fatal System Error is his third book, he also wrote All the Rave: The Rise and Fall of Shawn Fanning's Napster.

As a lead-in to his Open Group presentation, entitled "What You're Up Against: Mobsters, Nation-States, and Blurry Lines," Joe recently joined BriefingsDirect to explore the current cyber-crime landscape, the underground cyber-gang movement, and the motive behind governments collaborating with organized crime in cyber space.

Registration remains open for The Open Group Conference in San Francisco, beginning Jan. 30.

By: David Norfolk, Practice Leader - Development, Bloor Research Posted: 19th January 2012 Copyright Bloor Research © 2012

Sorry if this isn't hot news, but sometimes the business really doesn't like the IT group very much. The IT group is seen as "the people who say NO" when the business comes up with a neat new idea. And that is one reason why things like "end user computing", "business process automation" and so on are so popular. It's not the only reason, of course, but bypassing the IT group is often rather attractive.

More (business) agility is certainly worthwhile and past IT practices have often compromised it. To say nothing of the IT groups' occasional(!) arrogance… However, there is often a valid reason for IT "good practice" and there's a danger that useful things like testing, quality management and configuration management are neglected when the business adopts, say, business process automation. And, in the beginning, this may not matter much - as business process automation frequently starts as collaborative prototyping of simple automated processes with extremely supportive software. Automated process develops in an interactive "people centric" way and formal development disciplines such as testing, quality management and configuration management seem irrelevant. However, this situation doesn't last.

Things seldom remain simple as automated process changes and evolves, scales up to larger communities and is rolled out to a wider community. Business rules, to take just one example, can increase exponentially in complexity as new, possibly contradictory, rules are added. If you run your business on business process automation you will need a degree of "good governance" in this area. For example:

• Testing provides assurance that your process automation works reliably as intended and produces the expected business outcomes;
• Regression testing provides assurance that, say, adding a new business rule doesn't cause existing behaviours to break;
• Quality management provides assurance that the automated processes are resilient and reliable, reasonably effective, reasonably performant and reasonably efficient - that they produce desirable business outcomes without waste.
• Configuration Management is the basis for managing change and impact analysis; it provides assurance that an automated process that is known to work can be returned to after a failed change, after a disaster and if regulators question the operation of a past process

What this means is that business process automation tools must increasingly support formal development "good practice" activities - to whatever extent is appropriate (depending on the importance of the business process being automated and the degree of regulation that applies). Nevertheless, care must be taken to introduce "just enough" of this sort of governance, not governance for its own sake, if the increased "agility" that is the real business driver driver behind most business process automation efforts isn't to be compromised.

By: Philip Howard, Research Director - Data Management, Bloor Research Posted: 19th January 2012 Copyright Bloor Research © 2012

I have written about RainStor previously. Hitherto it has had a single product, what is now known as RainStor Data Retention but it has now announced a second: RainStor Data Analytics for Hadoop.

First, a re-iteration of what the basic product does. It provides a highly compressed file system. A couple of notable features are worth mentioning. The first is that if you are using RainStor for relational data (typically, for application retirement or archival) then RainStor ingests the schema as well as the data. It then supports schema evolution, so that you can make queries at a point in time (that is, you can look at the data exactly as it would have appeared at a particular point in time). Secondly, it includes a query engine that supports (translates) incoming SQL so that you can run conventional business intelligence environments against RainStor. Note that it is fully ANSI compliant.

Apart from archival (for which RainStor partners with both Teradata and Informatica - RainStor is embedded in the latter's archival offering) the major market for RainStor has been retention of call detail records (CDRs), IP detail records (IPDRs), log data and so forth. The product's compression rates are impressive in these areas: for example, getting 35 times compression for CDRs, 20 times for IPDRs and 15 times for log data. For certain types of data, compression rates can be 40 times or more. One of its customers is ingesting 17 billion IPDRs per day, which is the highest I have heard from any vendor.

However, we are here to discuss RainStor and Hadoop. What the company has done is to allow the storage of RainStor partitions within HDFS (Hadoop distributed file system) and it has added native MapReduce support to its existing SQL capabilities. The company has also partnered with Cloudera, HortonWorks and MapR so, in the last case at least, you don't have to worry about single points of failure in your Hadoop cluster.

There are four interesting things about this implementation. The first is the compression that you can get. Typical Hadoop implementations use pretty inefficient compression algorithms (something equivalent to zip files) that get you around a 3 times compression ratio. That means that using RainStor you only need, worst case, around 20% of the disk capacity that you would require otherwise and it could be very much less than this. That's a significant saving.

The second interesting feature of RainStor Data Analytics is that RainStor, as you might expect from its use for the retention of CDRs and so forth, understands timestamps, which Hadoop does not. In this sense the use of RainStor makes it a competitor to Cassandra and other column-family databases.

Thirdly, if you are using RainStor in HDFS then you can choose what data you want to flatten. So, because RainStor understands SQL, you can store relevant data in tabular format if you want to and you can flatten other data where that is not relevant. And, of course, the SQL supported by RainStor means that you can do functions like joins, which would not otherwise be possible. It is also worth noting that RainStor files are treated as first-class objects within the context of Hadoop and MapReduce, so there is no need to change existing scripts (in Pig, say) only to change a single parameter that points the query to RainStor partitions.

Finally, it is worth commenting on query performance. RainStor uses what are known as Bloom filters (named for the inventor Burton H. Bloom) which are space-efficient probabilistic data structures that are used to test whether an element is a member of a set. False positives are possible, but false negatives are not. Put simply, these filters tell the system where the data is not (rather like a Netezza ZoneMap) so that the software looks for the data it needs only within relevant data blocks. The advantage of using these Bloom filters (which, technically, are based on bit vectors) is that they greatly increase performance and, at the same time, they require much less management and overhead than indexes. Note that filters, like the data, are compressed. The upshot of all of this is that you get much better performance that you would when using HDFS on its own. Of course, you will get improved performance from simply having better compression but the use of Bloom filters takes this a significant step further.

This is an impressive list of features - (much) less disk, more capability and better performance - what's not to like? Perhaps the most obvious thing is that you might want to be able to combine MapReduce and SQL within a single query but I understand that this is on the company's roadmap. It might also make sense for RainStor to implement support for IBM's GPFS as well as HDFS. The only other point worth making is that RainStor offers an "append only" data store. That is, you cannot update records. This is necessary for compliance reasons in the company's current markets. For many Hadoop implementations this will not be an issue (event data is not typically updated) although there may be additional housekeeping: removal of previous data and writing of new data, rather than just updating. It is also possible to conceive of environments where you might genuinely want to update data, in which case the data will need to be stored outside of RainStor. Leaving these considerations aside, RainStor looks like a very competitive option for supporting Hadoop.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 18th January 2012 Copyright Interarbor Solutions © 2012

This BriefingsDirect thought leadership interview comes in conjunction with The Open Group Conference this month in San Francisco.

The conference will focus on how IT and enterprise architecture support enterprise transformation. Speakers in conference events will also explore the latest in service oriented architecture (SOA), cloud computing, and security.

We’re now joined by one of the main speakers, Andy Mulholland, the Global Chief Technology Officer and Corporate Vice President at Capgemini. In 2009, Andy was voted one of the top 25 most influential CTOs in the world by InfoWorld. And in 2010, his CTO Blog was voted best blog for business managers and CIOs for the third year running by Computer Weekly.

Capgemini is about to publish a white paper on cloud computing. It draws distinctions between what cloud means to IT, and what it means to business—while examining the complex dual relationship between the two.

As a lead-in to his Open Group conference presentation on the transformed enterprise, Andy draws on the paper and further drills down on one of the decade’s hottest technology and business trends, cloud computing, and how it impacts business and IT. The interview is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: Why do business people think they have a revolution on their hands, while IT people look cloud computing as an evolution of infrastructure efficiency?

Mulholland: We define the role of IT and give it the responsibility and the accountability in the business in a way that is quite strongly related to internal practice. It’s all about how we manage the company’s transactions, how we reduce the cost, how we automate business process, and generally try to make our company a more efficient internal operator.

When you look at cloud computing through that set of lenses, you’re going to see ... the technologies from cloud computing, principally virtualization, [as] ways to improve how you deliver the current server-centric, application-centric environment.

However, business people ... reflect on it in terms of the change in society and the business world, which we all ought to recognize because that is our world, around the way we choose what we buy, how we choose to do business with people, how we search more, and how we’ve even changed that attitude.

There's a whole list of things that we simply just don’t do anymore because we’ve changed the way we choose to buy a book, the way we choose and listen to music and lots of other things.

So we see this as a revolution in the market or, more particularly, a revolution in how cloud can serve in the market, because everybody uses some form of technology.

So then the question is not the role of the IT department and the enterprise—it’s the role technology should be playing in their extended enterprise in doing business.

Gardner: What do we need to start doing differently?

Mulholland: Let’s go to a conversation this morning with a client. It’s always interesting to touch reality. This particular client is looking at the front end of a complex ecosystem around travel, and was asked this standard question by our account director: Do you have a business case for the work we’re discussing?

The reply from the CEO is very interesting. He fixed him with a very cold glare and he said, "If you were able to have 20 percent more billable hours without increasing your cost structure, would you be bothered to even think about the business case?"

The answer in that particular case was they were talking about 10,000 more travel instances or more a year—with no increase in their cost structure. In other words, their whole idea was there was nothing to do with cost in it. Their argument was in revenue increase, market share increase, and they thought that they would make better margins, because it would actually decrease their cost base or spread it more widely.

That's the whole purpose of this revolution and that's the purpose the business schools are always pushing, when they talk about innovative business models. It means innovate your business model to look at the market again from the perspective of getting into new markets, getting increased revenue, and maybe designing things that make more money.

We're always hooked on this idea that we’ve used technology very successfully internally, but now we should be asking the question about how we’re using technology externally when the population as a whole uses that as their primary method of deciding what they’re going to buy, how they’re going to buy it, when they’re going to buy it, and lots of other questions.

... A popular book recently has been The Power of Pull, and the idea is that we’re really seeing a decentralization of the front office in order to respond to and follow the market and the opportunities and the events in very different ways.

The Power of Pull says that I do what my market is asking me and I design business process or capabilities to be rapidly orchestrated through the front office around where things want to go, and I have linkage points, application programming interface (API) points, where I take anything significant and transfer it back.

But the real challenge is—and it was put to me today in the client discussion—that their business was designed around 1970 computer systems, augmented slowly around that, and they still felt that. Today, their market and their expectations of the industry that they're in were that they would be designed around the way people were using their products and services and the events and that they had to make that change.

To do that, they're transformed in the organization, and that's where we start to spot the difference. We start to spot the idea that your own staff, your customers, and other suppliers are all working externally in information, process, and services accessible to all on an Internet market or architecture.

So when we talk about business architecture, it’s as relevant today as it ever was in terms of interpreting a business.

But when we start talking about architecture, The Open Group Architectural Framework (TOGAF) is a set of methodologies on the IT side—the closed-coupled state for a designed set of principles to client-server type systems. In this new model, when we talk about clouds, mobility, and people traveling around and connecting by wireless, etc., we have a stateless loosely coupled environment.

The whole purpose of The Open Group is, in fact, to help devise new ways for being able to architect methods to deliver that. That's what stands behind the phrase, "a transformed enterprise."

... If we go back to the basic mission of The Open Group, which is boundarylessness of this information flow, the boundary has previously been defined by a computer system updating another computer system in another company around traditional IT type procedural business flow.

Now, we’re talking about the idea that the information flow is around an ecosystem in an unstructured way. Not a structured file-to-file type transfer, not a structured architecture of who does what, when, and how, but the whole change model in this is unstructured.

Gardner: It's important to point out here, Andy, that the stakes are relatively high. Who in the organization can be the change agent that can make that leap between the duality view of cloud that IT has, and these business opportunists?

Mulholland: The CEOs are quite noticeably reading the right articles, hearing the right information from business schools, etc., and they're getting this picture that they're going to have new business models and new capabilities.

So the drive end is not hard. The problem that is usually encountered is that the IT department’s definition and role interferes with them being able to play the role they want.

What we're actually looking for is the idea that IT, as we define it today, is some place else. You have to accept that it exists, it will exist, and it’s hugely important. So please don’t take those principles and try to apply them outside.

The real question here is when you find those people who are doing the work outside—and I've yet to find any company where it hasn’t been the case—and the question should be how can we actually encourage and manage that innovation sensibly and successfully?

What I mean by that is that if everybody goes off and does their own thing, once again, we'll end up with a broken company. Why? Because their whole purpose as an enterprise is to leverage success rapidly. If someone is very successful over there, you really need to know, and you need to leverage that again as rapidly as you can to run the rest of the organization. If it doesn’t work, you need to stop it quickly.

In models of the capabilities of that, the question is where is the government structure? So we hear titles like Chief Innovation Officer, again, slightly surprising how it may come up. But we see the model coming both ways. There are reforming CIOs for sure, who have recognized this and are changing their role and position accordingly, sometimes formally, sometimes informally.

The other way around, there are people coming from other parts of the business, taking the title and driving them. I’ve seen Chief Strategy Officers taking the role. I’ve seen the head of sales and marketing taking the role.

Certainly, recognizing the technology possibilities should be coming from the direction of the technology capabilities within the current IT department. The capability of what that means might be coming differently. So it’s a very interesting balance at the moment, and we don’t know quite the right answer.

What I do know is that it’s happening, and the quick-witted CIOs are understanding that it’s a huge opportunity for them to fix their role and embrace a new area, and a new sense of value that they can bring to their organization.

Gardner: Returning to the upcoming Capgemini white paper, it adds a sense of urgency at the end on how to get started. It suggests that you appoint a leader, but a leader first for the inside-out element of cloud and transformation and then a second leader, a separate leader perhaps, for that outside-in or reflecting the business transformation and the opportunity for what’s going on in the external business and markets. It also suggests a strategic road map that involves both business and technology, and then it suggests getting a pilot going.

How does this transition become something that you can manage?

Mulholland: The question is do you know who is responsible. If you don’t, you'd better figure out how you're going to make someone responsible, because in any situation, someone has to be deciding what we're going to do and how we're going to do it.

Having defined that, there are very different business drivers, as well as different technology drivers, between the two. Clearly, whoever takes those roles will reflect a very different way that they will have to run that element. So a duality is recognized in that comment.

On the other hand, no business can survive by going off in half-a-dozen directions at once. You won't have the money. You won't have the brand. You won't have anything you’d like. It's simply not feasible.

So, the object of the strategic roadmap is to reaffirm the idea of what kind of business we're trying to be and do. That’s the glimpse of what we want to achieve.

There has to be a strategy. Otherwise, you’ll end up with way too much decentralization and people making up their own version of the strategy, which they can fairly easily do and fairly easily mount from someone else’s cloud to go and do it today.

So the purpose of the duality is to make sure that the two roles, the two different groups of technology, the two different capabilities they reflect to the organization, are properly addressed, properly managed, and properly have a key authority figure in charge of them.

The business strategy is to make sure that the business knows how the enablement model that these two offer them is capable of being directed to where the shareholders will make money out of the business, because that is ultimately that success factor they're looking for to drive them forward.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 17th January 2012 Copyright Bloor Research © 2012

I was first introduced to Splunk last year and, once I got used to its name, quickly came to realise that this was something that had amazing potential. Splunk is a leader in its market, which is the provision of real-time operational intelligence. What it does is collect, index and make available for exploitation all of the machine data that is being generated as part of the Big Data explosion that is going on all around us. What that means is that, as machines and processes increasingly have monitors attached at every significant node, the mass of data that tells us what is happening, or what is not happening and, having collected it, Splunk indexes it so that it is turned from a mass of data into information, which can then be exploited.

Splunk has just announced a new version, Enterprise 4.3, that adds to its already impressive capabilities. In keeping with all tools designed to operate in the real world of Big Data, Splunk already provides scaling, speed and cost effective operations to enable you to tackle all of that machine data in a way that is affordable and makes it compelling to tackle the problem of ensuring that you do deliver excellent service to the letter of an SLA, and it now does that even better. In 4.3 you have that other great essential of the current crop of BI tools, mobility. It has a non-Flash user interface, which delivers the power of Splunk to you wherever you may be, and, of course, being non-Flash means you can use your tablet of choice, even if that is an iPad. It is even more powerful, being able to handle up to 10x more concurrent users, and it's faster, being able to run search up to 10x faster on the same platforms as the previous edition and, finally, it is more user friendly, with its dashboards now being editable by business users and executives.

Splunk is probably one of the best-kept secrets going around at present. It is used wherever there are masses of data, and service is a business-critical issue; so the technical community in ISPs, ecommerce vendors, energy companies, financial service companies etc. rely on it. But it opens the door to so much that is required to deliver really effective Big Data solutions. Big Data is about getting a true 360-degree view of the world, integrating that data, and acting on it, to create new business opportunities. The explosion of data that represents Big Data is not caused by more of the same, there are not more transactions in the world now than there were a few years ago; in fact, with the current economic climate there are probably less, but there are now more data sources. There are, obviously, the standard transactions that form the basis of most traditional data warehouses, and to that we can now add analysis from the likes of social media, such as Facebook and Twitter, to understand more about how people interact, and view things as a social group; and we have machine data, which, in many ways, underpins everything else. There is not much point in understanding what we buy and sell, or what we think of things, if we do not understand the process that is the basis of that transaction or social commentary. This is where Splunk sits, giving that insight.

With the new features of Splunk, its mobility, its adaptability without needing to edit XML, its ability to access real-time and historical data in a single step, its speed and manageability, it is no longer a tool that should just be thought of as a back-office essential. This is information that is vital to ensuring that profitable customers are provided with excellence of service and retained; for providing the platform to extend the reach of your product range to other potentially profitable and loyal groups. So Splunk should be a true enterprise tool, seen as part of the corporate arsenal, providing competitive advantage, used as much by Marketing and Customer Service as by engineers.

The advantage that Splunk offers is that you can start small, build confidence and grow. It is not about giant leaps of faith. Because of its cost model you scale up as the business benefits are realised. We are not talking about the complexity of a full blown real time Complex Event Processing solution, which is a giant leap of faith; this is a logical set of building blocks that will allow you to get there incrementally and, with Splunk, you will be able to go all the way up to being able to leverage the data on Hadoop clusters in the near future, so this is not a path with limits.

The big question is, can those outside of the traditional base of Splunk realise this potential and be in the vanguard? Or will Marketers and others in the enterprise wait to be told by IT that this is technology you should look at, by which time a competitor with more imagination may have stolen a march on you. These are exciting times, and I believe that Splunk is part of the answer to our finding ways of reinvigorating our economic activity through innovation and effective customer focus.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 16th January 2012 Copyright Interarbor Solutions © 2012

The latest BriefingsDirect case study podcast discussion centers on how Nottingham Trent University gained strategic operational efficiency and improved IT management.

A combination of professional services and portfolio management technologies allowed the 25,000-student university—one of the U.K.’s largest—to improve end-user satisfaction while freeing up IT resources to pursue additional technology innovation.

To learn more, BriefingsDirect brought together Ian Griffiths, Director of Strategic Partnerships at Nottingham Trent University, and Michael Garrett, Vice President of Professional Services for HP EMEA. The discussion was moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: HP is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: What was the one glaring thing that needed to be changed when you began to think about improving how you did IT?

Griffiths: We were very, very good at moving forward and doing lots and lots of things, but delivering products at the end of that period was more difficult. We seemed to be running around in circles, and didn’t quite meet customers’ expectations. So we were doing a lot, working really hard, but not really delivering the last mile.

Gardner: Why did something like professional services become a priority for you?

Griffiths: We found that our processes were not really defined well enough. We really weren’t getting sign-off from the business, and the expectations were never really met. So it was clear that we were not doing something well, and we didn’t quite know what that was. And our teams within the department weren’t gelling that well together either.

Gardner: So perhaps having some outside additional authority and experience seemed to work for you?

Griffiths: Yes. That worked really well. We had had another attempt about 18 months before, and had some consultants in, but it didn’t really gel. We were aware that we had a partnership with HP, and HP Professional Services seemed a sensible way to go. But we were still doubtful as a management team within the university's Information Services (IS) Department whether it was really going to work. And we are very pleased with the outcome.

Gardner: Let’s learn about Nottingham Trent University. You’re in Nottinghamshire and you have 25,000 students. Tell us a bit more.

Griffiths: We’ve been a higher education establishment for about 160 years. We’re one of the biggest providers of "sandwich education," which means that students have two years at the university, a year in industry, and then a year at the university.

We're seen as a popular university that has good reputation for placing students at the end of their courses, and we got top of The Green Agenda twice in the last three years within the U.K. We have about 150 people working in the IS Department on three campuses and nine academic schools.

I have responsibility for the strategic partnership we have with companies and with firms. I have responsibility for the regional network within the East Midlands of the U.K., which is connecting all the universities in that region and all the further education colleges. And I also manage relationships with key suppliers, such as HP.

Gardner: Ian had a relationship with HP, but looked for something bigger.

Garrett: It’s often imagined that these organizations look to pure-play consulting organizations for that advisory activity. In Nottingham Trent’s situation they were willing to listen to a different type of vendor or organization in that space as to what they could offer in their approach. What’s different for HP Professional Services is that it forms part of HP’s Software organization. Our consulting capability is very focused on IT transformation, operations, organizations, and applications.

But it’s about bringing that into real practical use quickly with the support of technology. That's the real differentiator we wanted to bring to customers like Nottingham Trent, and hopefully that’s true with what we've seen in the practical implementation and the work we've done with them.

Gardner: Ian, how has this worked out for you?

Griffiths: We had some initial workshops where all the senior management team of the IS Department worked with HP and looked at what we wanted to achieve, and looked at what the journey might look like to get there. I have to congratulate HP. They were able to get that team to gel together within IS in a way that we hadn’t before.

We spent a lot of time working together and working through the structure, the plan of the department, and what we called the "tube map" of the department. Everything, in a sense, was allowed. HP was very good at giving us a straw man to look at. In other words, giving those examples of what other companies have done, but forcing us to discuss them in detail and change them into what was right for Nottingham Trent.

They weren’t trying to sell the straw man, but were using the straw man as an example to move us forward, and it worked extremely well. Although there were some heated discussions amongst IS staff, HP was very good at facilitating those discussions.

We had to go back to the rest of the department to try not to force something new on people that, as far as they could see, had no relevance to the situations they were in. We had to find a way, as well, of getting the business to buy into our new methodology, getting the business to feel some ownership, and getting the business to make some decisions during the planning of projects and the ending of projects.

Garrett: It’s that level of being able to bring the input, the straw man, and then guide organizations around that model. To customize from scratch takes a great deal of time and can take too much energy and cost. What we’re trying to do is bring our method and models at the start point and then work in a very collaborative, but directed, way to get clients to a point, although, a configured approach rather than a completely dispersed approach.

Therefore, we get to things more quickly, but absolutely meet the requirement of the individual organization. We’ve got to appreciate they are different across different industries and different areas, and strong cultural alignment is critically important. We certainly saw that in this program.

Griffiths: The important thing again was that we were producing our outline, and that outline allowed us to go away and do a lot more detail later. In other words, we got the big picture agreed upon and then all the details were passed back to teams within the department to build up details in the areas where they had real knowledge of what happened.

Gardner: Was there a point at some time where you needed to get an understanding of where and what’s going on in order to know how to measure any improvement?

Griffiths: An important step early on in this was beginning to define how many projects we were running as a department and to categorize work into projects that were developmental and projects that were more of the business-as-usual type.

We found in the end that we had over 100 projects running simultaneously. Some of those projects had been running for more than a year, some had no real defined endpoint, and the customer requirements weren’t documented in a thorough way.

It’s important to measure how many projects you’ve actually got, and actually have a start date and a planned finish date for them. One thing we learned was that 100 was too many for us to run, and we were able to cut down by finishing some off, to less than 50 that we have now.

Gardner: And what has that done now? What are some of the metrics of success by getting more of a handle over your portfolio and managing it?

Griffiths: Probably the biggest one is that projects are getting completed and the project didn’t become the be-all and end-all, and continue running forever. We were actually delivering something that the customer was expecting. And the customer, the student or the staff department, had a glow that they have had something delivered to them.

The student satisfaction with IS has gone up over the last two to three years. They're very happy with our technology and technology moving forward. But again, we found that people were happier with the delivery of an item, rather than as IS was before, striving for technical perfection.

Before, we had the figures of 80 percent [of IT projects] being used in the areas of business-as-usual, and only 20 percent in project and development work. We quickly moved to a 70/30 split and our target is to move towards 50 percent. We're not quite there yet, but we’re a lot more like 60 percent business as usual, 40 percent new development work.

It’s a virtuous cycle, and the other thing that is gained from that is appreciation amongst other departments within the university and with senior management with what IS was delivering, and getting them to prioritize what we did.

There was a problem, if we look back two or three years. IS very much decided what the priorities were. Now, the business is deciding and even deciding in the case that a project that was a favorite of a senior member of staff, he or she may decide that it no longer is a top priority, compared with other projects that needed to be delivered.

Gardner: Is there something about the products themselves, the portfolio management approach, that now allows the business side of the organization, the leadership in this case, to have more visibility or input? How were you able to get it?

Griffiths: More visibility and more input. The example we always give is of a jam jar. You can keep putting rocks into a jam jar, but in the end, it becomes full. Unless you allow something to come out of that, nothing happens. So you’ve got to be able to allow things to finish and give you some capacity.

The other thing that I talked about was looking at the business benefits of everything we were doing and deciding the nice-to-haves probably weren't going to get prioritized at this stage.

We're using [the tube map] outside the department to make people realize that we are working to an operational framework. As such, we have them stuck up round the department. And in the rooms where we have project meetings, they exist as well. As to vocabulary, we have senior staff using the phrase "the gate," where approval has to be given. The business has to be involved in the approval and deciding what priorities it has at that stage.

Gardner: Ian is describing being able to double their innovation budget, cut their project numbers in half, get buy-in from leadership, a sense of cooperation across the organizational boundaries. Is this typical? How would you describe this in terms of the industry at large?

Garrett: It's a typical situation that we see in a lot of organizations, even in very mature, even global and enterprise organizations that struggle with these challenges of organizational alignment and processes to support that. Project selection identification and transitioning to survey is the common problem we see.

With Nottingham Trent, we regulated it very quickly through that organizational design, then into the process to support that, and then working out what are the catalog and services that they offer. How do we then build that into projects and programs and then manage that into service transition?

It's very common. We see it in a lot of places. More mature organizations believe they do this very effectively. Nottingham Trent acknowledged that they needed help. It probably put them ahead of a lot of other organizations, especially in university space, which is a fast moving sector in the U.K., to be able to do something that many other large organizations just can't do.

If you build the right organizational relationship and engagement model, you take the workshop approach that we have up front and take your organization through that, right through to something tangible that’s delivering the real outcome in the business that’s very visible and usable. I think that’s very different than having different organizations do different types of consulting.

Gardner: We've come back to this workshop concept several times in discussion, I think that it's called the Transformation Experience Workshop. Why is that so powerful?

Garrett: It's something we've used for a few years now, something we developed in-house and we see as a really effective mechanism. It starts off in a fairly classic way of where are we, the current state, looking at future state, and workshop of the organization through that. But it's done in a very live, interactive way.

So it's not a classic style workshop. We walk people around the room. We take them on a journey, and we bring them together through that process. As Ian said, if you didn’t attend the early workshop process, then you struggle sometimes to buy into it. It takes more time, and we end up reiterating things later on. The Transformation Experience Workshop is a way of bringing people together and bringing them around their own problems in a very active physical way.

We can do it in a small period of time, but usually people dedicate a day or so to that process. What they get out of it is that they bring themselves together around the challenges, the problems, and as Ian said, the quick wins, the things we can then go and address quickly. So it has a very different feel and a very different outcome than a classic workshop approach that many consulting firms have.

Gardner: And Ian, is this something now that you’re building on?

Griffiths: That's correct. We produced a lot of what we call Level 3 processes from this and we looked at what our customers felt. We found that we’re having regular discussions about how we can tweak the diagrams and the systems that we’ve got in place. We see it very much as a live document, a live methodology and we’re looking at ways we can improve as time goes on.

It's important that you have all your senior staff together designing the system from the start. We found that if people miss the early workshop, we tended to go back around the loop again. So I would say get your staff together and devote enough energy to it.

But don’t go into all the detail. Leave your staff on the ground, who’ve got more knowledge of the details inner workings of some elements of it, to do some work so they feel some ownership. And very quickly get an appreciation with your senior staff within your organization, not within IS, but from outside the IS department, of what you're doing and what you're trying to achieve.

But in the end, you need a few quick wins. In other words, if you can get a couple of projects working through the scheme quickly, people begin to think it's going to work.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: David Norris, Practice Leader - Analytics, Bloor Research Posted: 16th January 2012 Copyright Bloor Research © 2012

We all know that the essence of Marketing is all about the 4 Ps, the right Product, in the right Place, at the right Price, with the right Promotion. But in this fast moving, very fragmented, and diverse market how are you supposed to find the right Place? I know that when I worked in marketing, of all the elements of the marketing mix that we found the hardest that was it. After what I have seen today I believe that, at least for those of us in the UK, there is an answer. The Data Works represents one of the most significant aids to Sales and Marketing that I have seen in a long time. I am adding it to my list of best buys to get us out of the economic bad times!

The engine behind the scenes in The Data Works is QlikView, which, for those of you who may have read these columns before will know, is a tool that we hold in very high regard. What QlikView does is to allow business users to slice and dice data, that is to dig down through data in an intuitive way to discover things of value, and this is achieved without having to pre warn IT that this is how we want the data set up, and having to then be trained to analyse the data. It is simple, it is fast and it is readily understood. But an engine, no matter how good, is only as good as the data fed to it. It is here that The Data Works excels. They take data from the very best sources, and they combine it to create a uniquely powerful data set. The data is coming from such sources as Companies House, Experian, Yell, The Royal Mail, The Office of National Statistics, The National and Property registry etc so it is the data that people are electing to judged by. The end result is a database of over 4 million active businesses spanning the UK, with up to 300 attributes per company. This then is the basis for you to go and find those companies that are most likely to be your ideal trading partners.

To illustrate what line of thought analysis is about suppose you want to find companies in North West London, who have a Turkish background, have a Marketing Director who is a woman, and who trade in confectionary, that is what QlikView allows you to do, and that is what the data assembled by The Data Work supports.

I cannot think of anyone working in the B2B market that does not need this sort of data. The big boys can probably do it for themselves, but for the rest of us there is The Data Works. This is a much needed solution, being delivered using one of the best tools available, with data that is, beyond question, the best there is, with a price that, if I am a judge, is fair, which leaves you wondering just why would anyone in Marketing not want to solve one of their biggest headaches right now? So, my advise is stop guessing, go out and look at The Data Works and go create new markets rather than analysing sub standard data in spreadsheets, in the vain hope you find something that useful, It's as simple as that, just do it! The Data Works demands to be taken seriously as the Marketing aid of 2012 and beyond.

By: Neil Ward-Dutton, Research Director, MWD Advisors Posted: 13th January 2012 This work is licensed under a Creative Commons License

Ever since I read Geoffrey Moore’s Future of IT paper introducing the concept of systems of engagement (published last year), I’ve had a slight uneasiness about the ongoing discourse. For a long time it was nothing I could put my finger on, but in some recent conversations about how technology can improve business agility, I finally realised what had been niggling me. In this blog I’m going to explain what I think is missing.

Systems of engagement vs. systems of record: a summary
If you’re not familiar with the narrative around systems of engagement, in brief it goes like this:

‘Systems of record’ – systems which manage core business information types and provide facilities for processing the information in place (think financial management, logistics, billing, CRM, and so on) – are no longer a source of competitive differentiation: they are a necessary condition of doing business, but because their capabilities are so commonplace their presence is merely table stakes. Increasingly, globalisation is forcing companies to focus more strongly on their core capabilities, and work more broadly and deeply with other companies and agencies to deliver value to customers. This means that communication and collaboration are more important than ever before. And this is where the concept of ‘systems of engagement’ comes in: the idea is that in this more distributed and  collaborative business environment, the ability to adopt the advanced communication and collaboration tools that people are familiar with in their home lives will become the new focus for competitive differentiation through technology in business environments.

Something’s missing
Before I go any further, I should be clear: I have nothing at all against the assertion that the business value of technology is shifting away from how well information is managed in place (by ‘systems of record’), towards how well information is communicated – between systems, people and organisations. I’m also absolutely in agreement with the assertion that social collaboration technologies have some really exciting roles to play in driving business improvement.

But based on my research and industry experience I think there’s something really important that’s omitted from the narrative that I’ve seen: and that’s to do with how the changing nature of value chains, combined with changing customer expectations, regulatory pressures and so on, is forcing a new kind of appraisal of how work needs to be supported by technology. Specifically, the key part of the ‘how’ here that we need to concentrate on is to do with the co-ordination of work.

It’s no longer enough (and in fact it hasn’t been enough for some years) to give people standalone tools (including those ‘systems of record’) and expect them to just get on with it and produce great work. The pressures highlighted above mean that to deliver great business performance, optimising work at the level of an individual’s contribution is a long way from enough; businesses need to be able to optimise how work gets done at a much larger scale.

However I’m not talking here about changes like the outcomes of the BPR programmes of the 1980s and 1990s – rigid process ‘improvements’ that ushered in massive changes to administration and operations through top-down diktat and only offered crude transactional ‘solutions’ that attempted to control information, the focus of work and change in a highly centralised fashion.

Instead we need to invest in systems and cultures that allow technology to be woven more closely into a broad range of types of work, where that work naturally happens, to help actively coordinate how work gets done between people, departments, and companies – and also, crucially, to gather intelligence and metrics that organisations can use to improve coordination and drive better business results.

Introducing ‘systems of co-ordination‘
Of course, anyone who’s been following my ramblings for a while will know that what I’m talking about is the application of Business Process Management (BPM) technologies and techniques*. These technologies and techniques, when used properly, create systems of co-ordination that enable businesses to systematically manage and improve their knowledge about ‘what works in work’ for them, and apply that knowledge directly in an operational context across people, departments and even corporate boundaries.

In his Future of IT analysis, Moore mentions co-ordination as a responsibility of systems engagement in passing, but I think that a mention in passing just isn’t enough. Given the length of time I’ve spent talking to companies that have implemented process improvement programmes and projects I might have too much bias colouring my view, but I think the value of these systems of co-ordination is just as crucial an element of business-technology strategy and investment as are systems of record and systems of engagement.

In an enterprise people don’t collaborate just for the fun of it – people collaborate to “get work done”. But how does knowledge about the best way to do work, and get the best results from work, get encoded, applied, managed and improved? Neither systems of record nor systems of engagement (at least, in terms of how the latter are painted in the discourse I’ve seen so far) have anything to offer in this respect.

Greasing the wheels between systems of engagement and systems of record
To my mind, then, the narrative needs to be refined. Businesses need to start to figure about the roles that systems of engagement should play in helping them maximise the effectiveness of business interactions, and consider how systems of engagement should play alongside systems of record: but crucially, the interface between systems of engagement and systems of record should not be a direct one. Between these two system layers – the grease between the wheels – should be systems of co-ordination.

What do you think? Am I the only one who thinks we need to look at this more closely? I’d love to hear your comments.

*There’s been a lot of debate about the limits of the applicability of some BPM technology in the face of different types of work – structured vs. unstructured, planned vs. unplanned, goal-oriented, collaborative, and so on – but I’m explicitly avoiding those details here and talking at a general level for the purposes of this piece.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 13th January 2012 Copyright Interarbor Solutions © 2012

This guest post comes courtesy of Tony Baer’s OnStrategies blog. Tony is a senior analyst at Ovum.

When we last left Oracle’s big data plans, there was definitely a missing piece. Oracle’s Big Data Appliance, as initially disclosed at last fall’s OpenWorld, was a vague plan that appeared to be positioned primarily as an appliance that would accompany and feed data to Exadata. Oracle did specify some utilities, such as an enterprise version of the open source R statistical processing program that was designed for multithreaded execution, plus a distribution of a NoSQL database based on Oracle’s BerkeleyDB as an alternative to Apache Hive. But the emphasis appeared to be extraction and transformation of data for Exadata via Oracle’s own utilities that were optimized for its platform.

As such, Oracle’s plan for Hadoop was competition, not for Cloudera (or Hortonworks), which featured a full Apache Hadoop platform, but EMC, which offered a comparable, appliance-based strategy that pairs Hadoop with an Advanced SQL data store; and IBM, which took a different approach by emphasizing Hadoop as an analytics platform destination enhanced with text and predictive analytics engines, and other features such as unique query languages and file systems.

Oracle’s initial Hadoop blueprint lacked explicit support of many pieces of the Hadoop stack such as HBase, Hive, Pig, Zookeeper, and Avro. No more. With Oracle’s announcement of general availability of the big data appliance, it is filling in the blanks by disclosing that it is OEM’ing Cloudera’s CDH Hadoop distribution and, more importantly, the management tooling that is key to its revenue stream. For Oracle, OEM’ing Cloudera’s Hadoop offering fully fleshes out its Hadoop distribution and positions it as a full-fledged analytic platform in its own right; for Cloudera, the deal is a coup that will help establish its distribution as the reference. It is fully consistent with Cloudera’s goal to become the Red Hat of Hadoop as it does not aspire to spread its footprint into applications or frameworks.

Question of acquisition
Of course, whenever you put Oracle in the same sentence as OEM deal, the question of acquisition inevitably pops up. There are several reasons why an Oracle acquisition of Cloudera is unlikely.

1. Little upside for Oracle. While Oracle likes to assert maximum control of the stack, from software to hardware, its foray into productizing its own support for Red Hat Enterprise Linux has been strictly defensive; its offering has not weakened Red Hat.
   
   
2. Scant leverage. Compare Hadoop to MySQL and you have a Tale of Two Open Source projects. One is hosted and controlled by Apache, the other is hosted and controlled by Oracle. As a result, while Oracle can change licensing terms for MySQL, which it owns, it has no such control over Hadoop. Were Oracle to buy Cloudera, another provider could easily move in to fill the vacuum. The same would happen to Cloudera if, as a prelude to such a deal, it began forking from the Apache project with its own proprietary adds-ons or substitutions.

OEM deals are a major stage of building the market. Cloudera has used its first mover advantage with Hadoop well with deals with Dell, and now Oracle. Microsoft, in turn, has decided to keep the “competition” honest by signing up Hortonworks to (eventually) deliver the Hadoop engine for Azure.

OEM deals are important for attaining another key goal in developing the Hadoop market: defining the core stack – as we’ve ranted about previously. Just as Linux took off once a robust kernel was defined, the script will be identical for Hadoop. With IBM and EMC/MapR forking the Apache stack at the core file system level, and with niche providers like Hadapt offering replacement for HBase and Hive, there is growing variability in the Hadoop stack. However, to develop the third party ecosystem that will be vital to the development of Hadoop, a common target (and APIs for where the forks occur) must emerge. A year from now, the outlines of the market’s decision on what makes Hadoop Hadoop will become clear.

The final piece of the trifecta will be commitments from the Accentures and Deloittes of the world to develop practices based on specific Hadoop platforms. For now they are still keeping their cards close to their vests.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 12th January 2012 Copyright Interarbor Solutions © 2012

This BriefingsDirect thought leadership interview comes in conjunction with The Open Group Conference this month in San Francisco.

The conference will focus on how IT and enterprise architecture support enterprise transformation. Speakers in conference events will also explore the latest in service oriented architecture (SOA), cloud computing, and security.

We’re now joined by of the main speakers, Jeanne Ross, Director and Principal Research Scientist at the MIT Center for Information Systems Research. Jeanne studies how firms develop competitive advantage through the implementation and reuse of digitized platforms.

She is also the co-author of three books: IT Governance: How Top Performers Manage IT Decision Rights for Superior Results, Enterprise Architecture As Strategy: Creating a Foundation for Business Execution, and IT Savvy: What Top Executives Must Know to Go from Pain to Gain.

As a lead-in to her Open Group presentation on how adoption of enterprise architecture (EA) leads to greater efficiencies and better business agility, Ross explains how enterprise architects have helped lead the way to successful business transformations. The interview is moderated by Dana Gardner, Principal Analyst at Interarbor Solutions. [Disclosure: The Open Group is a sponsor of BriefingsDirect podcasts.]

Here are some excerpts:

Gardner: How you measure or determine that enterprise architects and their practices are intrinsic to successful business transformations?

Ross: That’s a great question. Today, there remains kind of a leap of faith in recognizing that companies that are well-architected will, in fact, perform better, partly because you can be well-architected and perform badly. Or if we look at companies that are very young and have no competitors, they can be very poorly architected and achieve quite remarkably in the marketplace.

But what we can ascribe to architecture is that when companies have competition, then they can establish any kind of performance target they want, whether it’s faster revenue growth or better profitability, and then architect themselves so they can achieve their goals. Then, we can monitor that.

We do have evidence in repeated case studies of companies that set goals, defined an architecture, started to build the capabilities associated with that architecture, and did indeed improve their performance. We have wonderful case study results that should be very reaffirming. I accept that they are not conclusive.

We also have statistical support in some of the work we've done that shows that high performers in our sample of 102 companies, in fact, had greater architecture maturity. They had deployed a number of practices associated with good architecture.

Gardner: Is there something that’s new about this, rather than just trying to reengineer something?

Ross: Yes, the thing we're learning about enterprise architecture is that there's a cultural shift that takes place in an organization, when it commits to doing business in a new way, and that cultural shift starts with abandoning a culture of heroes and accepting a culture of discipline.

Nobody wants to get rid of the heroes in their company. Heroes are people who see a problem and solve it. But we do want to get past heroes sub-optimizing. What companies traditionally did before they started thinking about what architecture would mean, is they relied on individuals to do what seemed best and that clearly can sub-optimize in an environment that increasingly is global and requires things like a single face to the customer.

What we're trying to do is adopt a culture of discipline, where there are certain things that people throughout an enterprise understand are the way things need to be done, so that we actually can operate as an enterprise, not as individuals all trying to do the best thing based on our own experience.

The fundamental difference of being an architected firm is that there is some underlying discipline. I'll caution you that what tends to happen is great architects really embrace the discipline. They love the discipline. They understand the discipline, and there is a reluctance to accept that that’s not the only thing we need in our organization. There are times when ad hoc behaviors enable us to be much more innovative and much more responsive and they are exactly what we need to be doing.

So there is a cultural shift that is critical to understanding what it is to be architected. That’s the difference between a successful firm that’s successful because it hasn’t gotten into a world of really tough competition or restrictions on spending and things like that and an organization that is trying to compete in a global economy.

Gardner: What then is the proper role of the architect?

Ross: The architect plays a really critical role in representing the need for this discipline, for some standards in the organization, and for understanding the importance of shared definitions for data. The architect should be able to create a very constructive tension in the organization, and that’s the tension between individuality, innovation, local responsiveness, and the need for enterprise thinking, standardization, and discipline.

Normally, in most companies, the architect’s role will be the enforcer of discipline, standardization and enterprise thinking. ...We want to be architected enough to be efficient, to be able to reuse those things we need to reuse, to be agile, but we don’t want to start embracing architecture for architecture’s sake or discipline for discipline’s sake.

We really just need architecture to pull out unnecessary cost and to enable desirable reusability. And the architect is typically going to be the person representing that enterprise view and helping everyone understand the benefits of understanding that enterprise view, so that everybody who can easily or more easily see the local view is constantly working with architects to balance those two requirements.

Gardner: Is this a particularly good time, from your vantage point, to undertake enterprise architecture?

Ross: It’s a great time for most companies. There will be exceptions that I'll talk about in a minute. One thing we learned early on in the research is that companies who were best at adopting architecture and implementing it effectively had cost pressures. What happens when you have cost pressures is that you're forced to make tough decisions.

If you have all the money in the world, you're not forced to make tough decisions. Architecture is all about making tough decisions, understanding your tradeoffs, and recognizing that you're going to get some things that you want and you are going to sacrifice others.

If you don't see that, if you just say, "We're going to solve that by spending more money," it becomes nearly impossible to become architected. This is why investment banks are invariably very badly architected, and most people in investment banks are very aware of that. It’s just very hard to do anything other than say, "If that’s important to us, let’s spend more money and let’s get it." One thing you can't get by spending more money is discipline, and architecture is very tightly related to discipline.

In a tough economy, when competition is increasingly global and marketplaces are shifting, this ability to make tough decisions is going to be essential. Opportunities to save costs are going to be really valued, and architecture invariably helps companies save money. The ability to reuse, and thus rapidly seize the next related business opportunity, is also going to be highly valued.

The thing you have to be careful of is that if you see your markets disappearing, if your product is outdated, or your whole industry is being redefined, as we have seen in things like media, you have to be ready to innovate. Architecture can restrict your innovative gene, by saying, "Wait, wait, wait. We want to slow down. We want to do things on our platform." That can be very dangerous, if you are really facing disruptive technology or market changes.

So you always have to have that eye out there that says, "When is what we built that’s stable actually constraining us too much? When is it preventing important innovation?" For a lot of architects, that’s going to be tough, because you start to love the architecture, the standards, and the discipline. You love what you've created, but if it isn’t right for the market you're facing, you have to be ready to let it go and go seize the next opportunity.

Gardner: Perhaps this environment is the best of all worlds, because we have that discipline on the costs which forces hard decisions, as you say. We also have a lot of these innovative IT trends that would almost force you to look at doing things differently. I'm thinking again of cloud, mobile, the big data issues, and even social-media types of effects.

Ross: Absolutely. We should all look at it that way and say, "What a wonderful world we live in." One of the companies that I find quite remarkable in their ability to, on the one hand, embrace discipline and architecture, and on the other hand, constantly innovate, is USAA. I'm sure I'll talk about them a little bit at the conference.

This is a company that just totally understands the importance of discipline around customer service. They're off the charts in their customer satisfaction.

They're a financial services institution. Most financial services institutions just drool over USAA’s customer satisfaction ratings, but they've done this by combining this idea of discipline around the customer. We have a single customer file. We have an enterprise view of that customer. We constantly standardize those practices and processes that will ensure that we understand the customer and we deliver the products and services they need. They have enormous discipline around these things.

Simultaneously, they have people working constantly around innovation. They were the first company to see the need for this deposit with your iPhone. Take a picture of your check and it’s automatically deposited into your account. They were nearly a year ahead of the next company that came up with that service.

The way they see it is that for any new technology that comes out, our customer will want to use it. We've got to be there the day after the technology comes out. They obviously haven't been able to achieve that, but that’s their goal. If they can make deals with R&D companies that are coming up with new technologies, they're going to make them, so that they can be ready with their product when the thing actually becomes commercial.

So it's certainly possible for a company to be both innovative and responsive to what’s going on in the technology world and disciplined and cost effective around customer service, order-to-cash, and those other underlying critical requirements in your organization. But it's not easy, and that's why USAA is quite remarkable. They've pulled it off and they are a lesson for many other companies.

Gardner: Is The Open Group a good forum for your message and your research, and if so, why?

Ross: The Open Group is great for me, because there is so much serious thinking in The Open Group about what architecture is, how it adds value, and how we do it well. For me to touch base with people in The Open Group is really valuable, and for me to touch base to share my research and hear the push back, the debate, or the value add is perfect, because these are people who are living it every day.

Gardner: Are there any other major themes that you'll be discussing at the conference coming up that you might want to share with us?

Ross: One thing we have observed in our cases that is more and more important to architects is that the companies are struggling more than we realized with using their platforms well.

I'm not sure that architects or people in IT always see this. You build something that’s phenomenally good and appropriate for the business and then you just assume, that if you give them a little training, they'll use it well.

That’s actually been a remarkable struggle for organizations. One of our research projects right now is called "Working Smarter on Your Digitized Platform." When we go out, we find there aren't very many companies that have come anywhere close to leveraging their platforms the way they might have imagined and certainly the way an architect would have imagined.

It's harder than we thought. It requires persistent coaching. It's not about training, but persistent coaching. It requires enormous clarity of what the organization is trying to do, and organizations change fast. Clarity is a lot harder to achieve than we think it ought to be.

The message for architects would be: here you are trying to get really good at being a great architect. To add value to your organization, you actually have to understand one more thing: how effectively are people in your company adopting the capabilities and leveraging them effectively? At some point, the value add of the architecture is diminished by the fact that people don't get it. They don’t understand what they should be able to do.

We're going to see architects spending a little more time understanding what their leadership is capable of and what capabilities they'll be able to leverage in the organization, as opposed to which on a rational basis seem like a really good idea.

Gardner: When you're an organization and you've decided that you do want to transform and take advantage of unique opportunities for either technical disruption or market discipline, how do you go about getting more structure, more of an architecture?

Ross: That's idiosyncratic to some extent, because in your dream world, what happens is that the CEO announces, "This is what we are going to be five years from now. This is how we are going to operate and I expect everyone to get on board." The vision is clear and the commitment is clear. Then the architects can just say, and most architects are totally capable of this, "Oh, well then, here are the capabilities we need to build. Let’s just go build them and then we'll live happily ever after."

The problem is that’s rarely the way you get to start. Invariably, the CEO is looking at the need for some acquisitions, some new markets, and all kinds of pressures. The last thing you're getting is some clarity around the vision of an operating model that would define your critical architectural capabilities.

What ends up happening instead is architects recognize key business leaders who understand the need for reused standardization, process discipline, whatever it is, and they're very pragmatic about it. They say, "What do you need here to develop an enterprise view of the customer, or what’s limiting your ability to move into the next market?"

And they have to pragmatically develop what the organization can use, as opposed to defining the organizational vision and then the big picture view of the enterprise architecture.

So in practice, it's a much more pragmatic process than what we would imagine when we, for example, write books on how to do enterprise architecture. The best architects are listening very hard to who is asking for what kind of capability. When they see real demand and real leadership around certain enterprise capabilities, they focus their attention on addressing those, in the context of what they realize will be a bigger picture over time.

They can already see the unfolding bigger picture, but there’s no management commitment yet. So they stick to the capabilities that they are confident the organization will use. That’s the way they get the momentum to build. That is more art than science and it really distinguishes the most successful architects.

Listen to the podcast. Find it on iTunes/iPod. Read a full transcript or download a copy.

By: Dana Gardner, Principal Analyst, Interarbor Solutions Posted: 11th January 2012 Copyright Interarbor Solutions © 2012

Genuitec, LLC has revamped its MobiOne development tool to allow Windows operating system users to design and build App Store-ready iOS apps—native apps for iPhone, iPad and iPod Touch—without using a Mac. This means there is no longer an additional expense to buy a Mac machine or learn Objective-C to design apps that operate natively on iOS devices.

Previously, the Flower Mound, Tex. company's MobiOne supported a webapp-only model that allowed design of webapps that run on iOS devices. Now, users can design native apps or webapps with the same design files, using AppCenter, a cloud technology that Genuitec engineered, that allows app designers to test their native and webapps in a private Genuitec cloud. [Disclosure: Genuitec is a sponsor of BriefingsDirect podcasts.]

“By removing the barriers to entry for iOS app design and building, MobiOne is truly at the forefront of making mobile technologies accessible to the masses," said Wayne Parrott, vice president of product development. "If a Windows users has enough skill to design a PowerPoint slide, they can design and build iPhone and iPad apps with ease. Web developers with HTML5 and CSS3 skills will see even greater productivity.”

MobiOne is designed for web developers, marketing departments, business consultants, and anyone who wants to create and build App Store-ready iOS applications and webapps. MobiOne uses drag-and-drop functionality similar to stringing together a PowerPoint presentation, but has a powerful engine that allows users to build iOS apps or webapps from the same code base.

That engine is the AppCenter technology, which allows for easy testing of apps and webapps over the air using iOS 4+ or through iTunes. Testing links can be shared via email or SMS for multiple device testing and previews.

To learn more about the MobiOne Studio, go to http://www.genuitec.com/mobile/docs/highlights/current/. A 15-day free trial is available at: http://www.genuitec.com/mobile/download.html. After the free trial, the cost is $99 per license.

By: David Norfolk, Practice Leader - Development, Bloor Research Posted: 9th January 2012 Copyright Bloor Research © 2012

I am always rather concerned about the testing of large, complex, virtualised (cloud) automated business services, especially those involving 3rd-party applications such as SAP. Constraints on resources and, especially, time, may result in testing being compromised.

The complexity of such systems demands huge testing resources for conventional testing and then there's the danger that once you've built your code it all falls apart in integration testing and you're condemned to expensive rework. The answer, in part, is to start integration testing early, before all the code is finished - but building and maintaining an appropriate testing environment might be just about as big a job as building the business system - especially if you need experts in 3rd-party software to set up the test cases.

The real killer is time. Implementation delays cost the business money (assuming that the software is actually productive for the business) and avoiding such delays can discourage change and make the system less agile, even if it doesn't encourage you to compromise the testing as well.

Green Hat promises to address these issues by virtualising the testing environment and allowing developers to get the environment to process test data without needing application experts to input the data with external applications. Integration testing can start early, while integration defects have limited scope and are still cheap to address, because you can provide a virtualised simulation of anything you haven't written yet and just test what you're working on.

Of course, the devil is in the detail but customer stories suggest that this works well. There is the issue that you need confidence that the virtualised behaviours are exactly the same as the real thing (and you may need a final integration test on a real production image, for critical systems at least) but I'm told that developers find it easy to distinguish defects in what they're building from any possible defects in the accuracy of the virtualised environment.

The bottom line is the customer stories. A major telecom carrier reports testing time reduced by 50%, saving $800K over 3 years. A leading global financial services firm with integrated third-party systems that are often unavailable for testing, reduced manual testing from 10 days to 10 minutes, thus saving over $7 million to date. And so on. Obviously, Green Hat is targeting large enterprises but a SaaS offering providing low cost of entry is on the horizon.

Now, IBM has announced its intent to acquire Green Hat. It tells me that it will slot into its Rational product line and that there's little significant overlap with its existing tools. IBM should be getting good at integrating acquired technology, without losing the key people that made it worth acquiring, by now. I think this is an important acquisition for IBM, especially if IBM's resources can make it more widely available and, possibly, available to smaller organisations and start-ups.

I believe that Green Hat addresses a very significant yet, often, unrecognised issue facing developers today: the need for early and fast integration testing of complicated business automation environments involving external services and significant integration with 3rd-party applications. Early unit testing isn't enough.