Analytics in the Cloud: 5 Lessons Learned

CIO - Every company-from the smallest start-up to the largest firm-needs to be agile in today's market to respond to changing dynamics and new competition. But these days it's often the smaller companies who are better positioned to adapt: as the barriers to entry have decreased, emerging companies now have access to data streams-and techniques for analyzing them-that used to be the exclusive province of the largest companies. At the same time, the CIOs of larger organizations now find themselves as much bound by their legacy systems and data as they are empowered by them. The costs of managing these legacy systems are getting in the way: too much of the budget goes to maintenance, and not enough is left over for new development and technologies.

Nowhere is this dynamic more apparent than with Business Intelligence (BI). As BI once again rises to the top of priority and wish lists, CIOs are struggling with the costs of meeting internal demands while keeping within their budgets, and still finding time for innovation. The costs of proprietary servers and storage devices, as well as the space and energy to manage them, are off the charts and highly visible to every CFO, CTO and procurement professional. Proliferating copies of data into multiple one-off analytical systems-seemingly one for every question to be asked-only adds to the costs, and even new "data appliances" can cost in the tens of millions to scale up as requirements grow.

Clearly, new approaches are needed to cost-effectively scale BI systems while meeting the demand for information on the front lines. Here are some examples of how forward-looking organizations are doing large-scale analytics in the cloud to break the logjam. (For more background on the cloud technology being discussed here, see CIO.com's recent article, "Greenplum Spins 'Enterprise Data Cloud' Vision".)

1. Hold the line with commodity hardware. Most new analytic data engines run on inexpensive commodity hardware, transforming IT cost models and conventional wisdom about the costs of new systems. As Mark Dunlap, a consultant with Evergreen Technologies and a veteran of massive data warehouse projects at Amazon and Fox Interactive, puts it, "If you're using proprietary hardware, you're in a losing battle. Sooner or later, whatever company's developing that technology will not be able to keep up. We've seen it over and over and over again-they won't keep pace with what commodity systems are doing."

2. Buy capacity when you need it, not according to a closed appliance size Clint Johnson, VP of Business Intelligence at Zions Bancorporation, says he's avoiding locked-in purchase models as they tackle massive data challenges. "We like the ability to add hardware easily, incrementally," says Johnson. "Specialized appliances we looked at scaled in very specific size increments." Not only are those new purchases large, they may be substantially greater than near-term needs-but payment is not scaled to usage, it's by total capacity.