Opinion: Grid an option for data management challenges

Computerworld - With EMC Corp.'s acquisition of Acxiom Corp.'s grid computing software for $30 million last month (see "EMC Partners With Acxiom to Build Grid-based BI Systems"), enterprise customers started opening their eyes to the fact that grid is not just about raw horsepower and CPU utilization for high-performance computing environments.

So what was it that Acxiom did so well with its grid environment that caught EMC's attention? To put it simply: data management.

Acxiom has a very popular data-integration application called AbiliTec. It took the "scale out" commodity hardware route to scale and support a growing number of transactions (as Google Inc. and Amazon.com Inc. have done) and then built its own grid software to manage this new environment. In an article on Acxiom's environment last year, Computerworld reported that its grid had grown to 6,000 Linux nodes, processing more than 50 billion AbiliTec transactions per month (see "Case Study: Acxiom Corp.'s Homegrown Grid").

Performance and reliability have been at the heart of Acxiom's data management grid story, but there are some other very specific enterprise data challenges where grid has already been used in research and science. Today, enterprises are increasingly evaluating the capabilities of grid infrastructure to resolve data management issues ... above and beyond data processing horsepower.

Transporting Massive Amounts of Data

Your typical enterprise is probably not going to be dealing with data on the petabyte (1 quadrillion-byte) level any time soon, like particle physicists in the online science realm do today.

However, many commercial entities do transport enormous files on a daily basis. Consider cases like the British Broadcasting Corp., where one hour of preprocessed high-definition broadcast averages about 280 gigabits. These organizations are working with grid technologies today to make their data assets accessible to field reporters and users across a distributed network.

Moving large data sets at high speeds between distributed sites is a common challenge in many industries. Oil and gas companies are perhaps the poster children for moving large data sets, which they accumulate through seismic analysis and reservoir analysis. Getting the "whole picture" to make sound business decisions requires pulling large quanta of data from many different locations.

Other markets with massive data-transport requirements include the automotive industry (for computer-aided analysis and simulations), semiconductor companies (for mask layout based on instruction sets) and pharmaceutical firms (for molecular matching and chiral synthesis), to name just a few.

Getting Data Out of Complex Storage Systems

Grid pros have popularized the expression that "access to the data is as important as access to compute resources." Sometimes in enterprises, the challenge with data access -- beyond the size of data sets -- is the complexity of the protocols associated with storage systems.