Emerging Appliances Speed Data Analysis

Bundled systems replacing relational databases at some firms

Companies that are fed up with sluggish processing speeds and the tedious chore of maintaining traditional databases are increasingly turning to data warehousing appliances to analyze large data sets.

The appliances, dedicated devices that include varying combinations of high-performance hardware with database, storage and other types of software, are usually preconfigured for specific tasks like strategic analysis.

O'Reilly Media Inc. last week was expecting to receive an open-source data warehouse appliance jointly developed by Sun Microsystems Inc. and San Mateo, Calif.-based Greenplum Inc. The appliance, unveiled late last month, includes a Solaris 10-based Sun Fire X4500 data server and Greenplum's massively parallel distribution of the open-source PostgreSQL database.

Spending Time on Data

Roger Magoulas, director of market research at Sebastopol, Calif.-based O'Reilly Media, said the book publisher plans to use the appliance to perform analysis on 1TB and 2TB data sets that the company expects will double in size every year. The data sets are mostly made up of unstructured text and "didn't lend themselves to adequate performance with regular databases," he said.

With the appliance, "you can spend more time on the business reason you have the data, rather than optimizing or tuning the database, because you have unlimited disk space and parallel database performance," Magoulas said.

O'Reilly now uses the open-source MySQL database running on a cluster of machines, he added. The appliance will replace those systems, he said.

Doug Mackie, vice president of enterprise technology at The NPD Group Inc., said his company has been using a data warehouse appliance from Datallegro Inc. in Aliso Viejo, Calif., since April as a query engine for its customers and to process point-of-sale data. The Port Washington, N.Y.-based market research firm has also formed a so-called sandbox test bed to evaluate other areas where the appliance could be used, according to Mackie.

With the appliance, NPD was able to reduce the time needed to process some queries from hours and days down to minutes and seconds, Mackie said. Previously, the company completed the queries on an Oracle database.

'Single-Minded Device'

"It is a single-minded device, and it just runs the queries," Mackie said. "We think we will see efficiency and speed and that DBA activity requirements will be greatly reduced from an appliance versus continuing with the relational databases."

However, he noted that while the appliances have "great potential," the technology is relatively new and the vendors are inexperienced.

IDC in Framingham, Mass., estimates that the appliance market will reach $500 million in five years from $50 million to $75 million today.

Mark Beyer, an analyst at Gartner Inc. in Stamford, Conn., said companies are buying the appliances mainly because they are preconfigured and therefore easier to put into production than traditional databases. However, he cautioned that an appliance tuned for a specific use "will fail in some cases, and in other cases it will have performance problems" if it's used for a different application.

Ahold USA Inc., which operates the Stop & Shop, Giant Food and Tops Markets supermarket chains, has been using Netezza Corp.'s Netezza Performance Server appliance to analyze transactional level data from its Stop & Shop stores for the past two years, said Brent Buelo, manager of data warehouse services at the Quincy, Mass.-based company.

Next year, Ahold plans to begin using the appliance to analyze data from another of its chains.

Ahold users are getting faster queries with the appliances. One report that took 72 hours to run using the former Oracle-based system now takes nine hours, said Buelo. He added that users can now get answers to queries that couldn't have been answered before because queries took too long and timed out before providing a response.

Copyright © 2006 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon