Rising data-loading speeds may be leaving most BI users behind

Computerworld - One of the more prosaic parts of data warehousing is getting information into a warehouse in the first place. Vendors that sell data-loading tools have long operated in the shadows of the business intelligence market, with little pizazz or glory.

Even in the isolated world of extract, transform and load (ETL) software, the focus has traditionally been more on the problem of cleansing and modifying data to prepare it for analytical uses. Data loading seemed to be an afterthought — a piece of cake — in comparison.

But that's changing as BI and analytics become more of a near-real-time affair at many companies. In addition, the biggest BI users now operate data warehouses larger than a petabyte in size and need to import huge amounts of data into them. For instance, decision-support database vendor Teradata Corp. says that eBay Inc. loads 50TB of online auction and purchase data on a daily basis.

Over the past few months, several start-ups and relatively unknown vendors have tried to take advantage of such needs by touting screaming-fast data-loading speeds that they claim to have achieved either in the lab or with users in the field.

Database start-up Greenplum Inc. said customer Fox Interactive Media Inc. routinely loads 2TB of Web usage data into its data warehouse in half an hour. Meanwhile, rival Aster Data Systems Inc. claimed that its nCluster technology supports load speeds of up to 3.6TB per hour.

Not to be outdone, Expressor Software Corp., an ETL start-up offering so-called semantic data integration tools, said in-house tests show that its data processing engine can scale to a rate of nearly 11TB per hour.

Even Syncsort Inc., a 41-year-old company that began as a mainframe software vendor, has gotten into the act. Syncsort said that in lab tests, its data integration software loaded 5.4TB of data into a warehouse built around Vertica Systems Inc.'s column-based database in less than an hour.

If Syncsort and the other vendors are actually achieving those kinds of load rates, that's "really impressive," said James Kobielus, an analyst at Forrester Research Inc. "Anything above a terabyte per hour is good."

And what about more-established vendors? Two years ago, SAS Institute Inc. and Sun Microsystems Inc. demonstrated a SAS data warehouse running on Sun hardware and StorageTek disk arrays that could push through 1.7TB of data in 17 minutes — the equivalent of just under 6TB per hour.

But other big-name vendors have posted performance benchmarks that fall short of the load rates claimed by the upstarts. Last fall, for instance, Oracle Corp. and Hewlett-Packard Co. said their joint BI-oriented HP Oracle Database Machine could load up to 1TB per hour. And Microsoft Corp. said early last year that the data integration software built into SQL Server 2008 had loaded at a rate of 2.36TB per hour.