Zoom zoom: Upstarts speed past big BI vendors in data warehouse loading speeds
They're boasting load speeds of 4TB an hour or more
Computerworld - One of the more prosaic parts of data warehousing is, well, getting the data into the warehouse.
This has long been handled by vendors that are expert in the field of extract, transform and load (ETL). Even there, innovation focused more on the problem of transforming the data. Loading the data seemed a piece of cake by comparison.
That is, until business intelligence (BI) and analytics started becoming a round-the-clock affair. Also, today's biggest BI users -- banks, telecommunications providers, Web advertisers -- operate data warehouses larger than a petabyte in size and import huge swaths of data -- 50TB of data per day, as in the case of one of Teradata Inc.'s customers.
BI and ETL vendors are responding. The past several months have seen a number of start-ups and lesser-known firms touting screaming-fast data-loading speeds, both in the lab and in the field.
- Database start-up Greenplum Inc. said it has a customer routinely loading 2TB of data in half an hour, for an effective throughput of 4TB per hour.
- Rival database start-up Aster Data Systems Inc. claimed that its nCluster technology can enable customers to reach almost 4TB (specifically, 3.6TB) per hour.
- Data-integration vendor Syncsort Inc. said third-party-validated lab tests show its software can load 5.4TB of data into a Vertica Systems Inc. columnar data warehouse in under an hour.
- Not to be outdone, semantic data integration start-up Expressor Software Corp. claimed that in-house tests show its data-processing engine able to scale to nearly 11TB per hour.
"If they are really performing at this rate, it's quite significant and really impressive," said Jim Kobielus, an analyst at Forrester Research Inc., since "anything above a terabyte per hour is good."
Blazing past the incumbent BI and ETL vendors
What about the established firms? SAS Institute Inc. and Sun Microsystems Inc. two years ago demonstrated a SAS data warehouse running on Sun Microsystems hardware with StorageTek arrays that pushed through 1.7TB in 17 minutes, or the equivalent of nearly 6TB per hour.
But apart from SAS, other big-name vendors have posted data-integration performance benchmarks that fall well short of these upstarts.
- Three years ago, Informatica Corp. claimed its PowerCenter 8 software loaded data at a rate of 1.33 TB per hour. The company, which decline to comment today, hasn't posted any updated performance benchmarks.
- Oracle Corp. and Hewlett-Packard Co. last fall released the BI-oriented HP Oracle Database Machine, which they said loads data at up to 1TB per hour.
- Microsoft Corp. claimed at the launch of SQL Server 2008 a year ago that its SQL Server Integration Services 2008 had loaded the equivalent of 2.36TB in an hour.
- Seattle Children's Accelerates Citrix Login Times by 500% with Cross-Tier Insight Seattle Children's is a leading research hospital with a large and growing Citrix XenDesktop deployment. With ExtraHop, the IT team at Seattle Children's...
- McKesson Makes Application Hosting for Hospitals Faster, More Efficient With ExtraHop, McKesson identified the root cause of slow Citrix XenApp application launches and adopted a more intelligent, proactive IT operations model that...
- A Non-Geek's Big Data Playbook A visual playbook for the non-geek yet technically savvy business professional who is still trying to understand how big data impacts the enterprise...
- Reducing Expenses Starts with the Programming Skills of Operations Groups Reducing expenses can start with the programming skills of operations groups. Read this article to learn how to improve the bottom line, and...
- Charting Your Analytical Future - "Making predictive analytics part of your business processes" Webinar This session will show how predictive analytics can be used throughout the organization by anyone looking for answers and how organizations can make...
- Capturing Data in Motion: Delivering Real-Time Insight from Data Streams This webcast will help organizations of all types and sizes learn about a technology and business strategy for tapping into the wealth of... All Business Intelligence/Analytics White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!