High-Speed Databases Rev Corporate Apps

Survey: Business Intelligence Maturity and the Quest for Better Performance BI @ The Speed of Business Data Protection and Availability with Oracle Active Data Guard 11g Sign up to receive Security Resource Alerts

January 16, 2006 (Computerworld) -- Relational database management systems have become all but ubiquitous in enterprise computing since 1970, when they were first devised by E.F. Codd. But as powerful and flexible as those databases are, they've proved inadequate for a handful of ultrademanding applications that have to process hundreds or thousands of transactions per second and never go down. Now, the very-high-performance database technologies that sprang up to serve these niche markets, such as options trading and telephone call processing, are poised to move into mainstream computing.

Image Credit: Richard Borge

Some of the new products simply move the action from disk to memory, where access is a million times faster. Others are more radical departures from tradition, such as "streaming" technologies that store queries and pass data through them rather than run queries against stored data. Still others have found clever ways to sidestep much of the overhead -- such as table locking -- associated with the traditional RDBMS.
While some of these products do "store" data in memory-resident data-bases -- either relational or object-oriented -- the tools are primarily designed to speed transaction processing and analytics, not to act as data repositories.
Thanks for the Memory
Interact Inc., a Lincoln, Neb.-based communications service provider, has for more than 10 years used the in-memory database capabilities of Hewlett-Packard Co.'s NonStop servers to do real-time pricing of incoming telephone calls. But the big, expensive computers were overkill for some Interact customers, such as small mobile telephony resellers, says Tom Massey, director of business development.
So 18 months ago, Interact began to offer a call-pricing service that runs on Linux and Unix servers and uses Oracle Corp.'s TimesTen In-Memory database. "NonStop is big iron and more geared to larger operators," Massey says. "Linux and Unix platforms scale down much better, and operators often prefer them because they are not knowledgeable about NonStop."
Oracle acquired the TimesTen technology last June. Oracle saw the in-memory database as a way to extend its enterprise back-end data storage capabilities to high-performance real-time applications such as Interact's. Interact uses Oracle for back-end data storage as well and does not yet interface those databases with TimesTen, but Massey says he plans to do so.
"We need sub-10-millisecond response time, and you can't get that performance out of an Oracle relational database," says Ed McKee, director of applications at Interact. "To get that kind of performance, the amount of iron you'd have to have would be very significant."
On the other hand, he notes, the in-memory TimesTen product isn't suitable for large-scale data archiving. Interact can serve 1 million telephone subscribers with just 2GB of data in memory because only customer balance information is needed online.
Aspect Software Inc. in Westford, Mass., uses TimesTen for call center services. Traditional databases don't have the sub-500-millisecond call- routing capability it requires, says Chief Technology Officer Gary Barnett. Aspect's customers call centers typically have big databases of customer history behind them but cache key information upfront, in memory, for near-instantaneous response to customer requests, he says.
But deciding what data to replicate forward, and how often, can be tricky, Barnett warns. "There's a trade-off. The more data you have [in memory], the more intelligent we can be in routing calls. But the more data in real time, the more expensive it is."
Some users choose a memory- resident product for its features and then gain high performance as a byproduct. For example, Interstate Hotels & Resorts Inc. chose the TM1 financial analysis tool from Applix Inc. in Westboro, Mass., because it was easy to use. "We wanted to consolidate all the hotels to close the books each month," says Paul Bushman, senior vice president for IT at the Arlington, Va.-based manager of more than 300 hotels. "But now we use it on a daily basis."
TM1, which Applix calls "the world's fastest business intelligence analytical engine," moves disk-resident data from Oracle databases in Interstate's accounting system into memory in an Excel spreadsheet format. From there, users without technical expertise can run what-if financial models as well as do financial rollups by a variety of user-specified criteria.
They can also perform online consolidations of the type more typically performed in month-end batch processes, Bushman says. "In the accounting system, to produce one financial statement for one hotel for one month could take 30 minutes in a relational database," he says. "But here we can do a consolidation of all 300 hotels in a couple of seconds."
Fly-by Analysis
While most of these "real time" products achieve their scorching performance by moving data into memory, one high-performance product -- the StreamBase "stream-processing engine" from StreamBase Systems Inc. in Lexington, Mass. -- just grabs incoming data and analyzes it as it flies by.
StreamBase applications use an "inbound" query-processing model, in which records are processed before they're indexed and stored. The records flow through the query, which can also transform the data while it's moving.
Vision Systems & Technology Inc. (VSTI) in Ellicott City, Md., is helping several defense and intelligence agencies evaluate StreamBase prototypes. StreamBase can filter torrents of incoming data -- structured or unstructured -- and decide on the fly which should be presented to an analyst at once, which can be stored for later queries and which can be discarded, says Carol Lundquist, an IT consultant at VSTI.
The technology can generate alerts when a passing record contains, for example, a certain name or phone number. "You can put keywords in an Oracle table, and anytime a keyword is added, it gets dumped down to StreamBase immediately," Lundquist says.

"Some government systems are being flooded with data," she explains. "The Oracle systems are having trouble keeping up, and you get data falling on the floor." One government system Lundquist worked on loaded 1 billion records in a day, she says.
Filtering can be the salvation for some of those systems, says Bryan Harris, CTO of VSTI. "The idea is to load the needles, not the haystack," he says.
Harris says streaming technologies may complement rather than compete with traditional back-end RDBMSs. And they are not necessarily an alternative to the in-memory products, either, he says. "If you are doing queries across many different IT systems, that introduces a lot of processing across the entire network," Harris says. "In-memory data caching, if done right, can greatly reduce the amount of system resources used in total. But it doesn't really address the streaming issue." Depending on its mix of applications, a company could benefit from a combination of back-end databases, in-memory databases and streaming technology, he says.
Another variation on the in-memory database theme comes from Ants Software Inc. in Burlingame, Calif. Because Ants Data Server is a SQL-compliant relational database, the company says, it is readily compatible with the major back-end databases, such as Microsoft Corp. 's SQL Server, IBM 's DB2 and Informix, and products from Oracle, Sybase Inc. and MySQL AB. And because it can reside on disk, in memory or both, there's no need to build an interface between different front- and back-end databases. The combination of these characteristics makes Ants easily scalable, the company says.
But Ants' major claim to fame is that, although it uses relational technology, it avoids almost all of the table row locking that can slow a traditional RDBMS to a crawl under heavy loads. According to Ants CEO Boyd Pearce, the reason traditional RDBMSs fail under load is because they aren't very clever at detecting when a real conflict is occurring and a lock is needed. "There are very few cases where you really need locks," he says. Bellevue, Wash.-based Wireless Services Corp., which provides hosted data services to wireless carriers, chose Ants primarily because it provides an easy upgrade path from SQL Server. Small customers may reside entirely on a single SQL server, says CTO Curt Miller. When that system grows, it migrates to two or more SQL Server boxes. And when it reaches a certain point, Miller adds Ants servers for high-performance front-end message processing. Without Ants, the table-locking function at volumes much above 1,000 messages per minute "kills me," he says.
Because Ants supports the Open Database Connectivity standard, the migration is a snap. "Now if I want to use Ants, I just change the driver parameters to say, 'Talk to Ants,' " Miller says. While the traditional back-end database will continue to be the choice of most users with big data repositories, the rise of multitier systems, as well as an increasing number of applications that process torrents of data, seem likely to pull these newer technologies into the mainstream.

Print this Story

Send Us Feedback

E-mail this Story

Digg this Story

Slashdot this Story

Sidebar: Performance Is Where You Find It Sidebar: The Players High-Speed Databases Rev Corporate Apps

Sidebar: Driving Demand

"When white pages searches come up empty, it's often easier to let your fingers do the walking. It shouldn't be..." Read more... "Enterprise search continues to lag behind commerical search because companies lack a "findability" strategy, says one researcher...." Read more... Read more Business Intelligence posts or See all Blogs

DNS hole prompts synchronized patching effort by IT vendors Microsoft plugs nine holes in Windows, DNS, SQL Symantec warns of new Word attack More top stories...

Microsoft sets XP SP3 automatic download for Thursday Don't give Google a free pass on data collection, privacy advocates say after YouTube ruling XP SP3 to reach most users 'shortly,' says Microsoft

This old laptop: Revitalizing an aging notebook on the cheap All it takes is a couple hours and about $125 to breathe new life into an old laptop. Here's how. Bill Gates moves on Is Microsoft's Golden Age over? What are Gates' most memorable quotes? Find out in Computerworld's complete coverage of the end of the Bill Gates era at Microsoft. Five things you should never tell your boss There are some things your CIO definitely doesn't want to hear. Also don't miss the flipside, Five things you should always tell your boss. Hands-On: Firefox 3 fixes what's broke and keeps what's right With its latest version, Mozilla's browser continues to raise the bar for what Web browsers should be. Windows Vista A to Z Reviews, analyses, how-tos, visual tours, hot issues and predictions about Microsoft's new OS. More Continuing Coverage After 9/11: Homeland SecurityAsk a Premier 100 IT LeaderBlackBerry Legal BattleData Security BreachesElectronic VotingFirefoxH-1B VisasHP's Boardroom ScandalHandheld DevicesHurricane Katrina AftermathMicrosoft Legal IssuesMicrosoft's VistaOpen SourceRFID TechnologySCO's Linux FightSarbanes-Oxley ActSpamVoice Over IPWi-FiWindows Meta File VulnerabilityWindows XP Service Pack 3 IT Careers 2010 Four years from now, the IT field will be a vastly different place. Will you be ready? More Special Reports Premier 100 IT Leaders 2007 Best in Class2006 Computerworld Horizon Awards2006 Premier 100 IT Leaders2006 Salary Survey2007 Best Places to Work in IT2007 IT Forecast2007 Premier 100 IT Leaders40 Years of ComputerworldASPs, Take TwoBest Places to Work in IT 2003Best Places to Work in IT 2004Best Places to Work in IT 2005Best Places to Work in IT 2006Business Intelligence Home RunsBusiness Intelligence: Smarter BIBusiness Intelligence: The Future of BICRM Goes VerticalCRM: Sober CRMCareer Planning Guide 2005Careers: IT Profession 2010Computerworld Horizon Awards 2005Data Management: Mining for GemsData Management: Taming Data ChaosDevelopment: Hard-Workin' Web SitesDevelopment: New Tools New ChoicesDevelopment: The New World of Application DevelopmentDevelopment: The Web Services TsunamiDevelopment: Web Services HurdlesDisaster Recovery: Preparing for the WorstE-commerce Grows UpE-mail/Groupware: Big DecisionsFaces of Mobile ITForecast 2006Global MobileHardware: Multiple Cores, Multiple ChallengesHardware: On-Demand Un-HypedHardware: The Shape of Things to ComeHorizon Awards: 10 Cool Cutting Edge TechnologiesIT Management: Guide to Managing VendorsIT Management: Navigating Global ITIT Management: Recovery AheadIT Management: The Future of ITIT Management: The Resourceful Project ManagerInnovative Technology Awards 2004Management: Reinventing ITMicrosoft in TransitionMobile/Wireless Leaders and LaggardsMobile/Wireless: The Untethered WorkerMobile/Wireless: Tiny Gadgets, Huge CostsNetwork Management: Taking ControlNetworking: Turbulence Ahead!Networking: VoIP Goes MainstreamOperating Systems: In the Slow LaneOperating Systems: Linux Goes GlobalOperating Systems: Should You Nix Unix?Outsourcing: Offshore Buyer's GuideOutsourcing: Outsourcing DangersPremier 100 IT Leaders 2004 Best in ClassPremier 100 IT Leaders 2005 Best in ClassROI: Do the Math!Salary Survey 2003Salary Survey 2004Salary Survey 2005Security Action PlanSecurity: Compliance HeadachesSecurity: Proactive SecuritySecurity: Risk and RewardSecurity: Tips From Security ProsSouped-up SecurityStorage: Battling ComplexityStorage: Cheap & Secure Data StoresStorage: Stretching Your Storage DollarsStorage: The Lean Storage MachineStorage: The New Rules of StorageStorage: The Ultimate Backup GuideSupply Chain: Missing LinksSupply Chain: RFID Reality CheckThe Business of SecurityWeb 2.0 SecurityWireless: Wireless At Work All Zones Application Performance Zone Business Continuity Zone Data Center Management Zone Enterprise-Class Security Zone The File Data Management Zone Grid Computing on Windows Zone Security Management Zone ITIL Best Practices Zone The SAS Zone Storage Virtualization Zone Business Intelligence and Analytics Zone See your link here

Speeding the time to intelligence Get this Computerworld report free for a limited time, compliments of SAS. Time To Intelligence -- a concept defining how long it takes to get accurate and timely information into the hands of workers who need it most. Do it slower than your competitors and your company is toast. Do it faster, you scorch them. Business Intelligence is the key to optimizing Time To Intelligence, and success there is a combination of people, policies, and technology. Download this executive briefing  Web Security SaaS: The Next Generation of Web Security Download this whitepaper, free for a limited time, compliments of Webroot Software. (Source: Webroot Software) The Web is the new threat vector of choice for hackers and cybercriminals to distribute malware and perpetrate identity theft, financial fraud, and corporate espionage. This paper outlines the challenges facing many SMBs and provides solutions for overall security effectiveness and reducing the burden on IT departments. Download this white paper  Why SaaS is Vital to Email and Web Security Why SaaS is Vital to Email and Web Security Download this webcast, free, compilments of Webroot Software Go to the webcast  White Papers Read up on the latest ideas and technologies from companies that sell hardware, software and services. Virtualization Analysis for VMware A Guide to Understanding Messaging Archiving Archiving Compliance with Sunbelt Exchange Archiver View more whitepapers

• Sidebar: Performance Is Where You Find It
• Sidebar: The Players
• High-Speed Databases Rev Corporate Apps
• Sidebar: Driving Demand

• DNS hole prompts synchronized patching effort by IT vendors
• Microsoft plugs nine holes in Windows, DNS, SQL
• Symantec warns of new Word attack
• More top stories 

SAS Business and Analytics Zone No matter where your organization is on the path toward evolving your IT infrastructure, SAS can adapt to your situation to meet your long-term enterprise intelligence needs. We can help you drive intelligence evolution to the next level, while leveraging and extending the value of your existing IT investments. Learn more in the Business and Analytics Zone See All Zones

Intercept Spam & Viruses With MessageLabs MessageLabs is offering a complimentary 30 day trial of its managed Anti-virus and Anti-spam security solutions. MessageLabs guarantees complete protection against all know and unknown email threats. By providing 24 hour support, your business can increase productivity and decrease risk. Register for a complimentary trial and receive a free datasheet. Download this white paper now!

Best Places to Work in IT Where can you earn top dollar, get the best benefits, the latest IT and more? Find out in our 14th annual survey.

Try Fluke Networks' EtherScope Analyzer on your network FREE Quickly solve the wide range of problems you encounter - 10, 100 and Gigabit, twisted pair and optical fiber, LAN or wireless LAN. The EtherScope Analyzer combines the essential tools you need to monitor network traffic and switch interfaces, discover devices, networks, VLANs, access points, mobile clients and more. See the power of this portable network analyzer on your network. Request free trial now *Terms and conditions: Evaluation units are available only for a limited time and will be scheduled on a first-come first-served basis. Not available in all geographies. Limited quantities available; customers requesting evaluation units may be waitlisted for the next available unit. It will be at the discretion of Fluke Networks to accept or decline requests for this free evaluation.