Managing Data at RAM Speed

Curt Monash   Today’s Top Stories    or  Other Data Mining Stories  

Driving BI Value with Data Integration: Overcoming the Thorniest Challenges Survey: Business Intelligence Maturity and the Quest for Better Performance BI @ The Speed of Business Sign up to receive Security Resource Alerts

November 14, 2005 (Computerworld) -- Two of IT's most consistent mandates are lower costs and more speed. To meet this demand for cheap speed, standard disk-based database management systems need help. Increasingly, this help is coming in the form of memory-centric data management technology.
Conventional DBMSs are designed to get data on and off of disks as safely, quickly and flexibly as possible. Much of their optimization is focused on one key bottleneck -- the length of time it takes to find a random byte of data on disk, which is 1 million times as long as it might take to find the same byte in RAM. But the optimizations and access methods designed to address this bottleneck don't work so well once the data is safely in main memory. Memory-centric data management tools, using access methods that would be ridiculous in a disk-centric setup, can perform vastly better.
If you want to query a used-book database more than 1 million times per minute, that's hard to do in a standard relational DBMS. But Progress Software's ObjectStore gets it done for Amazon.com. If you want to recalculate a set of OLAP cubes in real time, don't look to a disk-based system of any kind. But Applix's TM1 can do just that. And if you want to stick DBMS instances on 99 nodes of a telecommunications network, all persisting data to a 100th node, a disk--centric system isn't your best choice -- but Solid Information Technology has a product that works just fine. At their core, each of those products relies on the same technical approach: vast amounts of pointer traversal. Access that random is pretty impractical on disk, where it can take over a millisecond to get from one point to the next. But it works great in 100- to 1,000-MHz RAM.
There's actually a broad variety of memory-centric products, most of them specialized for some particular kind of processing, whether OLAP or OLTP or event stream. They can be hard to find, being positioned as DBMS, quasi-DBMS, business intelligence features or some utterly new kind of middleware. They may come from top-tier software vendors or from the rawest of start-ups. But they are out there.
While memory-centric analytic technology has been around for a while, you may easily have missed it. It's been held back by the addressability limits of 32-bit processors and even more by the scalability limits on most parallel hardware architectures. But that was before massively parallel (a.k.a.

Continued...
1 | 2 | NEXT  

Print this Story

Send Us Feedback

E-mail this Story

Digg this Story

Slashdot this Story

"When white pages searches come up empty, it's often easier to let your fingers do the walking. It shouldn't be..." Read more... "Enterprise search continues to lag behind commerical search because companies lack a "findability" strategy, says one researcher...." Read more... Read more Business Intelligence posts or See all Blogs

DNS hole prompts synchronized patching effort by IT vendors Microsoft plugs nine holes in Windows, DNS, SQL Symantec warns of new Word attack More top stories...

Microsoft sets XP SP3 automatic download for Thursday Don't give Google a free pass on data collection, privacy advocates say after YouTube ruling XP SP3 to reach most users 'shortly,' says Microsoft

This old laptop: Revitalizing an aging notebook on the cheap All it takes is a couple hours and about $125 to breathe new life into an old laptop. Here's how. Bill Gates moves on Is Microsoft's Golden Age over? What are Gates' most memorable quotes? Find out in Computerworld's complete coverage of the end of the Bill Gates era at Microsoft. Five things you should never tell your boss There are some things your CIO definitely doesn't want to hear. Also don't miss the flipside, Five things you should always tell your boss. Hands-On: Firefox 3 fixes what's broke and keeps what's right With its latest version, Mozilla's browser continues to raise the bar for what Web browsers should be. Windows Vista A to Z Reviews, analyses, how-tos, visual tours, hot issues and predictions about Microsoft's new OS. More Continuing Coverage After 9/11: Homeland SecurityAsk a Premier 100 IT LeaderBlackBerry Legal BattleData Security BreachesElectronic VotingFirefoxH-1B VisasHP's Boardroom ScandalHandheld DevicesHurricane Katrina AftermathMicrosoft Legal IssuesMicrosoft's VistaOpen SourceRFID TechnologySCO's Linux FightSarbanes-Oxley ActSpamVoice Over IPWi-FiWindows Meta File VulnerabilityWindows XP Service Pack 3 IT Careers 2010 Four years from now, the IT field will be a vastly different place. Will you be ready? More Special Reports Premier 100 IT Leaders 2007 Best in Class2006 Computerworld Horizon Awards2006 Premier 100 IT Leaders2006 Salary Survey2007 Best Places to Work in IT2007 IT Forecast2007 Premier 100 IT Leaders40 Years of ComputerworldASPs, Take TwoBest Places to Work in IT 2003Best Places to Work in IT 2004Best Places to Work in IT 2005Best Places to Work in IT 2006Business Intelligence Home RunsBusiness Intelligence: Smarter BIBusiness Intelligence: The Future of BICRM Goes VerticalCRM: Sober CRMCareer Planning Guide 2005Careers: IT Profession 2010Computerworld Horizon Awards 2005Data Management: Mining for GemsData Management: Taming Data ChaosDevelopment: Hard-Workin' Web SitesDevelopment: New Tools New ChoicesDevelopment: The New World of Application DevelopmentDevelopment: The Web Services TsunamiDevelopment: Web Services HurdlesDisaster Recovery: Preparing for the WorstE-commerce Grows UpE-mail/Groupware: Big DecisionsFaces of Mobile ITForecast 2006Global MobileHardware: Multiple Cores, Multiple ChallengesHardware: On-Demand Un-HypedHardware: The Shape of Things to ComeHorizon Awards: 10 Cool Cutting Edge TechnologiesIT Management: Guide to Managing VendorsIT Management: Navigating Global ITIT Management: Recovery AheadIT Management: The Future of ITIT Management: The Resourceful Project ManagerInnovative Technology Awards 2004Management: Reinventing ITMicrosoft in TransitionMobile/Wireless Leaders and LaggardsMobile/Wireless: The Untethered WorkerMobile/Wireless: Tiny Gadgets, Huge CostsNetwork Management: Taking ControlNetworking: Turbulence Ahead!Networking: VoIP Goes MainstreamOperating Systems: In the Slow LaneOperating Systems: Linux Goes GlobalOperating Systems: Should You Nix Unix?Outsourcing: Offshore Buyer's GuideOutsourcing: Outsourcing DangersPremier 100 IT Leaders 2004 Best in ClassPremier 100 IT Leaders 2005 Best in ClassROI: Do the Math!Salary Survey 2003Salary Survey 2004Salary Survey 2005Security Action PlanSecurity: Compliance HeadachesSecurity: Proactive SecuritySecurity: Risk and RewardSecurity: Tips From Security ProsSouped-up SecurityStorage: Battling ComplexityStorage: Cheap & Secure Data StoresStorage: Stretching Your Storage DollarsStorage: The Lean Storage MachineStorage: The New Rules of StorageStorage: The Ultimate Backup GuideSupply Chain: Missing LinksSupply Chain: RFID Reality CheckThe Business of SecurityWeb 2.0 SecurityWireless: Wireless At Work All Zones Application Performance Zone Business Continuity Zone Data Center Management Zone Enterprise-Class Security Zone The File Data Management Zone Grid Computing on Windows Zone Security Management Zone ITIL Best Practices Zone The SAS Zone Storage Virtualization Zone Business Intelligence and Analytics Zone See your link here

Speeding the time to intelligence Get this Computerworld report free for a limited time, compliments of SAS. Time To Intelligence -- a concept defining how long it takes to get accurate and timely information into the hands of workers who need it most. Do it slower than your competitors and your company is toast. Do it faster, you scorch them. Business Intelligence is the key to optimizing Time To Intelligence, and success there is a combination of people, policies, and technology. Download this executive briefing  Web Security SaaS: The Next Generation of Web Security Download this whitepaper, free for a limited time, compliments of Webroot Software. (Source: Webroot Software) The Web is the new threat vector of choice for hackers and cybercriminals to distribute malware and perpetrate identity theft, financial fraud, and corporate espionage. This paper outlines the challenges facing many SMBs and provides solutions for overall security effectiveness and reducing the burden on IT departments. Download this white paper  Long Tail Supplier Collaboration - What's In It For You? Long Tail Supplier Collaboration - What's In It For You? Download this webcast, free, compliments of Sterling Commerce Go to the webcast  White Papers Read up on the latest ideas and technologies from companies that sell hardware, software and services. Virtualization Analysis for VMware A Guide to Understanding Messaging Archiving Archiving Compliance with Sunbelt Exchange Archiver View more whitepapers

• DNS hole prompts synchronized patching effort by IT vendors
• Microsoft plugs nine holes in Windows, DNS, SQL
• Symantec warns of new Word attack
• More top stories 

Understand Messaging Archiving Email storage is growing at an average rate of 35% annually - three out of five decision makers cite the growth of messaging storage as their leading messaging-related problem. This Osterman Research white paper discusses the several reasons to implement a messaging archiving system and provide an overview of Sunbelt Software's offering focused squarely on the archiving space. Download this white paper now!

Intercept Spam & Viruses With MessageLabs MessageLabs is offering a complimentary 30 day trial of its managed Anti-virus and Anti-spam security solutions. MessageLabs guarantees complete protection against all know and unknown email threats. By providing 24 hour support, your business can increase productivity and decrease risk. Register for a complimentary trial and receive a free datasheet. Download this white paper now!