Managing Data at RAM Speed

Two of IT's most consistent mandates are lower costs and more speed. To meet this demand for cheap speed, standard disk-based database management systems need help. Increasingly, this help is coming in the form of memory-centric data management technology.

Conventional DBMSs are designed to get data on and off of disks as safely, quickly and flexibly as possible. Much of their optimization is focused on one key bottleneck -- the length of time it takes to find a random byte of data on disk, which is 1 million times as long as it might take to find the same byte in RAM. But the optimizations and access methods designed to address this bottleneck don't work so well once the data is safely in main memory. Memory-centric data management tools, using access methods that would be ridiculous in a disk-centric setup, can perform vastly better.

If you want to query a used-book database more than 1 million times per minute, that's hard to do in a standard relational DBMS. But Progress Software's ObjectStore gets it done for If you want to recalculate a set of OLAP cubes in real time, don't look to a disk-based system of any kind. But Applix's TM1 can do just that. And if you want to stick DBMS instances on 99 nodes of a telecommunications network, all persisting data to a 100th node, a disk--centric system isn't your best choice -- but Solid Information Technology has a product that works just fine. At their core, each of those products relies on the same technical approach: vast amounts of pointer traversal. Access that random is pretty impractical on disk, where it can take over a millisecond to get from one point to the next. But it works great in 100- to 1,000-MHz RAM.

There's actually a broad variety of memory-centric products, most of them specialized for some particular kind of processing, whether OLAP or OLTP or event stream. They can be hard to find, being positioned as DBMS, quasi-DBMS, business intelligence features or some utterly new kind of middleware. They may come from top-tier software vendors or from the rawest of start-ups. But they are out there.

While memory-centric analytic technology has been around for a while, you may easily have missed it. It's been held back by the addressability limits of 32-bit processors and even more by the scalability limits on most parallel hardware architectures. But that was before massively parallel (a.k.a. blade/grid) architectures made it practical to link huge numbers of CPUs together. You want 100GB of RAM on your server? No problem. One terabyte is achievable too, if you can afford that much silicon. And it's just going to get easier, as the amount of RAM per board keeps doubling, based on the increasing performance of 64-bit chips.

Perhaps the simplest memory-centric approach to analytic data management is used in SAP's superfast BI Accelerator (nee HPA, nee Euclid). Basically, it takes an InfoCube (a prebuilt SAP star-schema warehouse), sucks it into RAM with suitable sparsity compression and executes all analytic queries as full-table scans. Competing with that is tough when you're bound by disk access speeds. The most unique approach is probably that of Applix's TM1, a memory-centric MOLAP tool that allows fully flexible evaluation rules without on-disk precalculation (and hence without the resulting data explosion). Thus, it simultaneously offers most of the benefits of analytic DBMS and most of the benefits of ordinary spreadsheets, and it deserves to be the basis of most planning applications unless and until something better comes along.

Memory-centric OLTP, not as dependent on total RAM available, has probably made more of a market impact to date, although mainly in a few niches. ObjectStore provides object-oriented memory-centric solutions for some of the most demanding OLTP apps in the world -- not just the Amazon bookstore, but some airline reservations and scheduling systems as well. Products from Progress Software, TimesTen and start-up StreamBase Systems are used for highly real-time financial trading systems. TimesTen and Solid Information Technology support real-time management of billing and other functions on huge telecommunications networks.

I suspect that the "niche" label is about to come off. Oracle acquired TimesTen, and it probably didn't do so just for a few particular apps. Did you buy a lot of Real Application Clusters recently? Well, the sequel is a generic caching-performance story, and it's coming soon from an Oracle salesman near you.

For backup detail to this column, please see the DBMS2 blog at

Curt A. Monash is a consultant in Acton, Mass. You can reach him at

Copyright © 2005 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon