Managing Data at RAM Speed

Computerworld - Two of IT's most consistent mandates are lower costs and more speed. To meet this demand for cheap speed, standard disk-based database management systems need help. Increasingly, this help is coming in the form of memory-centric data management technology.
Conventional DBMSs are designed to get data on and off of disks as safely, quickly and flexibly as possible. Much of their optimization is focused on one key bottleneck -- the length of time it takes to find a random byte of data on disk, which is 1 million times as long as it might take to find the same byte in RAM. But the optimizations and access methods designed to address this bottleneck don't work so well once the data is safely in main memory. Memory-centric data management tools, using access methods that would be ridiculous in a disk-centric setup, can perform vastly better.
If you want to query a used-book database more than 1 million times per minute, that's hard to do in a standard relational DBMS. But Progress Software's ObjectStore gets it done for Amazon.com. If you want to recalculate a set of OLAP cubes in real time, don't look to a disk-based system of any kind. But Applix's TM1 can do just that. And if you want to stick DBMS instances on 99 nodes of a telecommunications network, all persisting data to a 100th node, a disk--centric system isn't your best choice -- but Solid Information Technology has a product that works just fine. At their core, each of those products relies on the same technical approach: vast amounts of pointer traversal. Access that random is pretty impractical on disk, where it can take over a millisecond to get from one point to the next. But it works great in 100- to 1,000-MHz RAM.
There's actually a broad variety of memory-centric products, most of them specialized for some particular kind of processing, whether OLAP or OLTP or event stream. They can be hard to find, being positioned as DBMS, quasi-DBMS, business intelligence features or some utterly new kind of middleware. They may come from top-tier software vendors or from the rawest of start-ups. But they are out there.
While memory-centric analytic technology has been around for a while, you may easily have missed it. It's been held back by the addressability limits of 32-bit processors and even more by the scalability limits on most parallel hardware architectures. But that was before massively parallel (a.k.a. blade/grid) architectures made it practical to link huge numbers of CPUs together. You