Speed Readers

Stream processing tools monitor and analyze high volumes of data to spot trends and react to events in real time.

1 2 3 4 Page 2
Page 2 of 4

To keep latency low, stream processing systems place data that must be retained in memory and discard everything else. Nothing is stored on disk.

"Streaming databases say, 'Let's not try to store everything. Let's just watch everything as it flies by and keep running totals,'" such as the total number of transactions per second, says Eric Rogge, an analyst at Ventana Research Inc. in San Mateo, Calif.

At Bridgewater, Thieberger uses StreamBase's streaming technology to watch for delays in data feeds coming in from providers of market data. If one feed falls behind, StreamBase immediately issues an alert and splices in the missing data from another source. "The tool is very well suited to represent all of the rules we want to implement that lead to decisions about how we are trading," Thieberger says.

He measures the success of stream processing both in reduced development costs and faster time to market. "We haven't had to build a framework that does what StreamBase does," he says. In addition, once StreamBase is pointed at the data streams to be measured, business analysts can construct queries using a drag-and-drop user interface rather than rely on programmers, Thieberger says.

Stream processing also matches up well with another emerging technology: radio frequency identification. "Streaming is the only technology that can handle large volumes of RFID data that need to be analyzed on the fly," says Diaz Nesamoney, founder and CEO of Celequest Corp., a business intelligence tool vendor in Redwood City, Calif.

The challenge with RFID tags is that they broadcast the same data continuously, says Jan Vink, IT director at Boekhandels Group Nederland BV, a Hauten, Netherlands-based chain of 42 bookstores. When a pilot bookstore recently began checking in more than 1,200 books per day using RFID tags and a tag reader "tunnel," Vink used Progress Software Corp.'s Apama tool to filter out the repetitive messages and ensure that each book was received in the system just once. The 45 to 50 boxes a day the store receives now take a total of 125 seconds for incoming processing rather than the 125 minutes required before, says Vink.

The toughest part of the project wasn't the technology, however. "It was new for us to work with event streaming," he says. "We were used to batch processing."

1 2 3 4 Page 2
Page 2 of 4
7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon