Speed Readers

At investment management firm Bridgewater Associates Inc., access to real-time data is measured in market ticks. Data feeds containing quote and trade activity are expected to stream in at 124,000 messages per second this year, so even subsecond delays in the arrival of data can affect trading decisions and put the Westport, Conn.-based organization at a disadvantage.

Monitoring high volumes of data that have very low latency requirements is beyond the capabilities of transactional databases, which must write each transaction to disk, so financial services firms traditionally build their own custom applications to keep up.

"There is a lot of effort required to build a framework that could perform and deal with lots of data concurrently," says Ed Thieberger, head of training technology at Bridgewater.

Recently, however, Bridgewater and other financial services firms have found an alternative in stream processing tools. Stream processing software goes by a variety of names, including streaming databases and event stream processing. The technology includes an engine that monitors data as it flows into and out of databases and other applications and can easily tap into external data feeds or internal message queues. All the data the engine gathers is held in memory to speed processing.

With data volumes increasing, organizations are running out of options for real-time processing. Financial services firms have little choice but to pursue stream processing because data quantities are starting to outstrip the capabilities of even custom-developed tools.

"At these volumes, traditional techniques won't scale," says Mike Stonebreaker, co-founder and chief technology officer at Lexington, Mass.-based StreamBase Systems Inc. Bridgewater's custom C++ program could handle 18,000 messages per second -- more than the 900 a relational database could support, but far short of the data volumes it faces this year. In contrast, the StreamBase engine handles 140,000 messages per second, Stonebreaker says.

Having gained a following in financial services, the emerging technology is beginning to spread to other industries that need to monitor operational data and interpret and respond to events in real time. Businesses are using it in areas as diverse as compliance management, network monitoring and real-time fraud detection in telecommunications, retail and e-commerce.

Stream processing software is also ideally suited to leverage message-based data flows within a service-oriented architecture (SOA). "If your organization already has MQ or other message-oriented middleware, then this is relatively straightforward," says Charles Nichols, CEO of SeeWhy Software Ltd. in Windsor, England. Users set up rules- or time-based queries that tell the stream processing engine what to look for. The software then monitors one or more data streams and triggers the appropriate response when one of those conditions is detected.

AT A GLANCE Stream Processing What it does: Stream processing software monitors and analyzes operational data flows in real time to detect predetermined conditions or events. Pros: Can handle very large volumes of streaming data with very low latency requirements. Faster than custom-built applications. Integrates well with service-oriented architectures. Cons: Most applications are still in financial services. Products are relatively new and still maturing. Analysts expect a vendor shakeup in the next two years. Capabilities may eventually be embedded within existing tools such as databases and business activity monitoring. Best Use: Applications that require fast detection of and reaction to business events embedded in large volumes of data, such as fraud detection, network monitoring or transaction monitoring.