David Luckham Explains Stream Processing's Origins

Computerworld - David Luckham is research professor emeritus of electrical engineering at Stanford University. His work in complex event processing helped spur the development of today's stream processing tools. He consults on complex event processing, is the author of the book The Power of Events (Addison-Wesley Professional, 2002) and hosts the Complex Event Processing Web site at www.complexevents.com. He spoke with Computerworld's Robert L. Mitchell about how the concepts behind CEP have evolved into today's stream processing products.

What problem was CEP designed to solve? CEP was developed from 1989 to 1995 to analyze event-driven simulations of distributed systems containing both hardware and software. Since then, it's been used by researchers to analyze various systems, including industry standards in manufacturing and telecommunications, for example. In 1995, I got the idea that you could apply the CEP analysis system to events that were created in any kind of events system. So we decoupled it from the simulator and started applying it to the commercial middleware that had grown up by that time.

How is event stream processing different from CEP? Event stream processing is focused more on high-speed querying of data in streams of events and applying mathematical algorithms to the event data. The first commercial applications were to stock-market feeds in financial systems and algorithmic trading. CEP is focused more on extracting information from clouds of events created in enterprise IT and business systems. CEP detects patterns of events, abstracts and simplifies information in the patterns, and supports making management decisions.

What do stream processing products not do well? Very few of the event-processing products of the moment use any relationships between events other than timing. Why is one process putting a set of products on sale and another retail process applying [a second] promotional discount lowering the price of that very same set of products? They're not coordinating properly. I want to understand why my processes are not communicating when they should. Or I've got a trading process or set of trading processes, and they keep on timing out. Why is that?

When you get into these problems of business process design, you have to understand how events are related. You have to look at more than just the data in the stream of the event. You have to look at what caused different events to happen.

Read more about Applications in Computerworld's Applications Topic Center.