Q&A: David Luckham on the future of stream processing

Computerworld -

David Luckham, whose work in complex event processing helped spur the development of today's stream processing tools, speculates on the future of the technology in more from his interview with Computerworld's Robert L. Mitchell.

What other research influenced today's stream processing tools? There was a parallel stream of research going on that started in the mid-1990s when the database people realized that databases were too slow to do real-time data analysis. They started researching the idea of running continuous queries on streams of incoming data. That led to the event streams processing world.

How will the role of stream processing tools evolve in the future? The problems that these event-processing tools are applied to will become more sophisticated. They will cross multiple enterprises. There's little credit card fraud detection that's applied across different credit card [companies]. As things get better, people will understand that these tools can be applied to more challenging problems. A second area is the tremendous increase in the amount of automated data that's becoming available. We have these problems going on now with telephone surveillance. How is the government going to analyze the data when they get it? The third area is autonomic systems. We now have huge warehouses of servers [at] companies, and it's very expensive to maintain these things and diagnose them with human labor. There's a big effort going on in autonomic computing to automate the self-diagnosis of large systems of machines. I think the autonomic computing people are going to realize that complex event processing is part of the answer for that. There are also some very imaginative applications going on in medical epidemiology. You can predict flu outbreaks more readily by looking at over-the-counter medication sales. The world is becoming more automated.

What will be the challenges going forward? I have some technical worries for the future. Most CEP systems work by applying sets of rules that embody the event processing patterns they want to recognize. One of my technical worries is how to manage large rule sets. With any complex set of rules, you have a hard time reading and understanding that rule set. We need higher-level languages, more understandable languages. Another technical worry is the actual validation of the event processing tools. That is, there are a lot of claims out there as to what particular tools do, but nobody has the time to actually validate that the tool does exactly what they say it does. That's going to become more of a problem in the future. I don't think these challenges are going to be on people's minds for a while yet, let's say three to five years. Right now what people are doing is relatively simple.

Read more about business intelligence in Computerworld's Business Intelligence Knowledge Center.