The Future of the Search Engine

Computerworld - In a recent interview with Computerworld, Hadley Reynolds, research director at Delphi Group in Boston, talked about what's wrong with today's search engines and how that will change.

Q: How could search engines be improved?

A: When you go the Web and use a search box, you almost always have in mind receiving some sort of answer, or you wouldn't be there. There's a disconnect between wanting an answer and getting instead a list of 256,000 documents. It's ergonomically inappropriate. [If we] asked, "What is the gross national product of the U.S.?" [and] the system came back with an actual number instead of 256,000 documents, we'd all feel that our Web investigative life had been revolutionized. And that is where research is headed.

Q: How serious is this problem?

A: The business result is [that] people waste a lot of time trying to cope with this list of documents. We worked with a Global 10 telecommunications company that did an internal study of their intranet and determined that between 20% and 30% of time employees spent on the intranet was time spent looking for things.

Q: Why can't search engines do a better job today?

A: The basic problem with electronic information is that the person who stores the information in the first place typically doesn't know what the person asking the questions will ask. If we had an exact match between the document-posting and the kinds of questions people were going to ask, then this would not be a problem. It's a fundamental knowledge management issue.

Q: But aren't there document management systems that allow you to tag information for later retrieval?

A: That has to be applied by human beings, but most companies can't afford the investment to tag all their documents and put metadata on them. Anyone who's gone to save a word processor document has run into a little window that allows you to fill in all these fields to label the document with author, date, subject and so on. But as busy professionals, none of us want to take the time to do that. So it's the kiss of death for all attempts to try to label and provide richness to the documents in a manual way. Even if people were willing to do that manually, they can't anticipate what the questions are going to be.

Q: So, what's the solution?

A: One of the things technology can do is what Verity is developing. What we have thought of in the past as retrieval technology can also be used to essentially parse the documents going in. It's to try to figure out, What are the key points in the document, and how do the topics in this particular document that's just being contributed relate to other things I know about other electronic documents within the organization? The analytic software can begin to establish strong, medium or weak relationships between this document and some other documents.