Search Engines -- The Future

Most information junkies would be hard-pressed to name anything that has transformed their professional lives as much as Internet search engines have. The miraculous devices can take your hot topic of the day, scan millions of Web pages and in seconds bring back product announcements, research papers, the names of experts and more—things that would be difficult or impossible to find otherwise.
But as powerful as they are, search engines have huge weaknesses. For example, a recent Google search on the word Linux took just 0.4 seconds, but it had 95 million hits. Too bad if the one you need is No. 10,000 on the list.
But researchers are poised to revolutionize search technology over the next few years. The most common thrust is to personalize search engines so that they know, for example, that if you're an IT professional and you search for mouse, you're more likely to want information about PC devices than about animals.
Adele Howe, a computer science professor at Colorado State University in Fort Collins, and Gabriel Somlo, a CSU graduate student, have built a proof of concept called QueryTracker, a software agent that sits between a user and a conventional search engine and looks for information of recurring interest, such as the latest news about a user's chronic illness. QueryTracker submits a user's query to the search engine once a day and returns results from new Web pages and pages that have changed since the previous search.
The magic in QueryTracker comes from its automatic generation of an additional daily query—which Howe says is often superior to the user's original query—based on what it learns about the user's interests and priorities over time. It filters the results of both queries for relevance and sends them to the user.
QueryTracker's ability to generate its own searches can compensate for the poorly formed queries that many users write, Howe says. "Even people knowledgeable about the Web are often either lazy or they are just not informed about how to write good queries," she says. The most common mistake: queries that are too short, like the one-word Linux search.

Jeannette Jenssen, a mathematics professor at Dalhousie University in Halifax, Nova Scotia, is taking search personalization techniques a step further, to the "crawlers" that index Web content before it can be searched. She says the popular search engines have three drawbacks: They are increasingly charging corporate users for their services, they skew results in favor of advertisers, and they often retrieve huge amounts of irrelevant information. But Jenssen's "focused crawler" indexes only pages related