The Search Is On

Google's Universal Search for Business Driving BI Value with Data Integration: Overcoming the Thorniest Challenges Gain a faster, cheaper way to manage workload BI Trends: The Next Generation of Performance Management Massively Parallel Processing Database for Business Intelligence Surviving and Thriving in Tough Times The Business Value of Tape Storage Computerworld Technology Briefing: An open-source path to optimal virtualization Learn-Fast Guide: Get Up to Speed on Green IT Sign up to receive Business Intelligence Resource Alerts

April 15, 2002 (Computerworld) -- Researchers are inventing better ways to find and make sense of information. Efforts to improve data mining and searching are being driven by the deluge of information in this increasingly networked world and by companies' need to respond ever faster to changes. And, sadly, the field has gotten a big boost from the terrorist attacks on the U.S. last fall.

Computerworld looked at some of this research and found companies perfecting techniques for machine learning, real-time analysis of data flows, distributed data mining and the discovery of "nonobvious" relationships.

Known Associates

Systems Research & Development (SRD) developed its Non-Obvious Relationship Awareness (NORA) technology to help casinos identify cheaters by correlating information from multiple sources about relationships and earlier transactions.

Las Vegas-based SRD, which received funding from the CIA, is now developing several NORA plug-ins to reach further into the world of criminals and terrorists. Last month, the company unveiled a "degrees of separation" capability that finds deeper connections among people.

"It will tell you that the Drug Enforcement Agency's agent's college roommate's ex-wife's current husband is the drug lord," says Jeff Jonas, chief technology officer at SRD. NORA can bridge up to 30 such links, he says.

Prabhakar Raghavan, CTO at Verity Inc.

The new NORA module uses streaming technology that scans data and extracts information in real time as it flows by. That would allow it to, for example, instantly discover that a man at an airline ticket counter shares a phone number with a known terrorist and then issue an alert before he can board his flight. Jonas calls it "perpetual analytics," to distinguish it from periodic queries against an occasionally updated database.

SRD is also developing the concept of "cascading" NORA data warehouses for really big problems.

For example, Jonas says, each airline might have a copy of NORA processing its passenger data and sending the summarized results to a midtier NORA system at the Federal Aviation Administration. Car rental agencies might send their NORA results to a rental car association. And the U.S. Immigration and Naturalization Service could collect data from ports of entry.

All three midtier NORA systems would then send transactions to the top-tier system at the Office of Homeland Security in Washington. They would communicate with one another in a "zero administration" arrangement in which rules and filters would determine whether a piece of information got passed up or down the chain, Jonas says.

Outbreak Detection

If a bioterrorist attack occurred, it would be critical for health and law enforcement officials to find out quickly, even before people were diagnosed with a specific disease.

The key to doing that lies in distributed data mining, says Tom Mitchell, a computer science professor at Carnegie Mellon University in Pittsburgh. Carnegie Mellon and the University of Pittsburgh recently fielded the Real-Time Outbreak Disease and Surveillance (RODS) system, which takes data feeds from the emergency rooms of 17 local hospitals, loads it into a database and applies statistical techniques to predict the occurrence of diseases such as anthrax and smallpox. The universities also used RODS during this year's Olympic Winter Games in Salt Lake City.

The system considers 30 to 100 variables every few minutes over a large geographic area, says project co-director Andrew Moore, who is also director of the Biomedical Security Institute in Pittsburgh. "We are looking at between 1 million and 1 trillion possible strange things going on—possible indicators of various kinds of disease," says Moore. "If we are not careful, we'll use a year's worth of supercomputer time every day."

Project members are working on better algorithms and have increased processing efficiency by a factor of 10,000 in the past year, Moore says, but more improvements are needed. The system may be expanded to look at pharmacy cash-register data, school attendance records, animal sickness data, phone call records and vehicular traffic patterns, all of which may hold real-time clues about changes in a population's health.

But gathering all that information raises privacy and confidentiality concerns. At present, the hospital data comes into a central repository where it's carefully scrubbed of information that could be used to identify anyone. Carnegie Mellon researchers are looking at ways to push that scrubbing activity out to the data source.

"How can you design a data mining system that instead of running on a central repository, allows each hospital, store and so on to keep their own records and not reveal the identities?" asks Mitchell, director of Carnegie Mellon's Center for Automated Learning and Discovery. "What you want to do is give them some software that they can use to put their own privacy restrictions on."

That concept could be applied in many domains, Mitchell says. For example, intelligence agencies could use it to allow information-sharing across departments while protecting the sources of the information, he says.

Upside Down

"Instead of archiving data and running search queries through it, we archive search queries and run data through it," says Val Jerdes, vice president for business development at Streamlogic Inc. "It's a search engine on its head."

The advantage of an inverted search engine, he claims, is that it's 6,000 times more efficient than the conventional approach. It can handle huge volumes of data that would be expensive or impossible to process using the standard method of loading data into an archive, indexing it and then retroactively querying it.

Los Altos Hills, Calif.-based Streamlogic's feed-monitoring technology "strains" the information through query rules in real time, eliminating the archival requirement entirely. A demonstration at www.streamlogic.com runs all the postings to some 50,000 Usenet news groups—10 postings per second, or 2GB per day—through a database of user-specified topics and instantly sends an alert every time one of those topics appears in a post. It also turns unstructured information into data that can put into a relational database for further analysis.

A feed-processing engine plucks out information based on user-specified topics or keywords. A feed analysis engine uses statistical techniques to analyze, categorize and summarize information for identifying trends, advertisement-targeting and other applications. The engine improves with use as it learns the most relevant words and phrases, says Streamlogic.

The future of these concepts lies in applications that others will develop with Streamlogic's tool kit, which includes a collection of "metaware" and a language similar to SQL. For example, it could be used to speed and unify the flow of data throughout an enterprise, Jerdes says.

"So when a customer's order comes in, instead of moving from one database to another in functional silos, we are able to dissolve the walls so that the order gets through to manufacturing, customer relations, financial and sales systems," he says. "And all that could happen instantaneously."

What's the Answer?

When someone types the query "What is the population of the world?" into an Internet search engine, he most likely wants the numerical answer—6.2 billion—not pointers to hundreds of documents containing the words population and world. Unfortunately, today's search engines produce more document hits than answers.

But Verity Inc. is developing software that will be a lot smarter, says Prabhakar Raghavan, chief technology officer at the Sunnyvale, Calif.-based company. The approach involves putting human learning, or rules, into the software and enabling that software to teach itself in a process called machine learning.

Suppose you want to build a recruiting system that automatically extracts information from the scanned resumes of job applicants. Raghavan says specific rules could be written into the software to indicate that employment information is commonly found after the words employment, work history and experience, enumerating every possibility. Or one could train the system by giving it an initial batch of resumes annotated as to what information appears in each area of the resume.

"After it's looked at 50 or 100 resumes, it's started to figure out that all those phrases are variants on the same theme," Raghavan says.

Verity is using a relatively new technique called logistic regression classification to enable such machine learning. The best systems for information extraction use both hard-coded rules and machine learning, Raghavan says.

Verity is also working on software that can synthesize and summarize information. "That's difficult with an unstructured query like 'What percentage of Republicans in Santa Clara County are in favor of bombing Iraq?' " Raghavan says. It requires joining data from several sources and resolving conflicting information, and the technology to do that is still primitive, he says.

Raghavan says Verity software will be able to handle a query like the one about world population in about two years, but the capability to answer questions like the one about attacking Iraq will take considerably longer to develop.

Non-Obvious Relationship Awareness (NORA) Systems Research & Development’s NORA technology can take information from disparate sources about people and their activities and find obscure, nonobvious relationships. For example, it might discover that an applicant for a job at a casino shares a telephone number with a known criminal and issue an alert to the hiring manager. Source: Systems Research & Development, Las Vegas

The Search Is On: The Future of the Search Engine The Search Is On

Print this Story

Send Us Feedback

E-mail this Story

Digg this Story

Slashdot this Story

Mozilla updates Firefox 3.1 with Alpha 2 build Microsoft explains Seinfeld-Windows TV ad: just a 'teaser' Mozilla: Firefox is faster than Chrome More top stories...

iPhone 3G owner sues Apple, AT&T over dropped calls, app crashes At 10, Google reiterates commitment to CIOs Analysts: Google spreading itself too thin

Microsoft warns of IE8 lock-in with XP SP3 Users of Windows XP SP3 who try out IE8 Beta 2 won't be able to uninstall either one under certain circumstances. Opinion: Why Google has lost its mojo -- and why you should care Google has gone from innovative upstart to fat-and-happy industry leader in what seems like record time. Preston Gralla explains. Review: Internet Explorer 8 Beta 2 offers some nifty new features Microsoft's latest beta of IE8 includes better tab management, new services such as Web Slices and Accelerators, and the new 'porn mode.' IT Schools to Watch 2008 These leading-edge graduate schools are moving at the pace of the IT workplace, delivering coursework that's relevant to today's IT professionals. Windows Vista A to Z Reviews, analyses, how-tos, visual tours, hot issues and predictions about Microsoft's new OS. More Continuing Coverage After 9/11: Homeland Security Apple's iPhone Ask a Premier 100 IT Leader Bill Gates steps down BlackBerry Legal Battle Data Security Breaches Electronic Voting Firefox H-1B Visas HP's Boardroom Scandal Handheld Devices Hurricane Katrina Aftermath Microsoft Legal Issues Microsoft's Vista Open Source RFID Technology SCO's Linux Fight Sarbanes-Oxley Act Spam Voice Over IP Wi-Fi Windows Meta File Vulnerability Windows XP Service Pack 3 IT Careers 2010 Four years from now, the IT field will be a vastly different place. Will you be ready? More Special Reports Premier 100 IT Leaders 2007 Best in Class 2006 Computerworld Horizon Awards 2006 Premier 100 IT Leaders 2006 Salary Survey 2007 Best Places to Work in IT 2007 IT Forecast 2007 Premier 100 IT Leaders 40 Years of Computerworld ASPs, Take Two Best Places to Work in IT 2003 Best Places to Work in IT 2004 Best Places to Work in IT 2005 Best Places to Work in IT 2006 Business Intelligence Home Runs Business Intelligence: Smarter BI Business Intelligence: The Future of BI CRM Goes Vertical CRM: Sober CRM Career Planning Guide 2005 Careers: IT Profession 2010 Computerworld Horizon Awards 2005 Data Management: Mining for Gems Data Management: Taming Data Chaos Development: Hard-Workin' Web Sites Development: New Tools New Choices Development: The New World of Application Development Development: The Web Services Tsunami Development: Web Services Hurdles Disaster Recovery: Preparing for the Worst E-commerce Grows Up E-mail/Groupware: Big Decisions Faces of Mobile IT Forecast 2006 Global Mobile Hardware: Multiple Cores, Multiple Challenges Hardware: On-Demand Un-Hyped Hardware: The Shape of Things to Come Horizon Awards: 10 Cool Cutting Edge Technologies IT Management: Guide to Managing Vendors IT Management: Navigating Global IT IT Management: Recovery Ahead IT Management: The Future of IT IT Management: The Resourceful Project Manager Innovative Technology Awards 2004 Management: Reinventing IT Microsoft in Transition Mobile/Wireless Leaders and Laggards Mobile/Wireless: The Untethered Worker Mobile/Wireless: Tiny Gadgets, Huge Costs Network Management: Taking Control Networking: Turbulence Ahead! Networking: VoIP Goes Mainstream Operating Systems: In the Slow Lane Operating Systems: Linux Goes Global Operating Systems: Should You Nix Unix? Outsourcing: Offshore Buyer's Guide Outsourcing: Outsourcing Dangers Premier 100 IT Leaders 2004 Best in Class Premier 100 IT Leaders 2005 Best in Class ROI: Do the Math! Salary Survey 2003 Salary Survey 2004 Salary Survey 2005 Security Action Plan Security: Compliance Headaches Security: Proactive Security Security: Risk and Reward Security: Tips From Security Pros Souped-up Security Storage: Battling Complexity Storage: Cheap & Secure Data Stores Storage: Stretching Your Storage Dollars Storage: The Lean Storage Machine Storage: The New Rules of Storage Storage: The Ultimate Backup Guide Supply Chain: Missing Links Supply Chain: RFID Reality Check The Business of Security Web 2.0 Security Wireless: Wireless At Work All Zones Application Performance Zone Business Continuity Zone The File Data Management Zone Security Management Zone ITIL Best Practices Zone The SAS Zone Business Intelligence and Analytics Zone Windows Protection Zone Identity & Security Management Zone See your link here

Speeding the time to intelligence Get this Computerworld report free for a limited time, compliments of SAS. Time To Intelligence -- a concept defining how long it takes to get accurate and timely information into the hands of workers who need it most. Do it slower than your competitors and your company is toast. Do it faster, you scorch them. Business Intelligence is the key to optimizing Time To Intelligence, and success there is a combination of people, policies, and technology. Download this executive briefing  From Laggard to Leader: Transforming the Data Center From Laggard to Leader: Transforming the Data Center Register for this complimentary live webcast today! Go to the webcast  Rapid application development, rapid results Download this special report now! (Source: Intersystems) All too many businesses suffer from IT infrastructures that are a hodge-podge of disconnected databases and applications. What's needed is the ability rapidly develop connected applications under a unified service-oriented architecture. InterSystems Ensemble integration environment and Cache database are effective tools in answering this need, delivering a rapid ROI. Download this white paper  White Papers Read up on the latest ideas and technologies from companies that sell hardware, software and services. Death to PST: Hidden Cost of Email Mismanagement Extend, Replace, or Convert; which is the best way forward for COBOL Applications? The Trend from Unix to Linux in SAP Data Centers View more whitepapers

• Mozilla updates Firefox 3.1 with Alpha 2 build
• iPhone 3G owner sues Apple, AT&T over dropped calls, app crashes
• Microsoft explains Seinfeld-Windows TV ad: just a 'teaser'
• More top stories 

SAS Business and Analytics Zone No matter where your organization is on the path toward evolving your IT infrastructure, SAS can adapt to your situation to meet your long-term enterprise intelligence needs. We can help you drive intelligence evolution to the next level, while leveraging and extending the value of your existing IT investments. Learn more in the Business and Analytics Zone See All Zones

Intercept Spam & Viruses With MessageLabs MessageLabs is offering a complimentary 30 day trial of its managed Anti-virus and Anti-spam security solutions. MessageLabs guarantees complete protection against all know and unknown email threats. By providing 24 hour support, your business can increase productivity and decrease risk. Register for a complimentary trial and receive a free datasheet. Download this white paper now!

XenServer FREE trial Citrix XenServer is the simplest and most effective way to virtualize and provision servers. XenServer combines comprehensive server virtualization capabilities with unparalleled scalability, performance, economics, and ease-of-use. Based on the open source Xen hypervisor, XenServer delivers fast performance, easy management, and advanced features such as live migration. Request free trial now

Since You Asked A weekly storage column from storage analyst, Steve Duplessie of the Enterprise Strategy Group Click here to read the latest column by Steve Duplessie