Taming Text

Companies are increasingly using text mining tools to harness the information in their unstructured data.

By Drew Robb

Computerworld |

Unstructured data, most of it in the form of text files, typically accounts for 85% of an organization's knowledge stores, but it's not always easy to find, access, analyze or use.

"We are drowning in information but are starving for knowledge," says Mani Shabrang, technical leader in research and development at Dow Chemical Co.'s business intelligence (BI) center in Midland, Mich. "Information is only useful when it can be located and synthesized into knowledge."

But a new generation of text mining tools allows companies to extract key elements from large unstructured data sets, discover relationships and summarize the information. Many organizations are deploying or considering such software to deal with their mountains of text, despite the need for specialized skills to make implementations work.

For example, since 2000 Dow's research staff has been using ClearResearch software from ClearForest Corp. in New York to extract data from a century's worth of chemical patent abstracts, published research papers and the company's own files.

"By managing the information better and eliminating the irrelevant, we've been able to reduce the time it takes for [researchers] to find what they need to read," says Shabrang.

Text mining tools take a variety of approaches. ClearResearch uses a proprietary pattern-matching methodology to search for information, categorize it and graphically show its relationship to other data.

"The software can see, discover and extract concepts, not just words," says Shabrang. "It gives us a pictorial representation of the text in the documents in an easy-to-understand chart."

Adoption Roadblocks

The text mining software available now doesn't yet match the accuracy of data mining tools, but vendors are improving their products' ability to understand context, which is key to making text mining tools effective.

"Understanding linguistics and overcoming its challenges is a horizon that has not been dealt with well," says William McKnight, president of McKnight Associates Inc., a data warehousing consulting firm in Plano, Texas. "Basic text mining is possible, but the performance needs to be improved and the tools don't scale well."

Because of these limitations, text mining tools are still niche products generally restricted to specific parts of an organization. But they are starting to catch on.

"Over the last 12 to 18 months, I have seen a lot of interest in using these tools for regulatory compliance," says Brian Babineau, a research analyst at Enterprise Storage Group Inc. in Milford, Mass. "But once that seems to be under control, people will retrofit these applications for other purposes, like data warehousing and CRM."

While there are software systems that analyze both structured and unstructured data, many companies use traditional BI software on their structured data and then turn to separate tools to analyze text-based data. Electronic Data Systems Corp., for example, has all of its 130,000 employees fill out an online questionnaire about their jobs once a year. Another three times a year, 20,000 employees answer an additional survey.

Some of the survey questions are multiple choice, making it easy for EDS to plug the answers into BI software from SAS Institute Inc. in Cary, N.C., and SPSS Inc. in Chicago, where it's aggregated, dissected and analyzed. Some of the most important feedback, however, comes in the responses to open-ended questions. In the past, those responses were forwarded to the line managers to draw conclusions, since they didn't fit into any easy-to-manage structure.

Three years ago, EDS started looking for a better way to interpret those responses and harness the information they contained.

"There was so much richness in there that we needed to analyze it on a higher level and look for trends across the enterprise," says Greg Talkington, a human resources data analyst at EDS.

EDS began using PolyAnalyst from Megaputer Intelligence Inc. in Bloomington, Ind., which can mine intelligence from structured and unstructured data. PolyAnalyst is based on an implementation of the WordNet semantic dictionary developed by the Cognitive Science Laboratory at Princeton University. Among other functions, PolyAnalyst assigns words to subject categories and provides related words. Talkington uses PolyAnalyst for analyzing the open-ended questions but still uses traditional BI software for the multiple-choice questions and combines the information from the two in consolidated reports.

There are separate tools that specialize in analyzing either databases or text files, but there are also techniques that allow the two to be correlated. Patricia B. Cerrito, a professor of mathematics and a biostatistician at the University of Louisville in Kentucky, mines hospital records to discover ways to improve patient outcomes. She uses SAS Text Miner on text files, such as patient charts. But she also pulls in flat-file snapshots of billing and pharmaceutical databases and analyzes those as text, rather than as database entries.

"Where you have thousands or tens of thousands of categories, standard categorical analysis simply will not work," Cerrito says. "But by treating it as unstructured data, I can then get some very useful information from it."

By examining thousands of patient outcomes with Text Miner, she has found useful information - that prescribing certain medications can prolong hospital stays for patients, for example, and that the blood sugar levels of diabetes patients can be correlated to their risk of infection after cardiac surgery.

The differences in how hospitals record their patient information represent a major barrier to gathering accurate medical data, Cerrito says. Although they dutifully record massive amounts of data, it hasn't been cleaned or validated, which makes it difficult to analyze. That's why Cerrito uses mining software to cleanse and standardize it.

"I think my results are more accurate because I don't make the standard assumption that hospitals enter data uniformly across the country," she says.

Feeding Other Systems

Heidi Collins, global IT director for knowledge management at Air Products and Chemicals Inc. in Allentown Pa., is using SmartDiscovery from Inxight Software Inc. in Sunnyvale, Calif., to organize the company's internal information and make it more readily available. "We have an initiative to transform the organization from silos of business information to a business-process-focused, cross-functional organization," she says.

The company has more than 18,000 employees in 30 countries and more than 600 intranet and extranet sites. Its file servers contain 9TB of unstructured data, not counting e-mail or anything stored on local drives.

Among other things, Air Products is using SmartDiscovery to generate a catalog and index of the data repository so that it can be more easily accessed by Microsoft SharePoint Portal document management software. This catalog and the index are stored separately from the document repository.

Air Products is also using the software for Sarbanes-Oxley compliance, content life-cycle management and e-learning. By correctly categorizing the data, business rules can be applied to a category of documents rather than to individual documents. For example, if a document relates to aspects of operations covered by Sarbanes-Oxley, then the appropriate data-retention policies are applied to it.

"I call it the central nervous system for what we are doing with knowledge management," says Collins.

The Skills Gap

Installing a text miner is generally a simple process. Cerrito reports that she just needed to load six CDs to get SAS Text Miner running on her workstation. Shabrang says it took about an hour to set up ClearResearch. The hard part is getting meaningful results from a process that depends on the skill and knowledge of the person using the software. It takes a skilled analyst to properly interrogate text repositories.

In addition to having analytic skills, the user has to be familiar enough with the data set to understand what the results mean. For example, Cerrito is a mathematician working on medical data. She may find that a particular drug frequently comes up in certain settings, but she then needs to ask a pharmacist what that means medically. But the combination of her skills and those she consults with is saving lives.

"We are getting an increasing understanding of what things are possible with text mining," says Alexander Linden, an analyst at Gartner Inc. "But there is a huge skills problem in this area, which is why it hasn't gotten much traction so far."

This restricts the direct use of the tools to specialists such as Talkington and Cerrito. At Dow, Shabrang assists researchers in conducting searches.

To make the functionality more broadly available while they tackle usability problems, vendors are incorporating text mining tools as a background function to improve the effectiveness of more familiar search or document management applications.

Robb is a freelance writer in Los Angeles. Contact him at drewrobb@ sbcglobal.net.

The Future of BI

Stories in this report:

Business Intelligence

Bing’s AI chatbot came to work for me. I had to fire it.