Receive the latest technology news and information.

	Computerworld Daily News (First Look and Wrap-Up)
	Computerworld Blogs Newsletter
	The Weekly Top 10
	Cloud Computing

View all newsletters

E-mail Address:

Industry:

Job Title:

Company Size:

Country:

State:

I would like to receive offers via e-mail from Computerworld partners.

Business Intelligence: The Future of BI

Text mining tools take on unstructured data

Companies are increasingly using text mining tools to harness the information in their unstructured data.

Drew Robb Today’s Top Stories

or Other Business Intelligence Stories

June 21, 2004 (Computerworld) -- Unstructured data, most of it in the form of text files, typically accounts for 85% of an organization's knowledge stores, but it's not always easy to find, access, analyze or use.

"We are drowning in information but are starving for knowledge," says Mani Shabrang, technical leader in research and development at Dow Chemical Co.'s business intelligence (BI) center in Midland, Mich. "Information is only useful when it can be located and synthesized into knowledge."

But a new generation of text mining tools allows companies to extract key elements from large unstructured data sets, discover relationships and summarize the information. Many organizations are deploying or considering such software to deal with their mountains of text, despite the need for specialized skills to make implementations work.

For example, since 2000 Dow's research staff has been using ClearResearch software from ClearForest Corp. in New York to extract data from a century's worth of chemical patent abstracts, published research papers and the company's own files.

"By managing the information better and eliminating the irrelevant, we've been able to reduce the time it takes for [researchers] to find what they need to read," says Shabrang.

Text mining tools take a variety of approaches. ClearResearch uses a proprietary pattern-matching methodology to search for information, categorize it and graphically show its relationship to other data.

"The software can see, discover and extract concepts, not just words," says Shabrang. "It gives us a pictorial representation of the text in the documents in an easy-to-understand chart."

Adoption Roadblocks
The text mining software available now doesn't yet match the accuracy of data mining tools, but vendors are improving their products' ability to understand context, which is key to making text mining tools effective.

"Understanding linguistics and overcoming its challenges is a horizon that has not been dealt with well," says William McKnight, president of McKnight Associates Inc., a data warehousing consulting firm in Plano, Texas. "Basic text mining is possible, but the performance needs to be improved and the tools don't scale well."

Because of these limitations, text mining tools are still niche products generally restricted to specific parts of an organization. But they are starting to catch on.

"Over the last 12 to 18 months, I have seen a lot of interest in using these tools for regulatory compliance," says Brian Babineau, a research analyst at Enterprise Storage Group Inc. in Milford, Mass. "But once that seems to be under control, people will retrofit these applications for other purposes, like data warehousing and CRM."

While there are software systems that analyze both structured and unstructured data, many companies use traditional BI software on their structured data and then turn to separate tools to analyze text-based data. Electronic Data Systems Corp., for example, has all of its 130,000 employees fill out an online questionnaire about their jobs once a year. Another three times a year, 20,000 employees answer an additional survey.

Continued...
1 | 2 | 3 | 4 | NEXT

Print this Story

Send Us Feedback

E-mail this Story

Digg this Story

Slashdot this Story

Report Contents

Featured Articles

	IE9 to tap hardware to boost performance
	New attack fells Internet Explorer
	The 5 best and worst features of Google Chrome OS
	More top stories...

	Free Web apps to help organize your holidays
	Google's Chrome OS hits BitTorrent
	Global warming research exposed after hack

See your link here

Windows 7 tricks: 20 top tips and tweaks

Getting to know Windows 7? Don't stop now: From speeding up taskbar thumbnails to reining in UAC, here are 20 ways to make Windows 7 act the way you want.

Review: Taking the Droid on the road

Is Motorola's new Droid good enough to vanquish iPhone envy? To find out, we took it on a 3-day trip.

Opinion: Linux desktop turns 10; world yawns

Sure, you could always use Linux as a desktop OS, but Corel Linux 1.0 was the first distro designed for ordinary users. It's been a long, strange trip since then.