Digging Into Documents
As the need to exploit unstructured data grows, text mining technology is evolving to meet it, says ClearForest's Ronen Feldman.
December 22, 2003 12:00 PM ETComputerworld -
Formerly used primarily by the intelligence community and businesses that are strongly dependent upon research, text mining technologies are now beginning to find more general acceptance. The mounds of unstructured data that have been piling up in companies for decades are growing larger as a result of new regulatory requirements that are forcing companies to retain e-mail and other documents - and to be able to find specific information in them. The key to making text mining work for business -- not to mention the intelligence community -- is striking a balance between accuracy and speed, says Ronen Feldman, chief scientist at text mining software company ClearForest Corp. in New York. In a recent interview with Computerworld's Tommy Peterson, Feldman discussed how text mining technologies work and what promise they hold for business.
What is text mining? How do you squeeze information out of unstructured data? Basically, text mining is the same thing as data mining for structured data, but for documents. So the first thing that you have to do is create some structure. In order to create structure, you actually have several possibilities. The easiest way is to work with the bag-of-words model. Basically, each document is just a collection of words. That is purely statistical -- you're doing no semantic analysis. There are still some companies who are doing this. They basically use the simplest possible approach.
The next level is categorization. You basically provide tags for the whole document. The last way to structure the documents, which is the most sophisticated, is to do information extraction. There you don't provide tags for entire documents, but you actually extract entities and relationships from the document. But that means that the processing is much more sophisticated and obviously takes more time.
Do companies have to choose speed vs. sophistication? This is the spectrum -- [bag-of-words] is the easiest and of course the fastest, but it doesn't buy you a lot of mileage, because there is no semantic analysis. With [categorization], you have a little more, but still it's still not a good enough infrastructure, because you won't have enough tags per document -- usually two or three tags per document.
Let's take a document of two pages. If you do information extraction, you can expect 50 to 100 tags, a completely different order of magnitude. Clearly, you get a much better foundation for text mining. Information extraction is the key challenge, and it's what really lies at the heart of our ClearText product.

![]()
Ronen Feldman, president and chief scientist at ClearForest Corp. ![]()
The second camp is based on machine-learning algorithms. In machine learning, you basically learn by example. There are rules, but the rules are written automatically, so it's mainly statistical. The problem is that you need to provide thousands of examples sometimes, meaning thousands of documents. Thousands of documents can take you several months. We saw in a practical project [that] customers are just not willing to do it and in many cases just killed this approach completely. They prefer to use our approach because then they can rely on generic concepts that we have developed already. It's not as though we start from scratch -- we have already developed most of the domain-specific entities.
Does this end up being knowledge management? Knowledge management is a very broad term; people have used so many different tools to do knowledge management. We are at the infrastructure level, so most of the knowledge management tools should use our software.
Do you worry about legal issues? When corporate e-mails are mined, will employees feel that their privacy is being invaded? We provide the tools; the usage is up to the customer. They need to worry if they are doing something which is illegal. We create generic technology and sell it to customers. They have to live up to traditional promises not to snoop around their employees too much. The only area I can see it used is in compliance, and that should be legal for companies to check that their employees are not doing anything they shouldn't.
Can you give me a good idea of what this technology is going to bring to a specific business? Let's take a pharmaceutical company. The researchers need to read a lot of papers in order to make inferences and get more acquainted with the subject. And usually they spend a lot of time with [the] Medline [Web site] and [scientific] journals like that. And they spend a lot of time just searching. With an application like ours, they can can take entities they are familiar with -- genes, etc. -- and specify the queries in a much more focused way. And that means that they focus a lot more on the real development. The hard labor of searching for the information will be saved, which will shorten the time that they take to find new drugs.
How will this technology change the way companies do research? I think that most of the hard labor will be saved, and you will be able to focus on thinking and making inferences and conclusions -- things machines are actually not so good at.
Data Mining
Additional Resources



Learn the important issues you must consider before starting your next mobility initiative. Get your mobility white paper from IDC now, compliments of Sybase.
White Papers & Webcasts
Data Grids & SOA
Get this paper now!
Key Strategies for Managing Data Growth
What are you storage challenges?
Data Manager Report Excerpt: File System Inventory
Cut storage costs and boost operational efficiencies.
Extending Client Refresh - 11 Steps to Maximize Savings
Register Now!
Reducing Storage Costs with F5 ARX
Save money- deploy ARX Solutions.
Lower the Cost and Complexity of a Mobile Workforce through Automation
Download This Resource Now!
Southern Company
Download Now
Managing Mobility: Improve Data Security, Compliance and Manageability
Download This Resource Now!
Defending Against the Storm
Download Now
Consolidate Your Servers and Storage to Lower Costs with Oracle Database 11g
Register for this webcast!

