Coping With Infoglut

Computerworld - The Internet has buried companies under a mudslide of unstructured data. One of the most pressing problems facing IT is how to turn all that data that won't fit into rows and columns into useful information. And while the amount of unstructured data is growing exponentially, the tools for dealing with it haven't kept pace.
The magnitude of the deluge is staggering. Approximately 85% of all digital business information exists only as unstructured data, according to research by Merrill Lynch. Most of that comes from the increasing use of the Web as an internal and external business channel.
The majority of unstructured data consists of text documents. Some of those, such as memos, letters, marketing materials and research documentation, have presented a storage and retrieval problem in business since before there was digital media. And now, in addition to those documents, there are e-mail messages, customer queries and responses from sales and support representatives generated by CRM applications, user group postings and chat messages, as well as images, movies and Web pages with their hyperlinked information.
E-mail alone has burgeoned; market research firm IDC predicts that there will be more than 60 billion messages sent annually by 2006. And besides the business imperative to take control of the organization's knowledge base, federal regulatory initiatives increase the pressure on companies to both archive e-mail and develop a way to research the content of the messages.
The other 15% of all business information -- the structured data that generally resides neatly in spreadsheets and databases -- is being sliced, diced, massaged and squeezed for every bit of business intelligence it will yield. Technologies to address unstructured data can't match the functionality of these real-time analytics for structured data, and users have been slow to adopt them. Tim Berners-Lee, the Web's primary architect, has famously observed that most of the information on the Web is designed for human consumption and resists being organized or analyzed by any automated process.
What do companies lose by not having the means to use unstructured data? Employees' time for one thing -- recent studies indicate that information workers spend as much as a quarter of their time just finding and gathering job-related information. Nuanced information about trends and customer attitudes for another.
Vendors recognize both the challenge and the opportunity presented by unstructured data. When recently asked what the next big thing in business intelligence and data warehousing would be, Don Hatcher, SAS Institute's vice president of technology strategy, answered emphatically, "Unstructured [data], without a doubt. We're