Meeting the data-quality challenge

Computerworld - Ask any business or IT executive if high-quality data is important to the health of his organization, and the answer will be, "Of course." But each day, organizations around the world use applications and databases filled with inconsistent, outdated or inaccurate data.
Companies routinely suffer from an inability to answer basic questions such as, "How many customers do we have?" "Who are our best customers?" "Which of our products provide the greatest profit margin?" and "What do we purchase, in what quantities and from whom?"
Chances are, the answers to most of these questions reside within a system -- or, more likely, within several systems. But when companies rely on multiple applications to find the answer, they're likely to get conflicting answers.
One traditional approach to improving an organization's data, often referred to as "data quality" or "data cleansing," usually takes place while developing a data warehouse or when consolidating data from legacy systems into a new enterprise application.
For example, consider a manufacturer building a data warehouse to serve as a single repository for business-critical data. As a first step, the company might implement a broad data-quality program to fix problematic information before it reaches the data warehouse. It may inspect the data with a data-profiling tool. Then, it might use traditional data-quality or -cleansing technology to standardize data and correct errors.
Next, a data integration or consolidation phase would identify and resolve duplicate information across sources. Finally, a data enrichment phase would allow the company to add additional value to records, such as demographic data, geographic details or product specifications.
Once completed, the manufacturer would ostensibly have a data warehouse full of solid, reliable data. But now, almost like clockwork, a new challenge emerges. Data entering the warehouse from that point forward -- from partners, employees or customers -- often fails to meet the established standards.
Some companies then turn to data monitoring. In the data warehousing example above, a data monitoring regimen might:

• Detect problems with incoming data by validating it against established business rules


• Generate alerts that flag problematic data as it enters the system


• Identify trends in data quality showing when validity of the data starts to decline