Data Scrubbing
Computerworld - The need to scrub data is made pretty clear by simple questions like this one: Are Jerry L. Jonson of 16 Clarke St., Altuna, PA, and Gerry L. Johnson of 16 Clark Street, Altoona, Penn., the same person? You would probably say that most likely they are. But a computer, without help from specialized software, would deal with the information as though it were about two different guys.
The human eye and mind recognize that the differences between the two sets of data records are probably the result of mistakes or inconsistencies in data entry. Weeding out and fixing or discarding inconsistent, incorrect or incomplete data is what's called data scrubbing or cleansing.

![]()
Credit: Melinda Beck ![]()
The issue of data hygiene has become increasingly important as more and more corporations implement complex customer relationship management (CRM) systems and build data warehouses that merge information from many different sources.
Without data cleansing, the IT staffs of those companies face the unappetizing prospect of merging corrupt or incomplete bits of data from multiple databases. A single piece of dirty data might seem like a trivial problem, but if you multiply that "trivial" problem by thousands or millions of pieces of erroneous, duplicated or inconsistent data, it becomes a prescription for chaos.
Sources of Dirty Data
In its 2001 report about organizations implementing data warehouses for the purpose of business intelligence, Cutter Consortium identified the following causes of dirty data:
Poor data entry, which includes misspellings, typos and transpositions, and variations in spelling or naming.
Data missing from database fields.
Lack of companywide or industrywide data coding standards (a big problem in health care, for example).
Multiple databases scattered throughout different departments or organizations, with the data in each structured according to the idiosyncratic rules of that particular database.
Older systems that contain poorly documented or obsolete data.



- Excel 2010 Cheat Sheet
- Register for this Computerworld Insider Cheat Sheet and gain access to hundreds of premium content articles, guides, product reviews and more.
- X-Ray of the PCI Process-4 Proactive Steps
- This white paper from Forrester Research Inc., helps break PCI into understandable components. Security and risk professionals will gain knowledge and insight into...
- Forrester: Economic Impact of Switching to Google Apps
- Content provided by Google
Read this Forrester report on the "total economic impact" of Google Apps, and learn how switching to Google Apps creates... - Intelligent Systems: Unlocking Hidden Business Value with Data
- An intelligent system enables data to flow across an enterprise infrastructure, spanning the devices where valuable data is gathered from employees and customers,...
- Concepts of NonStop SQL/MX
- For DBAs and developers who are familiar with Oracle solutions and want to learn about NonStop SQL/MX, this whitepaper provides an overview of...
- HP Advanced Information Services for SAP In-Memory Appliance (SAP HANA)
- Organizations are eager to connect the vast amounts of data available within and outside their businesses to compete more effectively and make better... All BI and Analytics White Papers
- Quantifying the Business Value of VMware View - Webcast
- Many enterprises have discovered that the use of virtualization to support desktop workloads creates a range of significant benefits. These benefits include price...
- Good to Great - How to Take Business Analytics to the Next Level
- By attending this webcast you will learn how you can implement an effective BA strategy that will deliver maximum strategic value to your...
- Supporting Mobile Productivity With A Limited IT Budget
- Join us and hear from Kaseya mobile IT management experts as we discuss core strategies for supporting the mobile revolution on a shoestring...
- User Experience Monitoring
- In this webinar, you will learn hints & tips for improving end-user response times from Forrester Research analyst, Jean-Pierre Garbani.
- Hints & Tips Cisco
- Overwhelmed by tracking your Vblock, Flexpod or Cisco UCS performance? Spend one hour with Nimsoft to learn how you can eliminate the overhead... All BI and Analytics Webcasts