Computerworld - The need to scrub data is made pretty clear by simple questions like this one: Are Jerry L. Jonson of 16 Clarke St., Altuna, PA, and Gerry L. Johnson of 16 Clark Street, Altoona, Penn., the same person? You would probably say that most likely they are. But a computer, without help from specialized software, would deal with the information as though it were about two different guys.
The human eye and mind recognize that the differences between the two sets of data records are probably the result of mistakes or inconsistencies in data entry. Weeding out and fixing or discarding inconsistent, incorrect or incomplete data is what's called data scrubbing or cleansing.
Credit: Melinda Beck
The issue of data hygiene has become increasingly important as more and more corporations implement complex customer relationship management (CRM) systems and build data warehouses that merge information from many different sources.
Sources of Dirty Data
In its 2001 report about organizations implementing data warehouses for the purpose of business intelligence, Cutter Consortium identified the following causes of dirty data:
Poor data entry, which includes misspellings, typos and transpositions, and variations in spelling or naming.
Data missing from database fields.
Lack of companywide or industrywide data coding standards (a big problem in health care, for example).
Multiple databases scattered throughout different departments or organizations, with the data in each structured according to the idiosyncratic rules of that particular database.
Older systems that contain poorly documented or obsolete data.
- Deep Security +VMware vSphere with Operations Management Most midsize organizations are highly virtualized on VMware, and while this has produced significant savings, it also has created new challenges when it...
- Single-Vendor Security Ecosystems Offer Concrete Benefits Over Point Solutions IT security decision-makers from companies with 100 to 5,000 employees evaluates the current endpoint security solution market based on Forrester's own market data,...
- Unlocking the Promise of Demand Sensing and Shaping through Big Data Analytics Many organizations have limited insight into big data. These limitations have significant opportunity costs and can have a negative effect on identifying and...
- The Brave New World of Customer-Centric Manufacturing The Unique Opportunity for Manufacturers to Better Understand their Consumers
- Fundamentals of the Unified Communications BE 6000 The Cisco Business Edition 6000 delivers the superior performance, system redundancy, and broad application integration you need. Well suited for businesses with 100...
- Cisco Collaboration for Midsize Business Learn what Cisco Collaboration Solutions can do for your midsize business. In this 2-minute video, see how collaboration tools can make your business... All Business Intelligence/Analytics White Papers | Webcasts
Our new bimonthly Internet of Things newsletter helps you keep pace with the rapidly evolving technologies, trends and developments related to the IoT. Subscribe now and stay up to date!