Skip the navigation
)

Data Scrubbing

By Tommy Peterson
February 10, 2003 12:00 PM ET

Computerworld - The need to scrub data is made pretty clear by simple questions like this one: Are Jerry L. Jonson of 16 Clarke St., Altuna, PA, and Gerry L. Johnson of 16 Clark Street, Altoona, Penn., the same person? You would probably say that most likely they are. But a computer, without help from specialized software, would deal with the information as though it were about two different guys.

The human eye and mind recognize that the differences between the two sets of data records are probably the result of mistakes or inconsistencies in data entry. Weeding out and fixing or discarding inconsistent, incorrect or incomplete data is what's called data scrubbing or cleansing.

Data Scrubbing
Credit: Melinda Beck
"Dirty data" has been a problem for as long as there have been computers -- or maybe for as long as people have attempted to gather and analyze information. It's a large part of the "garbage in" that can result in the worthless "garbage out" of a computing process.

The issue of data hygiene has become increasingly important as more and more corporations implement complex customer relationship management (CRM) systems and build data warehouses that merge information from many different sources.

Without data cleansing, the IT staffs of those companies face the unappetizing prospect of merging corrupt or incomplete bits of data from multiple databases. A single piece of dirty data might seem like a trivial problem, but if you multiply that "trivial" problem by thousands or millions of pieces of erroneous, duplicated or inconsistent data, it becomes a prescription for chaos.

Sources of Dirty Data

In its 2001 report about organizations implementing data warehouses for the purpose of business intelligence, Cutter Consortium identified the following causes of dirty data:

• Poor data entry, which includes misspellings, typos and transpositions, and variations in spelling or naming.

• Data missing from database fields.

• Lack of companywide or industrywide data coding standards (a big problem in health care, for example).

• Multiple databases scattered throughout different departments or organizations, with the data in each structured according to the idiosyncratic rules of that particular database.

• Older systems that contain poorly documented or obsolete data.



What is Tech Briefcase?
TechBriefcase is a new, free service where IT Professionals can Search, Store and Share IT white papers and content like this. Learn more
Bookmark content
Speed up your research efforts with content across the web.
Search and Store
Find the white papers you need. Create folders for any topic.
View Anywhere
Open your briefcase on your iPhone, tablet or desktop. Share with colleagues.
Don't have an account yet?
Additional Resources
Security KnowledgeVault
WHITE PAPER
Security is not an option. This KnowledgeVault Series offers professional advice how to be proactive in the fight against cybercrimes and multi-layered security threats; how to adopt a holistic approach to protecting and managing data; and how to hire a qualified security assessor. Make security your Number 1 priority.

Read now.

Cut Communications Costs Once and for All
WHITE PAPER
New IP-based communications systems are being deployed by small and midsized businesses at a rapid rate. Learn how these organizations are enabling faster responsiveness, creating better customer experiences, speeding office or mobile interactions, and dramatically reducing existing communications costs.

Read now.

BI and Analytics White Papers
Thinking Outside The Data Warehouse
This high level, business problem focused eBook uses 5 customer scenarios to show how people and organizations are tackling real issues using IBM...
Using BD for Smarter Decision Making
This paper looks at new developments in business analytics and discusses the benefits analyzing big data bring to the business.
Measuring the Business Value of CI in the Data Center
One of the key strategies that IT teams are pursuing to reduce capital costs while boosting asset utilization and employee productivity is the...
Switching Schedulers - Not As Complicated As You Think
Changing or consolidating job schedulers may seem daunting. However, the benefits of switching to enterprise workload automation outweigh the risks. Read how BMC...
Capture-Enabled Business Process Management
Organizations today must deal with a vast amount of incoming information from many different sources. Efficient, automated business processes are critical to managing...
All BI and Analytics White Papers
BI and Analytics Webcasts
InfoSphere Warehouse Packs Demo
These flash modules make warehousing more tangible and relevant to business users through detailed explanations of the InfoSphere Warehouse Packs.
Delivery Management -- Extending Lifecycle Management
Date: Wednesday, June 20, 2012, 1:00 PM EDT

Siloed organizations continue doing the wrong things and doing things wrong, leading to increased costs,...
Leverage automation today to reduce IT complexity
Date: Tuesday, June 5, 2012, 2:00 PM EDT

Whether your B2B complexity is caused by multiple technologies due to M&A, business or application specific...
BMC Control-M - Single Point of Control Demo
With BMC Control-M, you schedule and manage everything - down to the very last platform and application - from one simple interface. It's...
BMC Control-M - Single Point of Control Demo
With BMC Control-M, you schedule and manage everything - down to the very last platform and application - from one simple interface. It's...
All BI and Analytics Webcasts
Newsletter Sign-Up

Receive the latest news test, reviews and trends on your favorite technology topics

Choose a newsletter
  1. View all newsletters | Privacy Policy
IT Jobs