Data Scrubbing

Transferring Best Practice Analytics from Experts to Everyone When The BI You Know Isn't The BI You Need Gaining Insights Through Analytics Sign up to receive Security Resource Alerts

February 10, 2003 (Computerworld) -- The need to scrub data is made pretty clear by simple questions like this one: Are Jerry L. Jonson of 16 Clarke St., Altuna, PA, and Gerry L. Johnson of 16 Clark Street, Altoona, Penn., the same person? You would probably say that most likely they are. But a computer, without help from specialized software, would deal with the information as though it were about two different guys.

The human eye and mind recognize that the differences between the two sets of data records are probably the result of mistakes or inconsistencies in data entry. Weeding out and fixing or discarding inconsistent, incorrect or incomplete data is what's called data scrubbing or cleansing.

Credit: Melinda Beck

"Dirty data" has been a problem for as long as there have been computers -- or maybe for as long as people have attempted to gather and analyze information. It's a large part of the "garbage in" that can result in the worthless "garbage out" of a computing process.

The issue of data hygiene has become increasingly important as more and more corporations implement complex customer relationship management (CRM) systems and build data warehouses that merge information from many different sources.

Without data cleansing, the IT staffs of those companies face the unappetizing prospect of merging corrupt or incomplete bits of data from multiple databases. A single piece of dirty data might seem like a trivial problem, but if you multiply that "trivial" problem by thousands or millions of pieces of erroneous, duplicated or inconsistent data, it becomes a prescription for chaos.

Sources of Dirty Data

In its 2001 report about organizations implementing data warehouses for the purpose of business intelligence, Cutter Consortium identified the following causes of dirty data:

• Poor data entry, which includes misspellings, typos and transpositions, and variations in spelling or naming.

• Data missing from database fields.

• Lack of companywide or industrywide data coding standards (a big problem in health care, for example).

• Multiple databases scattered throughout different departments or organizations, with the data in each structured according to the idiosyncratic rules of that particular database.

• Older systems that contain poorly documented or obsolete data.

Print this Story

Send Us Feedback

E-mail this Story

Digg this Story

Slashdot this Story

Data Cleansing Data Scrubbing

"Pop quiz: What do RedPrairie, FrontRange Solutions, Concur Technologies, Servigistics and Logility have in common? Find out from Don Tennant..." Read more... "We don't need al-Qaeda to blow us up. We are perfectly capable of lighting the fuse ourselves, courtesy of our..." Read more... Read more Business Intelligence posts or See all Blogs

Report: Google holds emergency meeting on revived Microsoft-Yahoo deal Analysts: Partial Microsoft-Yahoo deal won't appease Icahn, investors YouTube declines Sen. Lieberman request to remove terrorist-produced videos More top stories...

Video: The Top 10 -- plus 1 -- funniest YouTube parodies of the presidential campaigns Apple owns 66% of $1,000-plus retail market, NPD Group says PayPal plugs cross-site scripting hole that sidestepped stronger security

Shuttle Columbia's hard drive data recovered from crash site Specialists have retrieved about 99% of the data on a disk drive on board the crashed space shuttle Columbia. Don't miss the photographs of the recovered drive. The top 15 vaporware products of all time These big ideas were supposed to revolutionize technology, but they never actually appeared. In a few cases, you'll be glad they didn't. Opinion: Malware vs. anti-malware, 20 years into the fray Nearly 20 years after the first Internet worm, Steven J. Vaughan-Nichols takes stock of the malware/anti-malware landscape and spotlights how the two sides are approaching the battle. Leopard at six months: Does it live up to the early hype? Though some thought it was released too soon, Mac OS X 10.5 has matured into a solid operating system, says reviewer Michael DeAgonia. Windows Vista A to Z Reviews, analyses, how-tos, visual tours, hot issues and predictions about Microsoft's new OS. More Continuing Coverage After 9/11: Homeland SecurityAsk a Premier 100 IT LeaderBlackBerry Legal BattleData Security BreachesElectronic VotingFirefoxH-1B VisasHP's Boardroom ScandalHandheld DevicesHurricane Katrina AftermathMicrosoft Legal IssuesMicrosoft's VistaOpen SourceRFID TechnologySCO's Linux FightSarbanes-Oxley ActSpamVoice Over IPWi-FiWindows Meta File VulnerabilityWindows XP Service Pack 2 IT Careers 2010 Four years from now, the IT field will be a vastly different place. Will you be ready? More Special Reports Premier 100 IT Leaders 2007 Best in Class2006 Computerworld Horizon Awards2006 Premier 100 IT Leaders2006 Salary Survey2007 Best Places to Work in IT2007 IT Forecast2007 Premier 100 IT Leaders40 Years of ComputerworldASPs, Take TwoBest Places to Work in IT 2003Best Places to Work in IT 2004Best Places to Work in IT 2005Best Places to Work in IT 2006Business Intelligence Home RunsBusiness Intelligence: Smarter BIBusiness Intelligence: The Future of BICRM Goes VerticalCRM: Sober CRMCareer Planning Guide 2005Careers: IT Profession 2010Computerworld Horizon Awards 2005Data Management: Mining for GemsData Management: Taming Data ChaosDevelopment: Hard-Workin' Web SitesDevelopment: New Tools New ChoicesDevelopment: The New World of Application DevelopmentDevelopment: The Web Services TsunamiDevelopment: Web Services HurdlesDisaster Recovery: Preparing for the WorstE-commerce Grows UpE-mail/Groupware: Big DecisionsFaces of Mobile ITForecast 2006Global MobileHardware: Multiple Cores, Multiple ChallengesHardware: On-Demand Un-HypedHardware: The Shape of Things to ComeHorizon Awards: 10 Cool Cutting Edge TechnologiesIT Management: Guide to Managing VendorsIT Management: Navigating Global ITIT Management: Recovery AheadIT Management: The Future of ITIT Management: The Resourceful Project ManagerInnovative Technology Awards 2004Management: Reinventing ITMicrosoft in TransitionMobile/Wireless Leaders and LaggardsMobile/Wireless: The Untethered WorkerMobile/Wireless: Tiny Gadgets, Huge CostsNetwork Management: Taking ControlNetworking: Turbulence Ahead!Networking: VoIP Goes MainstreamOperating Systems: In the Slow LaneOperating Systems: Linux Goes GlobalOperating Systems: Should You Nix Unix?Outsourcing: Offshore Buyer's GuideOutsourcing: Outsourcing DangersPremier 100 IT Leaders 2004 Best in ClassPremier 100 IT Leaders 2005 Best in ClassROI: Do the Math!Salary Survey 2003Salary Survey 2004Salary Survey 2005Security Action PlanSecurity: Compliance HeadachesSecurity: Proactive SecuritySecurity: Risk and RewardSecurity: Tips From Security ProsSouped-up SecurityStorage: Battling ComplexityStorage: Cheap & Secure Data StoresStorage: Stretching Your Storage DollarsStorage: The Lean Storage MachineStorage: The New Rules of StorageStorage: The Ultimate Backup GuideSupply Chain: Missing LinksSupply Chain: RFID Reality CheckThe Business of SecurityWeb 2.0 SecurityWireless: Wireless At Work All Zones Application Performance Zone Enterprise-Class Security Zone The File Data Management Zone Grid Computing on Windows Zone Security Management Zone ITIL Best Practices Zone The SAS Zone Storage Virtualization Zone See your link here

Computerworld Report: Storage Gets Strategic Download this Computerworld Report, free, compliments of HP. (Source: Computerworld) Data Storage has emerged from the back room to become a key part of regulatory compliance, disaster recovery and strategic tecnhology plans. Learn more in this new this Computerworld report, a $49.95 value, available free for a limited time, compliments of HP. Download this executive briefing  Why SaaS is Vital to Email and Web Security Why SaaS is Vital to Email and Web Security Download this webcast, free, compilments of Webroot Software Go to the webcast  The Advantages of a Hosted Messaging Security Solution Get this report now! (Source: Microsoft Office Live Meeting) Messaging management is becoming more difficult thanks to the growing malware threat. At the same time, messaging system administrators are under enormous pressure to push their messaging infrastructures to do more than ever, including archiving messaging content for regulatory compliance, archiving to support legal discovery and for overall litigation support, providing services to a growing body of mobile users, and ensuring continuity by making the messaging system more reliable, and managing policies for message encryption. Download this white paper  White Papers Read up on the latest ideas and technologies from companies that sell hardware, software and services. Guide to Network Frontline Troubleshooting Securing Financial Services Beyond the Perimeter Meeting PCI Compliance with SonicWALL Global Management System View more whitepapers

• Data Cleansing
• Data Scrubbing

• Report: Google holds emergency meeting on revived Microsoft-Yahoo deal
• Analysts: Partial Microsoft-Yahoo deal won't appease Icahn, investors
• YouTube declines Sen. Lieberman request to remove terrorist-produced videos
• More top stories 

Intercept Spam & Viruses With MessageLabs MessageLabs is offering a complimentary 30 day trial of its managed Anti-virus and Anti-spam security solutions. MessageLabs guarantees complete protection against all know and unknown email threats. By providing 24 hour support, your business can increase productivity and decrease risk. Register for a complimentary trial and receive a free datasheet. Download this white paper now!



XenServer FREE trial Citrix XenServer is the simplest and most effective way to virtualize and provision servers. XenServer combines comprehensive server virtualization capabilities with unparalleled scalability, performance, economics, and ease-of-use. Based on the open source Xen hypervisor, XenServer delivers fast performance, easy management, and advanced features such as live migration. Request free trial now



Since You Asked A weekly storage column from storage analyst, Steve Duplessie of the Enterprise Strategy Group Click here to read the latest column by Steve Duplessie