Four Steps to Get Your Data in Shape

By Andrew Manby, Ascential Software

Computerworld |

As the economy improves, many companies are looking to business intelligence and corporate performance metrics to help improve productivity and cut costs across the enterprise. But what CIOs are learning is that the success of any BI implementation hinges on the accuracy, reliability and timeliness of the corporate data feeding the system.

For example, The Scotts Co., a world leader in do-it-yourself lawn and garden consumer products in Marysville, Ohio, selected BI tools from SAP AG to gain better visibility into its business ecosystem -- from suppliers through to distributors and retailers. But before getting started, the company applied data integration tools to prepare and load corporate data into the SAP system. The result: Scotts gained greater visibility into the requirements of its retailers, allowing the company to slash inventory by 30% and boost cash flow by $100 million.

Before information can be analyzed, relevant corporate data must be refined into actionable information using a technique called data integration. When applied to BI, data integration provides the necessary infrastructure to ensure that information is accessible, reliable, complete and consistent. Here are four steps to follow in preparing data for a BI system:

Step 1. Plan ahead and verify your assumptions about the data. Assume nothing about the state of your data when you begin a project. During the business requirements phase of a project, it's imperative to determine the true state of the data from all of the contributing sources within your enterprise. Specifically, assess (profile) the data in terms of availability, consistency and the validity of embedded business rules. This process should be automated and use as much data as is statistically important to you (whether that's 10%, 20% or 100%).

Profiling your data provides an immediate payback by quickly defining what format manipulations and standards are needed to ensure downstream accuracy and reliability. Failing to take this step can extend your project by 30% or more.

Step 2. Standardize, match and enrich the data. Now that you've identified any inconsistencies in your data, it's essential to correct the data; that is, standardize the names, addresses, product codes, birth dates, Social Security numbers and locations into consistent and reliable formats. There's about a 40% chance your data will require some level of enrichment to complete missing or inaccurate values. This may mean merging external information sources from Acxiom or Dun & Bradstreet with your own data. And because many firms store their customer data in 10 or more systems, it may be necessary to remove duplicate information. This process should use statistically proven matching techniques to ensure the best possible consistency and the ability to standardize any sort of data, not just customer information.

Step 3. Transform and deliver your data to order. To populate your BI system, you'll need to pack up and move your data from its source to the target data warehouse in a reliable and timely manner. This step requires the ability to extract, transform and load data from your myriad hardware platforms where it resides and deliver it to the consolidated data warehouse or data mart that feeds your BI systems. This can be one of the most time-consuming steps in the process, so you may want to investigate the benefits of parallel processing systems. These systems can save you time by extending the data movement process across all available CPUs. Alternatively, if you need immediate visibility as soon as a customer places an order, then you may be able to "trickle feed" the information instead of taking a bulk approach.

Step 4. Make sure the business can trust the data. The last step is to ensure that you can trust your data -- this means giving end users the ability to track and understand data composition and continuously monitor data quality. After delivering the data to the target system, you'll need to ensure that the business users have access to the source, definition and history of that data, as well as an understanding of how that data relates to other information throughout the enterprise. Business-friendly explanations of the definition, origin and relationship of data -- presented through a browser -- can remove ambiguity, lessen the IT support burden and help end users make better decisions.

Periodic data audits will also help ensure the continued reliability of your data. Data audits provide management with graphical assessments of data quality, based on business rules, and allow you to recognize anomalies as they develop and rectify them immediately.

In essence, the first key to BI success is to make sure you have a data integration system that can handle the full range of functions -- profiling, data quality, transformation and delivery -- and integrates easily with existing systems.

Andrew Manby is director of platform product marketing at Ascential Software Corp. in Westboro, Mass. He can be contacted at andrew.manby@ascential.com.

The Future of BI

Stories in this report:

Business Intelligence

Bing’s AI chatbot came to work for me. I had to fire it.