Metadata management returns to the fore

But some IT execs still fight for user, management buy-in.

In the wake of a homeland security emergency or natural disaster, officials in New Jersey up until a few months ago had no way to simply and quickly get comprehensive information on the state's 21 counties, like population statistics or building locations.

The problem: The data definitions -- or metadata -- within state systems described county data in five different ways, said Dan Paolini, director of data management services in the New Jersey Office of Information Technology.

"Many times in an emergency, you may have to, in an ad hoc way, pull data out of two different systems you wouldn't have thought to do before," he said. "All of a sudden, you can't trust the answer to something as simple as what county something is in."

But state officials hammered out a universal way for referencing the identity of a county -- and the genders and ethnicities of its populations -- and then rolled out a master reference-data registry in March. Now, all of the state government's future IT development projects will include the preferred metadata definitions, Paolini said.

New Jersey's IT operation is one of a growing number of organizations fueling something of a revival of metadata management, as technology and new processes make it possible to overcome earlier challenges. Yet, at some companies, IT managers still face problems gaining user and/or management buy-in.

Metadata is the technical information about data, like the type and field name. Vendors began rallying around the idea of managing metadata in the late 1980s and started assembling data dictionaries or repositories as part of mainframe projects and Cobol development. But enterprise metadata management has just recently caught the eye of IT managers.

Early Failures

Early metadata management tools -- like IBM's unsuccessful AD/Cycle metadata repository, which ran on a mainframe and DB2 in the late 1980s -- failed to catch on because vendor technologies and internal processes at user companies weren't mature enough, said Stuart Carty, founder and principal of Gavilan Research Associates LLC, a metadata research firm in Danville, Calif.

In addition, he said, it often took companies between six months and a year to harvest metadata from the old development tools and database schemas. Some of today's Web-based tools use a real-time architecture, allowing users to install metadata repository tools in the morning and begin harvesting definitions the same day, he said.

This rebirth, users said, is also tied to the increasing importance of metadata in ensuring that business intelligence projects can provide managers with reliable information to make near-real-time decisions and to give developers details about Web and Java services so they can reuse them in service-oriented architectures (SOA).

"I've been working in metadata management since the early '80s, and I finally feel like I have company," said Barbara Nichols, president of Acton, Mass.-based consulting firm Metaview360.

"The promise of building a data warehouse is that you can measure and steer your business with it, but the answers to queries are coming back ambiguous," she said. "You have all those definitions in there that may or may not mean the same thing, and they are being used to calculate if we should build a new manufacturing facility. [These are] bet-your-business kinds of decisions."

This month, Unilever Latin America will embark on a metadata management project as part of an effort to create a shared BI services group that will provide reports to users in 14 countries in South America, Mexico and the Caribbean, said Monica Parisi, information architecture manager at the Brazil-based company.

Unilever has disparate metadata stored in its extract, transform and load tools and other BI applications, she said.

"We don't have an integrated place to combine this metadata and really take advantage of it," Parisi said. "Today, we have many, many people working to re-create views and to combine data to use in the warehouse."

If workers in the shared-services group receive a request for a new view of data from the warehouse or a new report, they need a centralized location to access metadata, she added.

Vendors, meanwhile, have also been focusing more on metadata management. For example, a key reason for IBM's $1 billion acquisition of Ascential Software Corp. was to meld IBM's information management technology with Ascential's integration and metadata management tools.

Just after the acquisition closed in April, IBM officials outlined plans to build a single repository architecture -- including metadata discovery, exchange and management -- that will incorporate IBM products and Ascential tools [Quick Link 54230].

On the BI side, Business Objects SA in January rolled out BusinessObjects XI, which includes bolstered business and technical metadata management.

Meanwhile, Information Builders Inc. in March unveiled its new BI tool set called WebFocus 7, which includes new metadata management tools from its iWay Software Inc. subsidiary.

Verizon Communications Inc. is tackling enterprise metadata management for its SuperPages online directory. Verizon is using IBM's WebSphere MetaStage, a metadata directory built by Ascential that provides business users and developers with common data definitions.

For Verizon's business users, MetaStage provides data lineage -- details about the source of data -- so BI reports available via the company's executive portal have better context, said Mark Abramson, an enterprise data architect at the New York-based telecommunications company.

"People can go to the data dictionary, drill into that and see detailed information on data ... to allow them to understand what they are looking at when they look at a report," he said.

For developers, Verizon is using MetaStage to create common definitions of services available for reuse within its SOA, Abramson said.

For example, metadata about customers with advertisements in Verizon's SuperPages will be stored in a different system from the one with metadata about those who just have a listing within the directory.

With MetaStage, developers can find out which system a service came from and determine its format within that system so that the service can be easily shared, he added.


Tackling enterprise metadata management is still riddled with challenges. One of the biggest, IT executives said, is getting user buy-in.

The IT director at a large Europe-based investment bank, who asked not to be named, said his company has been able to gain user support for metadata management efforts only for specific integration projects.

"When you try and look at metadata management as a problem in its own right, it's hard to get people to agree on definitions and structures around the data," the IT director said. "We find it very difficult for people to take ownership for the metadata."

In New Jersey, the state's IT shop struggled with how to structure its metadata repository, said Paolini.

"One of the first mistakes we made was thinking, 'I need to get one vendor with these great repository tools,'" he said. "There is no single repository that you can buy that is going to adequately meet every metadata requirement."

Instead of building "one giant repository in the sky," the state has focused on building separate repositories for different types of data, like business definitions, data models and requirements, Paolini added.

To manage the reference data that points users to the various repositories, the state uses software from Englewood Cliffs, N.J.-based Data Foundations Inc.

Hewlett-Packard Co. in April launched an enterprise metadata management pilot that will run through next month. HP uses SuperGlue, a metadata analysis tool from Informatica Corp.

Like other companies tackling metadata management, HP is trying to develop a process that will lessen the time it takes to develop reports that analyze information companywide, said Deborah Poindexter, enterprise data architect at HP.

Still, Poindexter is struggling to justify the investment to upper management because there is no internal return-on-investment data available yet.

"In the BI environment ... you have a lot of analysis going on, and upper management is starting to understand metadata management," she said. "If I can prove to them I can produce a report that shows them where their data came from and where it is going to expedite analysis -- that is a huge win and easily demonstrated."

Copyright © 2005 IDG Communications, Inc.

Shop Tech Products at Amazon