Model Mania

CIOs are faced with a confusing array of quality frameworks. Here's a guide to their strengths and weaknesses.

Shocked and awed by the industrial might of Japan in the 1980s, U.S. companies got religion—the quality religion. They rushed to improve their business processes by adopting a host of quality frameworks, like ISO 9000 for the enterprise, Six Sigma for the plant and the Capability Maturity Model (CMM) for software engineering.

Today, IT managers have a bewildering array of quality disciplines to choose from. Some, such as Six Sigma, ISO 9000 and the Malcolm Baldrige program, may be dictated to you by your CEO. Others, such as Control Objectives for Information and Related Technology (CobiT), may be imposed by your auditors. And IT-focused disciplines may originate in your own shop, such as CMM for software development and the Information Technology Infrastructure Library (ITIL) for IT operations and services.

While there is some overlap among these quality frameworks, in most cases, they don't conflict. Indeed, most large companies use two or three of them. For example, IBM uses ISO 9000, CMM, ITIL, Six Sigma and several homegrown quality programs.

Meanwhile, other equally sophisticated companies don't use any of them, preferring to roll their own. For instance, MasterCard International Inc. has adapted parts of a number of programs to its own way of doing business. It underwent an external assessment for CMM a year ago and implemented some ideas from that, but it hasn't adopted the framework formally.

"We have a hybrid of quality programs," says Sheryl Andrasko, vice president for systems development at MasterCard. The program has reduced the development time for new software releases from 18 months to 12 and has reduced the number of software defects as well, she says.

Other companies, such as Nortel Networks Ltd., say the choice should be driven by customers and partners. Nortel uses a telecommunications-oriented version of ISO 9000 because that's what its customers use.

Model Mania
Credit: Maria Rendon
For some companies, an outside body's stamp of approval, such as an ISO 9000 or CMM certification, or the cachet that comes from a Baldrige award, may be an important factor. For example, a defense contractor may not be able to get work without a high CMM assessment. And an ISO 9000 badge may be a requirement for doing business, especially outside the U.S.

But a company can overspend on any of these programs, says Matt Light, an analyst at Gartner Inc. "We have a philosophy called 'just enough process,' " he says. "So to roll your own and apply it just where it makes sense is often the best choice for organizations that don't have certification requirements."

Nevertheless, you should do something on the quality front, urges Michael J. Ashworth, CIO of the investment banking unit at J.P. Morgan Chase & Co. "All of these things are just better ways of doing the things that people are trying to do on an ad hoc basis," he says. "They are not mumbo jumbo; they are codified common sense."

Capability Maturity Model Integration (CMMI)

Sponsor: Software Engineering Institute, Carnegie Mellon University

What it is: The CMMI extends and combines the Capability Maturity Model for Software (SW-CMM), the Systems Engineering Capability Model and the Integrated Product Development Capability Maturity Model. SW-CMM is a collection of best practices for software development and maintenance. It allows companies to assess their practices and compare them to those of other companies. The SW-CMM measures process maturity, which progresses through five levels: Level 1 (initial), 2 (managed), 3 (defined), 4 (predictable) and 5 (optimizing).

Strengths: Very detailed. Geared specifically to software development organizations. Focuses on continuous improvement, not just on maintaining a certification. Can be used for self-assessment.

Limitations: Doesn't address IT operations issues, such as security, change and configuration management, capacity planning, troubleshooting and help desk functions. Sets goals, but doesn't say how to meet them. (For example, CMMI says to do requirements analysis but doesn't say how to do requirements analysis.)

For 15 years, companies that wanted to significantly improve their software development practices—and earn a merit badge for all the world to see—embarked on a long, hard road called CMM for Software, a road map that can lead companies from a state of semichaos, where most are today, to one marked by the precision, repeatability and low error rates normally associated with a manufacturing assembly line.

CMMI, recently unveiled by the Software Engineering Institute, is a more comprehensive process-maturity framework that combines SW-CMM with broader disciplines in systems engineering and product development. The institute says it will eventually stop supporting SW-CMM in favor of CMMI.

The IT shop at J.P. Morgan Chase uses SW-CMM, while the company overall works under Six Sigma. "We've got our development teams up to CMM Level 2 and are pushing toward Level 3 in some cases," Ashworth says.

Ashworth says the move from Level 1 to Level 2 brought with it more reliable planning, so application features are more likely to be right the first time, reducing costly rework. The investment bank has seen the following additional benefits, he says:

  • A 20% to 25% reduction in postimplementation defects.
  • Reduced efforts to support operational systems because they are more reliable. "Emergency" releases to fix bugs have fallen by 60%.
  • Better management of globally distributed projects because terminology and specifications are standardized.
  • Better performance from suppliers because requirements are better specified.

Nevertheless, Ashworth cautions against "analysis paralysis" when it comes to evaluating the results of CMM. "We found it not useful to spend too much time trying to measure things, rather than just doing it," he says.

Motorola Inc. has software development units at all five SW-CMM levels, but most are at Levels 3 or 4, according to Anthony Carter, director of the Digital Six Sigma program at Schaumburg, Ill.-based Motorola. He says that as groups reach Level 5, they'll migrate to CMMI. The product development framework in CMMI makes it an attractive choice for a company that makes products such as cell phones that contain software, he says.

The IT organization at Capital One Financial Corp. in McLean, Va., is at Level 1 and plans to reach Level 2 by the end of this year and Level 3 by the end of 2005, says Ray Frigo, vice president of IT management services. But unlike, say, a defense contractor that wants to become certified at a high CMM level in order to sell to the Pentagon, Capital One doesn't feel compelled to follow CMM disciplines to the letter.

"We developed a process framework to provide repeatable, consistent delivery," Frigo says. "We are picking and choosing elements of CMM and using CMM scoring to assess where we need to develop processes."

Moving from one maturity level to the next can entail two years or more of hard work, and in some cases, it's not worth the effort, users say. For example, Allstate Insurance Co. wants to move from Level 1 to Level 3 and stop there. "We really don't see the need to go to Level 4 or 5," says Robin Richmond, an assistant vice president at Allstate Protection Technology. "We can see payback from getting to Level 2 and 3. We are hoping for speed to market, efficiencies and improved quality."

And Richmond says she won't migrate to CMMI anytime soon. "It's very difficult to find people with experience in it as assessors or as implementers," she says.

Control Objectives for Information and Related Technology (CobiT)

Sponsor: Information Systems Audit and Control Association and the IT Governance Institute

What it is: An audit-oriented set of guidelines for IT processes, practices and controls. Geared to risk reduction, focusing on integrity, reliability and security. Addresses four domains: planning and organization, acquisition and implementation, delivery and support, and monitoring. Has six maturity levels, similar to CMM's.

Strengths: Good checklists for IT. Enables IT to address risks not explicitly addressed by other frameworks and to pass audits. Can work well with other quality frameworks, especially ITIL.

Limitations: Says what to do but not how to do it. Doesn't deal directly with software development or IT services. Doesn't provide road map for continuous process improvement.

Lance Turcato, managing director for technology infrastructure and security oversight at Charles Schwab & Co., calls CobiT "an IT governance tool" to help IT managers understand what controls are needed and how to measure the effectiveness of those controls. "There's an audit tool that's part of it, so that auditors can audit against those same criteria," he adds.

CobiT takes considerable effort to integrate into a company's processes. "The statements in CobiT are very generic, so we had to turn it into 'Schwab-speak' so people could understand it," Turcato says. "The biggest challenge was getting everyone to buy into it. What we had to do is determine who are the appropriate people throughout the technology group that own these controls and educate them in CobiT."

Lockheed Martin Corp. has four units at CMMI Level 5. It also uses Six Sigma and ISO 9000 disciplines in various parts of its IT organization, but CobiT is the "umbrella quality framework," says CIO Joseph R. Cleveland. He says it provides useful checklists in each of its four domains.

For example, he says, for something as simple as adding the BlackBerry PDA to the company's catalog of approved devices, CobiT will ask whether there's help desk support for it, whether security has been addressed, whether procedures are in place to acquire and maintain the device and so on.

Cleveland says CobiT fits in nicely with CMMI, with CobiT pinpointing the need for certain controls and CMMI putting them into place. Auditors' questions can often be satisfied by pointing to aspects of CMMI, he says.

IT Infrastructure Library (ITIL)

Sponsor: The U.K. Office of Government Commerce, Pink Elephant Inc. and others.

What it is: Best practices for IT service management and operations (such as service-desk, incident, change, capacity, service-level and security management). Especially popular in Europe.

Strengths: Well established, mature, detailed and focused on IT production and operational quality issues. Can combine with CMMI to cover all of IT.

Limitations: Doesn't address the development of quality management systems. Not geared to software development processes. Use is highly dependent on interpretation.

While CMM is the de facto quality standard for software development processes, ITIL for many is the tool of choice for the operations and infrastructure side of IT, particularly for IT services.

Capital One rolled out an ITIL program for internal and external customers in 2001 in the wake of very rapid growth accompanied by an increasing number of "service interruptions," says Gregory Gannon, vice president of technology delivery. By 2003, Capital One had reduced "production incidents"—such as system crashes and software-distribution errors—by 30% and had reduced "business-critical" or "Severity 1" incidents by 92%, he says.

ITIL tracks problems in IT service areas such as help desk, applications support, software distribution and customer-contact system support, and it overlaps CMM in certain areas such as configuration management. For example, Gannon says, ITIL tracks the changes made to operational systems, but the quality of those changes—in terms of the number and severity of problems resulting from them—is more a CMM metric.

ITIL facilitates root-cause analysis of problems, Gannon says. "We used to be pretty good at service restoration, but the reason we had to do so much service restoration was because we were restoring service, but not fixing the problem," he adds.

ITIL isn't a substitute for ISO 9000, Gannon says, because ISO 9000 is more relevant to certification of processes. Capital One has some Six Sigma efforts under way, but they're more on the business side of the house than on the IT side, he adds.

Six Sigma

Sponsor: Developed by Motorola Inc.

What it is: A statistical process-improvement method focusing on quality from a customer's or user's point of view. Defines service levels and measures variances from those levels. Projects go through five phases: define, measure, analyze, improve and control. The Design for Six Sigma variant applies this method's principles to the creation of defect-free products or services, rather than the improvement of existing ones.

Strengths: A data-driven approach to finding the root causes of business problems and solving them. Takes into account the cost of quality. In IT, best applied for relatively homogeneous, repeatable activities such as call center or help desk operations. Design for Six Sigma can help develop good software specifications.

Limitations: Originally designed for manufacturing environments; may be difficult to apply to processes that aren't already well defined and measurable. Can improve a process but doesn't tell you if you have the right process to begin with.

1 2 Page 1
Page 1 of 2
7 questions to ask your EMM provider about GDPR compliance
Shop Tech Products at Amazon