Metrics fall short of mark on security

A better approach: combine information security metrics with other performance indicators

LAS VEGAS -- Metrics that measure only the performance of security personnel and the tools they use are of limited value when it comes to assessing the true effectiveness of data protection investments, said IT managers at a conference here last week.

Attendees at the conference, which was held by the Information Systems Audit and Control Association, said that a better approach is to combine information security metrics with other performance indicators, such as the effect that security problems have on internal business processes or the availability of applications to end users.

"Most metrics only indicate that a security program is doing the things it needs to do," said Scott Blake, chief information security officer (CISO) at Boston-based Liberty Mutual Insurance Co. "It doesn't get us to a point where we have a real understanding of the risks to the data."

For instance, metrics that measure compliance with internal requirements or the time it takes to patch systems might offer good insight into how effectively a security program is working, Blake said.

"But in my mind, it only tells us that we are doing things," he added. What's really needed, Blake said, is insight into whether the overall risk to business operations has been reduced as a result of investments in IT security.

Qualitative Assessments

The key is to focus on metrics that demonstrate "quality of accomplishment," said Nancy DeFrancesco, CISO at the U.S. Department of Commerce in Washington. "Metrics are extremely important, but you need to have both quantitative and qualitative ones."

For example, DeFrancesco said, in addition to having a set of quantitative metrics, it helps if a security staff can show that it has a repeatable process in place for handling security incidents.

"A quantitative measure may indicate your posture at one point in time, whereas a qualitative measure would promote the overall maturity of your IT security organization," she said.

Technology-oriented metrics can give IT staffs "important data points to either validate or invalidate" questions about issues such as attack trends, said John Pironti, principal security consultant at Unisys Corp. in Blue Bell, Pa.

But because those metrics don't give a picture of the true business impact of security investments, it's also necessary to track key performance indicators on the business side and show how attacks against pieces of a company's IT infrastructure can affect operations, Pironti said.

Examples include measuring the amount of time that end users are unable to access their systems because of a worm infection, or tracking the number of complaints resulting from systems being unavailable to users, he said.

"Business managers definitely want to be talked to in business terms," Pironti noted. "They want to know how a business process might be affected by a potential threat."

Focusing on measuring only the work that's being done internally to protect systems gives security managers little of the information they need to respond proactively to changing business needs, said Howard Schmidt, a former White House cybersecurity adviser who is now CEO of R&H Security Consulting LLC in Issaquah, Wash.

A corporate security team might be able to demonstrate that its perimeter defenses blocked more than 15,000 viruses during a specific period, Schmidt said. "But how do you measure how many infections you prevented by blocking all those viruses?" he asked.

Another challenge "is that often you can measure a negative [event] but not a positive one," he added. As a result, it becomes very difficult to demonstrate the business value of security programs, Schmidt said.


Security Trackers

Key performance indicators that CISOs should be paying attention to include:

The annual number of information security incidents.

The business impact of security threats.

Enterprisewide awareness of the information security program.

The accuracy of security threat analyses.

Reductions in incident management and response costs.

SOURCE: John Pironti, Unisys Corp.

Copyright © 2005 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon