Metrics fall short of mark on security

Computerworld - LAS VEGAS -- Metrics that measure only the performance of security personnel and the tools they use are of limited value when it comes to assessing the true effectiveness of data protection investments, said IT managers at a conference here last week.

Attendees at the conference, which was held by the Information Systems Audit and Control Association, said that a better approach is to combine information security metrics with other performance indicators, such as the effect that security problems have on internal business processes or the availability of applications to end users.

"Most metrics only indicate that a security program is doing the things it needs to do," said Scott Blake, chief information security officer (CISO) at Boston-based Liberty Mutual Insurance Co. "It doesn't get us to a point where we have a real understanding of the risks to the data."

For instance, metrics that measure compliance with internal requirements or the time it takes to patch systems might offer good insight into how effectively a security program is working, Blake said.

"But in my mind, it only tells us that we are doing things," he added. What's really needed, Blake said, is insight into whether the overall risk to business operations has been reduced as a result of investments in IT security.

Qualitative Assessments

The key is to focus on metrics that demonstrate "quality of accomplishment," said Nancy DeFrancesco, CISO at the U.S. Department of Commerce in Washington. "Metrics are extremely important, but you need to have both quantitative and qualitative ones."

For example, DeFrancesco said, in addition to having a set of quantitative metrics, it helps if a security staff can show that it has a repeatable process in place for handling security incidents.

"A quantitative measure may indicate your posture at one point in time, whereas a qualitative measure would promote the overall maturity of your IT security organization," she said.

Technology-oriented metrics can give IT staffs "important data points to either validate or invalidate" questions about issues such as attack trends, said John Pironti, principal security consultant at Unisys Corp. in Blue Bell, Pa.

But because those metrics don't give a picture of the true business impact of security investments, it's also necessary to track key performance indicators on the business side and show how attacks against pieces of a company's IT infrastructure can affect operations, Pironti said.

Examples include measuring the amount of time that end users are unable to access their systems because of a worm infection, or tracking the number of complaints resulting from systems being unavailable to users, he said.