Measuring the Value of Metrics

Computerworld - One topic most information security professionals hate is metrics. When I was working as a security analyst and later as a security engineer, I always hated when my boss asked me to pull logs or query the remedy ticketing system and then use the metrics to report on various aspects of what I did or how well our security infrastructure was protecting the company.

Now that I’m the manager, I’m the one who asks for such measurements every quarter, and my guys snicker at me. But as a manager, I’ve learned to appreciate what metrics can tell me — and what they can help me get.

Every major department within the IT organization of my company is responsible for gathering quarterly data into key performance indicators (KPI). An Internet search will turn up several definitions of KPIs, some saying they are interchangeable with balanced scorecard metrics. In my mind, they are the same thing: metrics.

At my company, we use KPIs to measure the effectiveness and efficiency of key areas of IT. For example, to measure the effectiveness of our help desk, we report on the number of trouble tickets closed within a certain amount of time. To measure the efficiency of our e-mail infrastructure, we report on the average time it takes for an e-mail message to be delivered. (That measurement also serves as a capacity planning tool. When e-mail delivery time increases, we know we need to increase our e-mail delivery capacity, whether in the form of network bandwidth, hardware or memory.) We report on the percentage of backups that fail, the percentage of IT projects delivered on time and so on.

In the security realm, calculating a balanced scorecard can be somewhat difficult, and alternative metrics are needed. You might think that if the company didn’t get hacked, wasn’t robbed of intellectual property, didn’t suffer a denial-of-service attack or have malicious code such as viruses, worms and Trojan horse programs propagating through its network, then the metrics would be easy — clearly, we’re doing our job. Well, it doesn’t work like that, and I have to be creative in my quarterly metrics.

My predecessor tackled this problem by using metrics that, for all intents and purposes, seemed to have been pulled out of some CISSP book. They provided nothing meaningful. For example, one metric measured the percentage of the infrastructure meeting ISO 17799 compliance. First of all, the company has never been through an ISO 17799 certification. Second, metrics on the elements of the ISO 17799 framework don’t really tell you anything.