When taking stock of performance, beware the benchmarking blues

Rare is the business or IT department that runs without performance benchmarks. Done well, they provide a basis for sound decision making. Yet, badly crafted benchmarks are remarkably common. Even worse, a poorly crafted benchmark can intuitively look like a good one, thereby discouraging a deeper benchmarking evaluation that could deliver superior results.

A surefire way to get funding for green IT projects

When misapplied, or employed by the unskilled, benchmarking can do far more harm than good informing decision-making.  Given the ubiquity of benchmarking studies in many industries, it is surprising how little this is appreciated. So how can you know if the benchmark you are using is a good tool or a harmful one? Here is the process I use.

In the most abstract sense, benchmarking simply quantifies the ratio of commercial output to some input to allow normalized comparisons between peers.  Thus, if my factory produces ten widgets per person per day, and my peer produces eleven, we can reasonably, though not unequivocally, conclude that my peer is more “productive,” or conversely, that I am less than maximally efficient and should realistically be able to eke out higher productivity from my staff.

Benchmarking studies also identify the reasonable limits – both high and low – of expected performance.  These limits can otherwise be difficult to define or articulate – indeed, they may not meaningfully exist without the context provided by comparison with peers.  In this sense, benchmarking also defines the universe of reasonable expectations.

Benchmarks can also easily be used as a carrot, rather than a stick. One mid-west facility management corporation developed a highly sophisticated benchmarking tool that allowed for quarterly performance contests between facility managers.  The contest was embraced in the spirit of fun.  More significantly, it successfully encouraged ongoing efficiency improvements and operating cost savings across their portfolio of properties (the benchmark was sufficiently sophisticated that each facility “started fresh” each quarter, so that no one had cause to feel uncompetitive and to opt out of the program.)

When All you Have in a Hammer

It is important to keep in mind that benchmarks can drive wrenching decisions for the enterprise, particularly those pertaining to budgeting and staffing.  Because of this, it is incumbent upon senior managers to carefully assess the performance metrics that underpin a benchmarking exercise.  A metric chosen cavalierly, or accepted based upon consultant recommendations or “industry standards,” can lead to improper efficiency comparisons that may encourage decisions that are actually detrimental to the competitive posture of the organization.

A simple thought experiment demonstrates why this is so. 

Corporations often evaluate the relative performance of their facilities based upon a few surprisingly simple metrics, such as these actual examples:

Energy per square foot per yearFull-time equivalents (FTEs) per square footCost per square foot per year

On the face of it, these metrics seem wholly acceptable.  They are simple, they report data that pertains directly to operating costs, and they reflect standard benchmarking protocols employed in industry, government and elsewhere.

But consider two manufacturing facilities.  Both are 250,000 square feet, both employ 1,200 people, both consume 30M kilowatt-hours of electricity per year, and both have non-payroll operating costs of $4M per year.

We observe that the corporate benchmark will obviously score these two facilities identically.  However, there is a signal difference between the two locations I neglected to mention.  Factory A produces 200 widgets per day, whereas Factory B produces 250.  This leads to an inescapable and essential conclusion about the indicated corporate benchmarks: They are utterly unable to identify relative facility “efficiency” based on the benchmarking metrics that were chosen.  Assuming, of course, that by efficiency we mean “lower cost to provide a product or service.”

One might argue that the metrics selected, such as FTE per square foot, are so simplistic as to be nothing more than a straw man.  Surprisingly, however, I have seen billion-dollar enterprises use exactly this metric in developing staffing target profiles.  In other words, while this might look like a straw man, the sad reality is that inappropriate metric ratios are rather common. Energy use is also very commonly expressed on a “per-square-foot” basis without regard to production density.

It is also important to recognize that the inability of these metrics to identify the more efficient of two facilities is not a “trick” of presentation or a spurious finding that is not significant in general.  Rather, this simple thought experiment demonstrates that the selected metrics are fundamentally flawed and incapable of quantifying facility performance.  This cannot be overstated – the enterprise is managing to a metric that reports data of only tangential significance to the bottom line.

Clearly, a proper metric would examine the output product relative to an expended resource, such as “widgets produced per person per day” or “widgets produced per annual operating cost dollar” or something similar.  If we stop to think about it, how big or small the factory itself happens to be is a rather arbitrary matter.  Yet, per-square-foot metrics are employed everywhere.

Where’s the Harm?

The danger in selecting an incorrect or artificial benchmarking metric goes beyond its inability to accurately capture the relative performance of a facility or department.  If hiring and firing decisions are predicated upon inappropriate metrics, the enterprise can hurt itself badly by gutting effective facilities or departments that contribute substantially to profitability but that benchmark poorly.

For example, if one production line consumes 25% more energy than another, but produces 30% more widgets, it would benchmark poorly against a per-square-foot metric.  Even worse, this metric could be used to justify targeting the more efficient production line first should layoffs be required – exactly the wrong result.  Compounding the damage, this facility or department will be understaffed and unprepared to seize the opportunity when market conditions eventually rebound.  And the more efficient employees who were laid off will be far less willing to return to the enterprise when the marketplace rebounds and their skills are in demand.

Note too that benchmarks are indicators at best, not targets.  Physical or ergonomic constraints may make it impossible for one entity to perform equivalently with a peer, even if all other factors are precisely the same.  So they are best used to compare a facility or department against itself over time. Benchmarking lends itself to abuse when it is used to compare peers, particularly in the short run, a fact that is often “overlooked” for the sake of expediency in decision making.

Benchmarking Blues

Improper benchmarking can do more than target the wrong department or facility for inefficient operation.  Once targeted, effective facility managers and department heads must dedicate precious time and resources in a rear-guard action to protect their staff and operational reputation. This can be draining and frustrating, particularly because such managers typically have a very secure understanding of their role and effectiveness within the organization.  And obviously, time spent defending against a bad benchmark is time not spent improving the bottom line.

For this very reason, it is often counterproductive to develop benchmarks without consultation of middle level managers and even staff. Far from “gaming” the metric development, employees will often uncover essential inputs to production that the benchmarking consultant will have no awareness of. With the knowledge that the metrics are based upon a rational analysis of inputs and outputs, there is a much greater incentive to trust and use the benchmarking metrics for their intended purpose – to improve efficiency and to perform well relative to peers.  In contrast, it is difficult to secure staff buy-in for a poor benchmark that is “handed down” from on high.

Where’s the Solution?

With these simple steps, the humble benchmark can be turned into a vastly more effective decision-making tool.

Turner is the utility manager for Brigham and Women's Hospital in Boston.

Read more about data center in Network World's Data Center section.

This story, "When taking stock of performance, beware the benchmarking blues" was originally published by Network World.

6 tips for scaling up team collaboration tools
Shop Tech Products at Amazon