Users Begin to Demand Software Usability Tests

Computerworld - WASHINGTON -- The Boeing Co. is changing the way it buys software and is making a product's usability—the ease with which end users can be trained on and operate the product—a fundamental purchasing criterion. It's a move the aerospace giant sees as an essential means of controlling IT costs.

"We simply can't afford to pay for products that cost us a lot of overhead anymore," said Keith Butler, a technical fellow at Boeing's Phantom Works research and development arm. When thousands of end users are involved, design flaws can cost millions of dollars in lost time and productivity, he said.

What's helping Boeing change its purchasing approach is the recent development of a standard for comparing product usability that was spearheaded by the National Institute of Standards and Technology.

Called the Common Industry Format for Usability Test Reports, the standard outlines a format for reporting test conditions and results and gives user companies enough information about a test to replicate it. It's a means for objectively evaluating software, say its backers.

Next month, NIST intends to seek international standards recognition. The standard has already received American National Standards Institute certification.

CIF's success as a purchasing tool depends on whether other companies follow Boeing and make usability a "peer," as Boeing officials put it, of such traditional purchasing criteria as a product's functionality, price and system requirements. If that happens, users say, the standard could have a far-reaching effect in improving the usability of software.

"The real value of CIF, quite honestly, is that if vendors know we are expecting it, meaning large software purchasers, they will focus their attention on usability and hopefully make their products better before they ever come out the door," said Jack Means, superintendent of usability at State Farm Insurance Cos. in Bloomington, Ill.

Boeing played a lead role in the development of CIF after its experience and internal studies showed that usability played a significant role in total cost of ownership. In one pilot of the CIF standard on a widely deployed productivity application, the Chicago-based company said improved product usability had a cost benefit of about $45 million.

Butler said it's much better to have vendors refine an interface design "than to have thousands of end users doing it involuntarily on top of their jobs and then just feeling frustrated."

Spotting Problems Early

Doug Francisco, director of IS architecture at Boeing's commercial airplane division, maintains that CIF will improve the ability of the IT department to spot problems before a product is rolled out to employees. The company has looked at usability in purchasing, "but sometimes we wouldn't discover the inefficiencies of a software product until we brought it in-house," he said.