Automating the 40 Monkeys

Computerworld - Some call it "the 40 monkeys test." Others prefer the more diplomatic "Friday night pizza party." Whatever the name, it's the traditional method for predeployment software testing: coercing a few dozen employees to come in after hours and beta-test an application to uncover bugs and performance problems before it's rolled out to customers and employees.

Unfortunately, it doesn't always work. "You ask a guy to run a hundred tests, he's eventually going to miss something," says Arthur Povlot, business development manager at Tescom Software Systems Testing Ltd. in London.

It's also virtually unrepeatable. "If 50 people find problems, which I fix and then want to do another test ... well, it just isn't easy to get everyone back again," says Dave Wollin, managing group director of IT at AXA Financial Inc. The New York-based finance and insurance company used Mercury Interactive Corp.'s testing tool before deploying a Web application to field agents.

The desire for repeatability and accuracy is one reason organizations are moving to automate testing, both to uncover bugs and ensure that they meet performance standards.

Melissa Webster, research director at IDC in Framingham, Mass., says worldwide sales of automated testing tools rose by 8.5% in 2003 and are expected to grow at an annual compound rate of 9.3% through 2007. "This sector has recovered faster than the overall IT market," she says.

The increasingly integrated nature of IT systems is another factor pushing companies toward automated testing. Human testers can't always accurately re-create the flow of transactions between applications. That's why Varian Medical Systems Inc., a Palo Alto, Calif.-based maker of medical devices, opted to purchase an automated functional testing tool, SilkTest, from Segue Software Inc. last year.

Image Credit: Richard Borge

Vendors such as Mercury, Segue Software, Compuware Corp., IBM's Rational unit, RadView Software Ltd. and Empirix Inc. offer products for testing client/server and Web-based applications, as well as mainframe and other legacy systems. Most of these products provide an interface for programming test scripts, as well as a capture-replay tool or visual scripting option for nontechnical users. Also included are reporting tools, and many vendors are starting to offer diagnostics capabilities, as well as monitoring tools for tracking application performance after deployment.