The Testing Time Bomb

Computerworld - If your company develops software, you are sitting on a time bomb. Sooner or later, your software is going to explode.

Here's why. Software is a constant accumulation of functionality over time, requested by various users for different reasons and implemented by a stream of developers using their own styles, some of whom are long gone, along with the knowledge and reasoning they applied. The result is an inventory of logic and behavior that may or may not even be known, let alone understood.

Add to this volatile mix the fact that every time a change is introduced, it has the possibility of unintended and therefore unforeseen effects on other areas.

Here's the detonator: If a new release adds just 10% to the existing code base, the amount of testing that's required to be sure the new stuff works, and everything that used to work still does, is 110% of the previous release -- yet the time and resources applied to testing all of this functionality are at best flat release over release, and more likely declining. And, it was never more than a fraction of the development effort to begin with.

Do the math. It would require a ratio of 11:1 testers to developers for this release, yet most companies feel blessed to have a 1:5 ratio.

As you can see, the math simply doesn't work. Or, as they say in Texas, that dog won't hunt. There is no way the testers can possibly verify the entire inventory of functionality (even if they know what it is, which they don't), and so they are forced into making trade-offs, usually focusing on what they know has changed and hoping that what they don't know won't hurt them.

Yet the facts say their hopes are unfounded. A senior development manager at a large financial services firm recently pointed out that a postmortem analysis of production software defects showed 70% of problems arose when functionality that used to work was broken.

But what to do?

Semantics Sometimes Matter

If you ask most testers to choose between testing new functionality or old -- and they are always forced to make this choice -- they will choose the new. Why? For the obvious reason that it's new and therefore has more risk. And, from a development point of view, this is true.

But from an operational point of view, the opposite is true: The greatest risk is that something that used to work quits working, because that's functionality users are already relying on. If