September 23, 2005
(Computerworld)
One of the supposed truisms of software development is that out of three attributes -- fast, good and cheap -- you can pick only two. Turns out the data from a recent research project says otherwise. Spending more money makes little difference in speed and actually reduces quality.
Quantitative Software Management (QSM) maintains the most comprehensive database of software development project metrics that I know of, including over 7,200 projects spanning almost 30 years. Its data comes from its own projects as well as the repository of an estimation product called SLIM. These projects are 58% IT projects (finance, human resources, etc.), 28% engineering (scientific, process control, etc.) and 14% real-time embedded systems (firmware, hardware controllers, etc.).
Although I haven't used it yet, SLIM first came to my attention when a customer told me in total amazement that her planned six-month project was projected to be three months late by the SLIM model. Her management dismissed this projection, of course, but she quietly kept track of the project anyway ... only to discover it accurately predicted the final delivery within a two-day margin! Accurate estimating is one of a Holy Grail of software development, and what I like about the SLIM approach is that it's based on experience, not theory.
Anyway, QSM recently repeated a study it did in 1996 to measure the effect of adding resources to compress a delivery schedule. This is a common reaction to being in a hurry or panicking over delays: Think fast and good but not cheap.
It studied small teams (five or fewer resources) and large teams (20 or more resources) developing between 10,000 and 200,000 lines of code and took a number of measurements including lines of code developed, schedule progress and defect creation. What it found wasn't a surprise, because the results simply confirmed the 1996 findings, but they should surprise you.
The difference between a large team (29 resources) and a small one (2.5) developing 40,000 lines of code was only about 12 calendar days earlier delivery! It took 191 person-months of resources for the large team and only 40 person-months for the small team. The difference at $12,000 per person month was $1.8 million. Now of course that results in a business decision: If 12 days of product availability is worth more than $1.8 million, it may be worth the investment. But it is worth doing the math.
One reason for the nonlinear productivity was explained by the fact that large teams produced more than six times as many defects as small ones, which took additional time to discover, fix and retest. So adding 11.6 times as many resources produced an increase of six times as many defects ... pretty close to a ratio of 2:1. In case this didn't sink in, let me repeat it: There were six times as many defects in the same amount of code written by almost 12 times as many programmers.
In other words, adding resources was not that much faster, the software produced wasn't good, and the project wasn't cheap. Only one out of three.
Lest you be tempted to explain these results away by the added complexity of the latest software technology, you should read the original classic in this area, The Mythical Man-Month by Frederick P. Brooks, written 30 ago. Some try to dismiss the relevance of the 30th anniversary edition of this book by waving the banner of "virtual programming, faster machines and bigger memories" as though the phenomenon observed in the book -- that is, the nonlinear economics of resource scalability -- can be explained away by hardware.
Yet the undeniable fact is that the same results observed in the book have been statistically confirmed by QSM 10 years ago and now again this year. Simply put, adding resources has a declining marginal benefit to schedule compression and an increasing negative impact on quality.
What does this tell us?
Clearly, the problem has nothing to do with software or hardware and everything to do with individuals and group behavior. More complex software doesn't account for it, and faster hardware with more memory won't fix it. The only consistent explanation is that as you add people, efficiency and quality decrease.
The drop in efficiency is obvious to me from my own experience. A team of four or five people can congregate in a small room and work through issues on a whiteboard, while a team of 20 requires collaboration software (and the requisite infrastructure and administrator to manage it), time to gather and synthesize information, and more time to coordinate a meeting, have the meeting and discuss it. A one-hour meeting for five people is, of course, five person-hours. With 20 people in the room ... well, you get it.
But it's the drop in quality that's more interesting. I have my own theory about this, which is simple: The smaller you break a problem down, the less you see the whole picture. So if 20 people write 40,000 lines of code, each person only sees 2,000 lines, whereas five people would see 8,000 each. It's not until you stitch these smaller pieces together that you find the problems that arise from an incomplete understanding.
But whatever the whole story is, and however it's explained, the facts stubbornly resist the ravages of time and technology. When it comes to developing software, haste makes waste.