Why, given that iterative and incremental [1] approaches dominate the literature, does business remain so wedded to the waterfall [2] approach to software development? Over the past year or so, I've spent a lot of time reflecting on and researching this question. And to be completely honest, it's been a bit of a personal quest.
A little over a year ago, I wrote an op/ed piece about extreme programming [3] (XP) in which I gave XP credit for some of its technical practices. But in the software development world I currently work in, I said, "That dog don't hunt." But even as I wrote, I knew that, had I been back in the product world where I spent the first half of my career, XP or something very much like it would have been my unquestioned choice of approach. And anyone who knows me will tell you that that sort of contradiction is just not something I can let lie.
It took me longer than expected, but I finally figured it out. My thinking had been conditioned by a linguistic trap. We've used the words design/development to have different meanings depending on context, and that has confused our thinking. In the product life cycle, design/development refers to the set of activities that occurs before a product design is released to manufacturing. But we've used the same words to represent measures of progress within the design/development phase of the product life cycle for internal software (as opposed to retail) -- a product that has a trivial, almost unnoticeable, manufacturing phase. And this has led both customers and software folks into an apples vs. oranges comparison, into an expectation that the coding of software should be as predictable as, and managed similarly to, the manufacturing of hardware. That's an invalid and unattainable expectation.
As I argued in my last column, software development is entirely a design/development phase activity (in the "product life cycle" use of the phrase). You might wonder why I spent an entire column on that argument. There were a couple of reasons. First, that's what it took to convince myself I had it right. Second, I found that my thinking was so thoroughly stuck in that linguistic trap that I needed the argument laid out, so it was there to refer to, to work through its logical implications. And I thought you would, too. So here goes.
Why does an iterative and incremental approach to internal software development make good business sense?
Because ...
- Design/development phase activities in hardware domains are managed using an iterative, incremental approach ("design/development phase" being used in the "product life cycle" sense).
- Software development (i.e., the creation of working software) is entirely a design/development phase activity.
- The ability to turn a software design (i.e., source code) into working software simply by compiling and linking it significantly reduces the costs (in both time and money) of an iteration.
The points above lead me to the conclusion that software development efforts can and should take an even more Iterative and Incremental approach than hardware development efforts, given similar levels of Innovation embodied in the product or project.
Now, I know some of you don't have the benefit of experience in a hardware development environment, so let me share some of my experience.
I had the good fortune to work at Compaq in the early days of PCs, when Innovation still characterized the desktop and portable market. When Compaq introduced its first laptop, for example, we produced prototypes that had the sole purpose to be carried around so the weight and form factor of the design could be validated. But Innovation has largely disappeared from the desktop and portable markets.
So I called a friend who manages an engineering group in HP/Compaq's storage-area network area, a product area still characterized by lots of innovation, to find out if the approach still holds. It does. His group typically sees five to eight full prototypes of a product while it's in design/development, before it's released to manufacturing. And if a new product has boards in it that can be tested in an existing product, you can bet they don't wait. Iteratively and incrementally -- that's how it's done in hardware development.
How iteratively? How incrementally? Well, that depends on the amount and types of Innovation embodied in the product and project. And it depends on the costs of producing versions of the product to get feedback that can be incorporated to make the product better.
Innovation, as I'm using the word, can mean "newness" in any of the product or project dimensions. There are the obvious ones: new product category, new form factor, new technologies, new teams, new tools and so on. And there are less obvious ones. The most important one, for our purposes, relates to the business itself. All business automation projects trade process efficiency for process flexibility. So even in situations where the technologies, the team and the tools are all familiar, it may be that the project embodies innovation of the most complex and critical type possible: the type where the business is new and unfamiliar. To the extent that it does, and to the extent that either the product or project embodies other types of Innovation, we're going to benefit from feedback during the design/development phase.
Innovation drives the need for feedback because the development team's job is not just to produce a product. It's to produce a product that the customer likes, that satisfies a need or desire. The type and amount of "newness" embodied in the product or the project affects a team's ability to communicate and understand, in advance of actually using the product, what characteristics are both necessary and sufficient to produce that satisfaction. Our ability to give feedback on whether or not a product will satisfy our needs depends in large part on the thing we're giving feedback on. I find it helpful to use a scale to describe the utility of the thing. The scale describes what the customer is capable of doing with the (eventual) product at the point of providing feedback:
Imagine -> See -> Handle -> Use.
The further to the right the customer's capability is on this scale, the higher the quality of the feedback.
While the need for feedback is driven by Innovation, our ability to satisfy that need is constrained by costs. The cost constraints associated with getting feedback vary by domain. In some domains, the costs of producing versions of a product on which to get feedback are prohibitive. So proxies must be used. Building architects, for example, begin with sketches of a building's elevation and floor plans. Then they move to scale models. Today, they take clients on virtual walkthroughs. The strategy is to get the most and highest-quality feedback on the design they can, given the cost constraints. In domains like computer hardware, on the other hand, the cost constraints are reduced. Several iterations of working product prototypes are produced. And if a new product has components that can be rolled out incrementally, they take full advantage of the opportunity.
To the extent driven by Innovation, as constrained by costs, all hardware design/development is done iteratively and incrementally.
As I discussed in the last column, software development is entirely a design/development phase activity. The product life cycle begins with a requirements phase wherein a business case is made for proceeding with a project. The product enters the design/development phase where the design is produced and validated as both working as required and being manufacturable. Then the product design is released to manufacturing where copies are produced for sale and/or use. This model holds for all products, hardware or software. But in software development, manufacturing consists of simply copying executable code to one or more production boxes after development is complete. So, except for that copying, all of internal software development is design/development.
And the nature of software dramatically reduces the costs that constrain hardware development efforts. The cost of producing working software, of compiling and linking the source code so that customers can actually use it, is trivial. This truth not only greatly reduces the cost barriers to taking an Iterative approach, but it also has a similar impact on our ability to deliver product functionality Incrementally.
So why do so many businesses limit the quality of the feedback they get prior to "the end" of the project? Why are customers constrained to doing document reviews that allow them only to imagine what they're getting? Hardware design/development teams fight for the chance to get more feedback, to produce "just one more" prototype to validate their designs. Why is it we don't do the same in internal software development? I'm hoping the answer is based in a misunderstanding, one that has its roots in a linguistic trap. I'm hoping that because this is about money. Any business that's using a waterfall approach to developing nontrivial software is wasting money. Potentially, lots of money. I'm going to show you how, how much, and how to avoid it in upcoming columns.
 |   |
| Bill Walton has 15 years of experience in mostly large software development organizations. He now works as an independent IT consultant specializing in requirements engineering and software test management. He can be reached at bill.walton@jstats.com. |
In the shorter term, though, I think it's important to remember that there's a lot of momentum behind the waterfall approach, and it's probably fueled by some additional misunderstandings. A couple of possibilities come right to mind. I give the "But we use the PMI [4] approach" argument the highest probability, especially given the growing number of job postings for project managers I've seen that "prefer Project Management Professional certification." Another possibility is "But we work in a regulated environment. We have to use waterfall."
So I'm going to devote the next couple of columns to addressing these two potential objections. I've had the pleasure of meeting a couple of guys who speak with authority on these topics, and I'm pleased to tell you they've both agreed to help.
Bill Duncan was the primary author of the 1996 and 2000 editions of the Project Management Institute's "A Guide to the PMBOK." I expected Bill to take issue with my argument that software development is best managed using an Iterative and Incremental approach. He didn't. And he has agreed to help clear up some misconceptions about PMBOK and what it "recommends."
Glen Alleman is a vice president in an IT services organization at a company doing a large, multiyear project for the Department of Energy. His teams use both CMMi Level 3 processes and XP-inspired software development methods. And in Glen's experience, there is no conflict.
And then we'll move on to the money. I look forward to your feedback.
Post Feedback
Want to post a response to this article? Send the editor an e-mail message.
READERS RESPOND:
Other factors, too
I very much agree with what Bill Walton says in this article.
While part of the reason so many companies continue to develop software using variations of waterfall is the misconception that the analysis phase of waterfall completes the design and the rest of the process is just non-creative execution of programming skills, there are several other significant contributing factors, including: simple resistance to change; the misconception that because software development utilizes several specialized activities, each activity must be conducted in a distinct isolated phase rather than being integrated into a holistic process; moving to an iterative approach would mean admitting that when we start a software development project, we do not know enough to tightly plan, scope, estimate and control the project (Phillip Armour's The Laws Of Software Process nicely explains why we cannot have this knowledge before we start the project).