Iterative vs. waterfall software development: Why don't companies get it?
Bill Walton
February 20, 2004 (Computerworld)
Why, given that iterative and incremental [1] approaches dominate the literature, does business remain so wedded to the waterfall [2] approach to software development? Over the past year or so, I've spent a lot of time reflecting on and researching this question. And to be completely honest, it's been a bit of a personal quest.
A little over a year ago, I wrote an op/ed piece about extreme programming [3] (XP) in which I gave XP credit for some of its technical practices. But in the software development world I currently work in, I said, "That dog don't hunt." But even as I wrote, I knew that, had I been back in the product world where I spent the first half of my career, XP or something very much like it would have been my unquestioned choice of approach. And anyone who knows me will tell you that that sort of contradiction is just not something I can let lie.
It took me longer than expected, but I finally figured it out. My thinking had been conditioned by a linguistic trap. We've used the words design/development to have different meanings depending on context, and that has confused our thinking. In the product life cycle, design/development refers to the set of activities that occurs before a product design is released to manufacturing. But we've used the same words to represent measures of progress within the design/development phase of the product life cycle for internal software (as opposed to retail) -- a product that has a trivial, almost unnoticeable, manufacturing phase. And this has led both customers and software folks into an apples vs. oranges comparison, into an expectation that the coding of software should be as predictable as, and managed similarly to, the manufacturing of hardware. That's an invalid and unattainable expectation.
As I argued in my last column, software development is entirely a design/development phase activity (in the "product life cycle" use of the phrase). You might wonder why I spent an entire column on that argument. There were a couple of reasons. First, that's what it took to convince myself I had it right. Second, I found that my thinking was so thoroughly stuck in that linguistic trap that I needed the argument laid out, so it was there to refer to, to work through its logical implications. And I thought you would, too. So here goes.
Why does an iterative and incremental approach to internal software development make good business sense?
Because ...
- Design/development phase activities in hardware domains are managed using an iterative, incremental approach ("design/development phase" being used in the "product life cycle" sense).
- Software development (i.e., the creation of working software) is entirely a design/development phase activity.
- The ability to turn a software design (i.e., source code) into working software simply by compiling and linking it significantly reduces the costs (in both time and money) of an iteration.
The points above lead me to the conclusion that software development efforts can and should take an even more Iterative and Incremental approach than hardware development efforts, given similar levels of Innovation embodied in the product or project.
Now, I know some of you don't have the benefit of experience in a hardware development environment, so let me share some of my experience.
I had the good fortune to work at Compaq in the early days of PCs, when Innovation still characterized the desktop and portable market. When Compaq introduced its first laptop, for example, we produced prototypes that had the sole purpose to be carried around so the weight and form factor of the design could be validated. But Innovation has largely disappeared from the desktop and portable markets.
So I called a friend who manages an engineering group in HP/Compaq's storage-area network area, a product area still characterized by lots of innovation, to find out if the approach still holds. It does. His group typically sees five to eight full prototypes of a product while it's in design/development, before it's released to manufacturing. And if a new product has boards in it that can be tested in an existing product, you can bet they don't wait. Iteratively and incrementally -- that's how it's done in hardware development.
How iteratively? How incrementally? Well, that depends on the amount and types of Innovation embodied in the product and project. And it depends on the costs of producing versions of the product to get feedback that can be incorporated to make the product better.
Innovation, as I'm using the word, can mean "newness" in any of the product or project dimensions. There are the obvious ones: new product category, new form factor, new technologies, new teams, new tools and so on. And there are less obvious ones. The most important one, for our purposes, relates to the business itself. All business automation projects trade process efficiency for process flexibility. So even in situations where the technologies, the team and the tools are all familiar, it may be that the project embodies innovation of the most complex and critical type possible: the type where the business is new and unfamiliar. To the extent that it does, and to the extent that either the product or project embodies other types of Innovation, we're going to benefit from feedback during the design/development phase.
Innovation drives the need for feedback because the development team's job is not just to produce a product. It's to produce a product that the customer likes, that satisfies a need or desire. The type and amount of "newness" embodied in the product or the project affects a team's ability to communicate and understand, in advance of actually using the product, what characteristics are both necessary and sufficient to produce that satisfaction. Our ability to give feedback on whether or not a product will satisfy our needs depends in large part on the thing we're giving feedback on. I find it helpful to use a scale to describe the utility of the thing. The scale describes what the customer is capable of doing with the (eventual) product at the point of providing feedback:
Imagine -> See -> Handle -> Use.
The further to the right the customer's capability is on this scale, the higher the quality of the feedback.
While the need for feedback is driven by Innovation, our ability to satisfy that need is constrained by costs. The cost constraints associated with getting feedback vary by domain. In some domains, the costs of producing versions of a product on which to get feedback are prohibitive. So proxies must be used. Building architects, for example, begin with sketches of a building's elevation and floor plans. Then they move to scale models. Today, they take clients on virtual walkthroughs. The strategy is to get the most and highest-quality feedback on the design they can, given the cost constraints. In domains like computer hardware, on the other hand, the cost constraints are reduced. Several iterations of working product prototypes are produced. And if a new product has components that can be rolled out incrementally, they take full advantage of the opportunity.
To the extent driven by Innovation, as constrained by costs, all hardware design/development is done iteratively and incrementally.
As I discussed in the last column, software development is entirely a design/development phase activity. The product life cycle begins with a requirements phase wherein a business case is made for proceeding with a project. The product enters the design/development phase where the design is produced and validated as both working as required and being manufacturable. Then the product design is released to manufacturing where copies are produced for sale and/or use. This model holds for all products, hardware or software. But in software development, manufacturing consists of simply copying executable code to one or more production boxes after development is complete. So, except for that copying, all of internal software development is design/development.
And the nature of software dramatically reduces the costs that constrain hardware development efforts. The cost of producing working software, of compiling and linking the source code so that customers can actually use it, is trivial. This truth not only greatly reduces the cost barriers to taking an Iterative approach, but it also has a similar impact on our ability to deliver product functionality Incrementally.
So why do so many businesses limit the quality of the feedback they get prior to "the end" of the project? Why are customers constrained to doing document reviews that allow them only to imagine what they're getting? Hardware design/development teams fight for the chance to get more feedback, to produce "just one more" prototype to validate their designs. Why is it we don't do the same in internal software development? I'm hoping the answer is based in a misunderstanding, one that has its roots in a linguistic trap. I'm hoping that because this is about money. Any business that's using a waterfall approach to developing nontrivial software is wasting money. Potentially, lots of money. I'm going to show you how, how much, and how to avoid it in upcoming columns.
 |   |
| Bill Walton has 15 years of experience in mostly large software development organizations. He now works as an independent IT consultant specializing in requirements engineering and software test management. He can be reached at bill.walton@jstats.com. |
In the shorter term, though, I think it's important to remember that there's a lot of momentum behind the waterfall approach, and it's probably fueled by some additional misunderstandings. A couple of possibilities come right to mind. I give the "But we use the PMI [4] approach" argument the highest probability, especially given the growing number of job postings for project managers I've seen that "prefer Project Management Professional certification." Another possibility is "But we work in a regulated environment. We have to use waterfall."
So I'm going to devote the next couple of columns to addressing these two potential objections. I've had the pleasure of meeting a couple of guys who speak with authority on these topics, and I'm pleased to tell you they've both agreed to help.
Bill Duncan was the primary author of the 1996 and 2000 editions of the Project Management Institute's "A Guide to the PMBOK." I expected Bill to take issue with my argument that software development is best managed using an Iterative and Incremental approach. He didn't. And he has agreed to help clear up some misconceptions about PMBOK and what it "recommends."
Glen Alleman is a vice president in an IT services organization at a company doing a large, multiyear project for the Department of Energy. His teams use both CMMi Level 3 processes and XP-inspired software development methods. And in Glen's experience, there is no conflict.
And then we'll move on to the money. I look forward to your feedback.
Post Feedback
Want to post a response to this article? Send the editor an e-mail message.
READERS RESPOND:
Other factors, too
I very much agree with what Bill Walton says in this article.
While part of the reason so many companies continue to develop software using variations of waterfall is the misconception that the analysis phase of waterfall completes the design and the rest of the process is just non-creative execution of programming skills, there are several other significant contributing factors, including: simple resistance to change; the misconception that because software development utilizes several specialized activities, each activity must be conducted in a distinct isolated phase rather than being integrated into a holistic process; moving to an iterative approach would mean admitting that when we start a software development project, we do not know enough to tightly plan, scope, estimate and control the project (Phillip Armour's The Laws Of Software Process nicely explains why we cannot have this knowledge before we start the project).
-- Steven Gordon, PhD.
Manager
Arizona State University Software Factory
Author responds:
Steven,
Thanks for making time to read the column and for sending feedback. I appreciate both!
WRT "The Laws of Software Process," I haven't been able to make time to read it yet but the description at Amazon.com leads me to think I'd probably have at least one fundamental disagreement with Mr. Armour.
To paraphrase the description there, Mr. Armour takes the position that the problems that have "traditionally plagued" software development result from treating software systems as products "in the classical sense."
I hold the position that just the opposite is true. The problems that plague us, IMO, stem from failing to treat software systems developed for companies' internal use as products. If we did treat them as products, we'd not miss the point that copying executable code to servers is the manufacturing phase of the product lifecycle and, having recognized that, we'd not make the mistake of managing the coding activity as anything other than part of the design/development phase.
I suspect I'd agree with his premise regarding software systems as a medium for conveyance of information, and based on your recommendation I suppose I'll have to push it higher on my reading list. I don't think we need to redefine "the nature and purpose of software" to fix our problems. But I do agree with him that we need to change the way we think about the activity. It's a design/development activity. Not manufacturing or construction.
Thanks again!
-- Bill
Costs can be higher
I don't know if Bill Walton will include this in his calculations but depending on the method used to get feedback, the costs can be a fair bit higher then he indicated.
If he is only referring to the ability to quickly compile a project and throw it into a beta test environment, then yes, he is correct in his evaluation. The cost in this case is quite trivial because every developer worth his salt regularly compiles his work and does his own testing. To give a client access to this working copy of the program would often be trivial. A fair amount of feedback can be received using this method but it will be FAR from complete.
The testing can't be anywhere near complete because the foundation of nearly all business applications is the data and data structure. When developing or reworking a business application, significant changes to the data structure are generally required. Those changes will not exist on the production data. This means that anyone testing the program would not be able to test it by doing their normal work. Any serious functionality testing would require a significant commitment in time from those departments that will eventually be using the new software. That kind of commitment is harder to come by then might be expected.
If the new software can be pointed at production data then you have another issue. Are you really sure that the new software will not mess up business critical data? To do this safely would require the addition of some real labor-intensive auditing by an individual skilled (and expensive) enough to catch any problems before they hit production.
There is also the issue of what to do in the event that an important functionality point is broken. How difficult is it to roll back the software update? This is also not as easy as many people think, especially if data structure changes were involved. If the software update is managed badly, it may involve entire departments being forced to deal with a crippled application, or an extended break while the old version is restored.
Applications are not like appliances. If an appliance does not work as expected, you generally still have the old version so that you can continue to do your work. If you roll out a new version of a business application it generally replaces the older version. Going back often requires a significant amount of work and embarrassment.
Simply put, this means that any attempt at rolling out an incremental update requires a very serious investment in testing and auditing. This is not a minor expense and depending on the size of the application it may take a significant amount of time. This testing is primarily to ensure that all the pre-existing functionality is still working. Most users only like being a beta tester if they can be assured that at the end of the day they can still do all their original work. If you make the mistake of breaking your clients' work environment too often then they may be the ones calling for your head or putting the brakes to your development efforts.
-- Anthony van Orizande
Technical analyst
The Platinum Group
Author responds:
Hi Anthony,
Thanks for making time to read the column and for your thoughtful response. It's not very far off the arguments I used to make regarding iterative and incremental Development, and XP in particular. And then I realized that I had been making a fundamental mistake regarding the true nature of software development.
Your points are well taken, given the set of circumstances they seem to imply. But, IMO, those circumstances do not reflect good testing practice.
For example, your comments about "data and data structure" seem to me to imply the lack of a well-designed test system or set of systems. Given the cost of hardware and software today....
Your comments about "significant commitment of time" from the user community give me pause on a couple of fronts. First, if the user community is not committed to devote the time to ensure the system meets their business needs, then someone needs to ask whether or not the application is really needed. Second, if the testing group isn't automating the user tests, the user community has an honest grievance against them for wasting their time.
Lastly, with respect to rollbacks, I have to wonder if the development group has made an adequate investment in configuration management and version control.
I very much agree with your comment about "breaking your clients' work environment too often." In my experience, if the development organization hasn't invested in test systems, automated testing, and CM, they're wasting their customers' time and money and heads should roll.
Thanks again for making the time to read the column and for your feedback. I hope you'll give some thought to my response. Feel free to post additional comments if you'd like to converse about it, either here or directly to me off-line.
-- Bill
How to iterate in-house support processes?
The software projects I am familiar with are for in-house support of business processes. These are analogous to retooling a mass production line in a factory without interrupting the work. Many of the issues that Anthony van Orizande mentions in his note I will corroborate.
The effort in user acceptance testing, operational readiness testing, migration and controlled rollout is comparable to the software development effort and internal testing. It typically involves many organizations across the enterprise. How to meaningfully iterate in these circumstances is an interesting problem.
-- Arun Gupta
Systems analyst
Large Telecomm Company
Best path is in the middle
I greatly enjoyed Bill's article it is well written and really got me thinking. I have been in IT since the early '80s and have done a lot of development. I have worked for really small companies (7 people) and some really big ones (200K people). Everything from "seat of the pants" development to very structured waterfall and iterative. There are risks to both approaches and I really think the best path is somewhere in the middle.
Waterfall development emphasizes requirements management and solid software architecture, while iterative development emphasizes constant feedback, and quicker, smaller releases. The problem with true waterfall is that the project takes on a life of its own and can be hard to change once it starts while run away iterative can put a lot of pieces together quickly but they don't always fit together and often don't scale well. In addition, iteration without architecture shortens the life span of the product and makes it hard to enhance and update.
I agree with Bill, in software development, iterative just makes sense. It has been my experience that the more "show and tell" development has with the customer the more likely the finished product will fit the business need. We have used this approach even in waterfall development projects. If the first look the customer gets is at the finished product chances are really good it will take several "maintenance releases" to fix it so the customer can really use it.
I think the middle ground is to put greater emphasis on good requirements management, without creating a bureaucracy and good solid component based architecture for the iterations. Both of these practices fit very well in an iterative development cycle, provide some discipline without slowing the iterations, but provide for a more stable product that can be improved and will lengthen the life span of the software.
Thanks for a good article.
-- Dick Weldon
Software Engineering Processes Technical Leader
Look forward to next article
Bill Walton's article certainly resonates with me.
I've worked on software languages and tools, embedded systems, large banking systems and hardware.
The first use of a system -- i.e. the first iteration -- invariably highlights new design-level issues that are missed by the sole application of diligent forethought. Hardware managers get this point, software managers often don't get it. Too much software is released before even the first iteration is complete.
As for the expense of testing and releasing software in a highly interdependent, live environment -- that's a design issue. A solution must be stretched to fit neatly over all aspects of its environment. If the problem is important, it should be accorded a place in the design cycle. In the hardware world, testability is an important issue and people are employed full-time to ensure that testability is designed-in.
Improving and/or shrinking the design cycle through iteration is important. It works, even in big-iron shops. In developing automation for big banks, our first software iteration showed that the most obvious approach would have resulted in untenable amounts of testing and re-certification. Without that first iteration, we would have continued on the expensive path for too long. Further iterations caused us to come up with a design that was amazingly simple and effective -- and unforeseeable.
I especially like Mr. Walton's "Imagine -> See -> Handle -> Use" scale. It illustrates a benefit available to software designers - one can quickly and repeatedly skip to the most effective end of the scale (Use).
I look forward to reading the next articles in this series.
--Paul Tarvydas
Development cycle never ends
If we accept the fact that no software solution is ever perfect, and this is well-supported by the continuous stream of bug fixes, patches, enhancements and new releases of all major software products, then we can argue that the development cycle never actually ends. IMO what actually happens is that in order to get the budget approved a large portion of the development effort is shoe-horned into an overly large first iteration. Only this single iteration appears on the books as development, while the rest of the iterative development is carried forward as maintenance and support.
The sooner that management accept that this is what happens, the sooner we will be able to reap the benefits of XP. An indirect benefit of this will be that we can start to set realistic expectations for our users and markets.
-- Mike Kenney
Want to increase productivity? Delay!
When we use the word 'development,' as in new product development or software development, we are referring to a knowledge-generating process.
It is a great mistake to apply practices that work in knowledge-replication processes to knowledge-generation processes, as Bill Walton has pointed out. It is true that knowledge-creation processes need discipline.
For example, core software development disciplines would be naming conventions and coding standards, a configuration management system, an automated build process, a suite of automated unit tests that are built and maintained as part of the code, daily build/integrate/test cycles, acceptance testing integrated into the development process, and usability testing immediately after the features are implemented. Assuring that these disciplines are in place is fundamental to the smooth flow of any software development process.
However, when we are developing a new product or a new software system, the fundamental thing we are doing is discovering what needs to be in the system in order to delight the customer. There are two basic processes for improving the knowledge creation process: short, frequent learning cycles and delayed commitment. It is well known that repeated cycles of discovery are the best way to generate knowledge. However, we must also defer decisions so as to give ourselves time to generate the knowledge necessary to make the best decisions.
Delaying decisions, particularly scope decisions, can lead to huge productivity increases. How can this be? Studies show that most of the features we put into software systems are never going to be used. When we freeze scope early, we encourage our customers, who don't really know what they want, to ask for everything they can imagine. If instead we delay commitment on scope until we are well into the knowledge generating process, we end up reducing scope down to the minimum set that is really going to pay off.
-- Mary Poppendieck,
author, Lean Software Development: An Agile Toolkit
EDITOR'S NOTES:
[1] The iterative and incremental approaches involve a number of short cycles in which steps such as requirements gathering, coding, testing and deployment, are conducted to produce small parts of the final project. The software system grows incrementally, and user feedback can be used throughout the process.
[2] The waterfall philosophy is a strictly sequential approach in which a project is completed in a series of steps, such as analysis, design, coding, testing and deployment. Each step, such as requirements gathering, is undertaken only once and must be completed and verified before the next phase.
[3] Extreme programming is a software development approach built around rapid iterations, an emphasis on code writing and working closely with end users to achieve business results. The 12 basic practices of XP include continual testing and the idea that programmers should work in pairs. (See Computerworld QuickStudy.)
[4] Project Management Institute methodology involves recommended best practices that use a cycle of processes -- initiating, planning, executing, controlling and closing -- to manage a project's scope, time, costs, risks and so on.