Computerworld - Why, given that iterative and incremental [1] approaches dominate the literature, does business remain so wedded to the waterfall [2] approach to software development? Over the past year or so, I've spent a lot of time reflecting on and researching this question. And to be completely honest, it's been a bit of a personal quest.
A little over a year ago, I wrote an op/ed piece about extreme programming [3] (XP) in which I gave XP credit for some of its technical practices. But in the software development world I currently work in, I said, "That dog don't hunt." But even as I wrote, I knew that, had I been back in the product world where I spent the first half of my career, XP or something very much like it would have been my unquestioned choice of approach. And anyone who knows me will tell you that that sort of contradiction is just not something I can let lie.
It took me longer than expected, but I finally figured it out. My thinking had been conditioned by a linguistic trap. We've used the words design/development to have different meanings depending on context, and that has confused our thinking. In the product life cycle, design/development refers to the set of activities that occurs before a product design is released to manufacturing. But we've used the same words to represent measures of progress within the design/development phase of the product life cycle for internal software (as opposed to retail) -- a product that has a trivial, almost unnoticeable, manufacturing phase. And this has led both customers and software folks into an apples vs. oranges comparison, into an expectation that the coding of software should be as predictable as, and managed similarly to, the manufacturing of hardware. That's an invalid and unattainable expectation.
As I argued in my last column, software development is entirely a design/development phase activity (in the "product life cycle" use of the phrase). You might wonder why I spent an entire column on that argument. There were a couple of reasons. First, that's what it took to convince myself I had it right. Second, I found that my thinking was so thoroughly stuck in that linguistic trap that I needed the argument laid out, so it was there to refer to, to work through its logical implications. And I thought you would, too. So here goes.
Why does an iterative and incremental approach to internal software development make good business sense?
Because ...
Design/development phase activities in hardware domains are managed using an iterative, incremental approach ("design/development phase" being used in the "product life cycle" sense).
Software development (i.e., the creation of working software) is entirely a design/development phase activity.
The ability to turn a software design (i.e., source code) into working software simply by compiling and linking it significantly reduces the costs (in both time and money) of an iteration.
The points above lead me to the conclusion that software development efforts can and should take an even more Iterative and Incremental approach than hardware development efforts, given similar levels of Innovation embodied in the product or project.
Now, I know some of you don't have the benefit of experience in a hardware development environment, so let me share some of my experience.
I had the good fortune to work at Compaq in the early days of PCs, when Innovation still characterized the desktop and portable market. When Compaq introduced its first laptop, for example, we produced prototypes that had the sole purpose to be carried around so the weight and form factor of the design could be validated. But Innovation has largely disappeared from the desktop and portable markets.
So I called a friend who manages an engineering group in HP/Compaq's storage-area network area, a product area still characterized by lots of innovation, to find out if the approach still holds. It does. His group typically sees five to eight full prototypes of a product while it's in design/development, before it's released to manufacturing. And if a new product has boards in it that can be tested in an existing product, you can bet they don't wait. Iteratively and incrementally -- that's how it's done in hardware development.
How iteratively? How incrementally? Well, that depends on the amount and types of Innovation embodied in the product and project. And it depends on the costs of producing versions of the product to get feedback that can be incorporated to make the product better.
Innovation, as I'm using the word, can mean "newness" in any of the product or project dimensions. There are the obvious ones: new product category, new form factor, new technologies, new teams, new tools and so on. And there are less obvious ones. The most important one, for our purposes, relates to the business itself. All business automation projects trade process efficiency for process flexibility. So even in situations where the technologies, the team and the tools are all familiar, it may be that the project embodies innovation of the most complex and critical type possible: the type where the business is new and unfamiliar. To the extent that it does, and to the extent that either the product or project embodies other types of Innovation, we're going to benefit from feedback during the design/development phase.
Innovation drives the need for feedback because the development team's job is not just to produce a product. It's to produce a product that the customer likes, that satisfies a need or desire. The type and amount of "newness" embodied in the product or the project affects a team's ability to communicate and understand, in advance of actually using the product, what characteristics are both necessary and sufficient to produce that satisfaction. Our ability to give feedback on whether or not a product will satisfy our needs depends in large part on the thing we're giving feedback on. I find it helpful to use a scale to describe the utility of the thing. The scale describes what the customer is capable of doing with the (eventual) product at the point of providing feedback:
Imagine -> See -> Handle -> Use.
The further to the right the customer's capability is on this scale, the higher the quality of the feedback.
While the need for feedback is driven by Innovation, our ability to satisfy that need is constrained by costs. The cost constraints associated with getting feedback vary by domain. In some domains, the costs of producing versions of a product on which to get feedback are prohibitive. So proxies must be used. Building architects, for example, begin with sketches of a building's elevation and floor plans. Then they move to scale models. Today, they take clients on virtual walkthroughs. The strategy is to get the most and highest-quality feedback on the design they can, given the cost constraints. In domains like computer hardware, on the other hand, the cost constraints are reduced. Several iterations of working product prototypes are produced. And if a new product has components that can be rolled out incrementally, they take full advantage of the opportunity.
To the extent driven by Innovation, as constrained by costs, all hardware design/development is done iteratively and incrementally.
As I discussed in the last column, software development is entirely a design/development phase activity. The product life cycle begins with a requirements phase wherein a business case is made for proceeding with a project. The product enters the design/development phase where the design is produced and validated as both working as required and being manufacturable. Then the product design is released to manufacturing where copies are produced for sale and/or use. This model holds for all products, hardware or software. But in software development, manufacturing consists of simply copying executable code to one or more production boxes after development is complete. So, except for that copying, all of internal software development is design/development.
And the nature of software dramatically reduces the costs that constrain hardware development efforts. The cost of producing working software, of compiling and linking the source code so that customers can actually use it, is trivial. This truth not only greatly reduces the cost barriers to taking an Iterative approach, but it also has a similar impact on our ability to deliver product functionality Incrementally.
So why do so many businesses limit the quality of the feedback they get prior to "the end" of the project? Why are customers constrained to doing document reviews that allow them only to imagine what they're getting? Hardware design/development teams fight for the chance to get more feedback, to produce "just one more" prototype to validate their designs. Why is it we don't do the same in internal software development? I'm hoping the answer is based in a misunderstanding, one that has its roots in a linguistic trap. I'm hoping that because this is about money. Any business that's using a waterfall approach to developing nontrivial software is wasting money. Potentially, lots of money. I'm going to show you how, how much, and how to avoid it in upcoming columns.
Security is not an option. This KnowledgeVault Series offers professional advice how to be proactive in the fight against cybercrimes and multi-layered security threats; how to adopt a holistic approach to protecting and managing data; and how to hire a qualified security assessor. Make security your Number 1 priority.
New IP-based communications systems are being deployed by small and midsized businesses at a rapid rate. Learn how these organizations are enabling faster responsiveness, creating better customer experiences, speeding office or mobile interactions, and dramatically reducing existing communications costs.
This Executive Brief discusses IDC's perspective on how enterprise workload management requirements are changing and highlights the ways that workload automation solutions can...
View this demo and learn how IBM InfoSphere Guardium database activity monitoring can help protect your sensitive data in distributed DBMS environments with...
Free Insider Guide