March 29, 2004 (Computerworld) -- Sabre Airline Solutions had many years of experience with its AirFlite Profit Manager, a big modeling and forecasting package that many airlines use to wring more income out of flight schedules. Even so, Release 8 of the software was four months late in 2000 after final system testing turned up 300 bugs. The first customer found 26 more bugs in the first three days of its acceptance testing, and subsequent joint testing by Sabre and the customer uncovered an additional 200 defects.
"Our Sabre person on-site was as upset with the quality as the customer was," recalls Vinit Karandikar, a principal in Sabre's Airline Products Development group.
Sabre's experience with Release 8, which had almost 500,000 lines of code, isn't unusual. Software quality guru Watts Humphrey, a fellow at the Software Engineering Institute in Pittsburgh, estimates that commercial software typically ships with one to eight defects per 1,000 lines of code.
But a lot has changed in the development labs of Sabre Airline Solutions, part of Sabre Holdings Corp., a $2 billion air-travel systems company based in Southlake, Texas. The software development firm has embraced extreme programming (XP), a controversial development framework in which testing precedes coding and programmers work in pairs. Sabre Airline Solutions has 62 software products with 13 million lines of code, and it claims that XP has dramatically boosted both the productivity of its 300 developers and the quality of their work.
For example, fewer than 10 bugs surfaced in Release 10 of its Profit Manager software in the two months after it began shipping to airlines in December 2002. Now, 16 months later, just 100 defects have been found. Release 10 was written in Java, while Release 8 was written in C and C++. But Sabre says it was XP, not Java, that produced the dramatic quality improvements.
And the initial benefits from developing better code may be eclipsed by long-term cost savings, Karandikar says. For Release 10, Sabre assigned just three developers to support 13 customers, while Release 8 required 13 people to support 12 customers.
The evidence is anecdotal so far, but there are enough anecdotes at Sabre to make a compelling case for XP. In another project, the company converted the user interface of its AirServ airline cabin provisioning optimization system from C++ to Java for the Web, a two-year effort that required rewriting about 100 graphical user interface programs. Programmer productivity jumped 42%—as measured by the number of labor hours required for each screen—after the development team switched to XP halfway through the project.
In yet another project, the Host Access Tool, which provides a common application programming interface for accessing legacy host systems and has 15,000 lines of code, was written from scratch using XP practices and has remained bug free in the 20 months since it shipped. Similarly, Sabre's Peripheral Manager, which manages interaction between host systems and peripheral devices, has 28,000 lines of code and has had just four bugs show up in 15 months.
Establishing Best Practices
Sabre Airline Solutions adopted XP in 2001. With its new model, Sabre does iterative development in small, simple steps. The company uses two-week iterations, and customers see a new release every one to three months. Features, called "stories," are expressed in user terms and must be simple enough to be coded, tested and integrated in two weeks or less.
Automated unit tests (against the programmer's criteria) and broader acceptance tests (against customer requirements) must be passed at the end of each iteration before the next can begin. Unit and acceptance tests for each feature are written before the feature is coded. If a developer has trouble writing a test, he doesn't clearly understand the feature.
Actual coding is done in pairs by teams in open labs, promoting collective ownership of code, although individuals sometimes do the simplest tasks. Programmers are re-paired frequently, often every day or two. They sign up for the tasks they want to do and the person they want to pair with.
Every project team has an "XP coach" and an application subject-matter expert called the XP customer. The XP customer stays in or near the programming lab all or most of the time. He decides on and prioritizes product features, writes the stories for programmers and signs off on the results.
"Refactoring" code—rewriting it not to fix bugs or add features but to make it less redundant and more maintainable—is strongly encouraged. Sabre says the concept hardly existed at the company before XP because it was too difficult.
Brad Jensen, a senior vice president at Sabre Airline Solutions Image Credit: Reid Horn
Finally, simplicity is paramount. The simplest things are done first, and code is never made more complicated in order to accommodate an anticipated future need that may never materialize.
Brad Jensen, senior vice president for airline product development at Sabre, says the quality improvements come largely from XP's continuous testing and integration. "Every two weeks what you've completed has got to be production-ready," he says. "You code as you test. You actually write an automated unit test before you code the unit, so if bugs do creep in, you find out about it right then."
Damon Hougland, director of airline products and services, says pair programming is hard for some to swallow at first because it suggests that programming costs will double. But the method actually reduces cost, he says, because the extra time it takes to write a line of code is more than offset by the reduced effort required for testing, fixing bugs and maintaining the code.
And, because at least two people know every chunk of software, and because Sabre reassigns and re-pairs people frequently, there is always a backup on hand. "Everyone on the team works on every part of the system," Hougland says. "You have the weaker people paired with the stronger people, and business knowledge and coding knowledge are transferred very quickly."
Beyond XP
XP doesn't encompass all the practices that a software development organization should follow, Hougland says. "XP really focuses on what [programmers] do," he says. "It doesn't cover the traditional project management you have to do with customers, such as customer communications, and a lot of the testing we do is not covered in XP. A lot of people try XP and fail because they assume that XP will do everything for a development methodology."
Sabre doesn't yet follow XP 100%. XP doctrine says that both unit tests and acceptance tests should be automated. Unit tests at Sabre are conducted with the open-source JUnit testing tool, but many of the more complex acceptance tests are conducted with manual scripts. "We've struggled to automate them, and we have made some progress," Hougland says.
Jensen compares XP to the Capability Maturity Model for Software, saying XP covers many SW-CMM practices. The difference, he says, is that CMM tells organizations what they should do while XP is more oriented to saying how to do those things. "CMM is all about management processes; it's less about technical processes, the techniques of coding," says Jensen.
Jensen, who sponsored the adoption of XP at Sabre Airline Solutions, says he considered using other so-called agile programming methods, such as Agile RUP, a version of IBM's Rational Unified Process; Scrum, an iterative, user-oriented methodology; and Feature-Driven Development. "None of them are as fully developed as XP," he says.
Following extreme programming (XP) practices, Sabre Airline Solutions groups programmers into pairs that work closely together. They are frequently re-paired to broaden knowledge.
Jensen says he especially likes XP metrics because they come in user terminology rather than the technical jargon of some methodologies. "With XP, we are checking off features and stories that I can understand like a business person," he says. "There's a lot of increased communication between the customer and the development team because everything that's being prioritized is in terms the user can understand."
Jensen acknowledges that his developers deviate somewhat from pure XP practices. Sometimes the XP customer has to travel and can't be in his lab full time, as XP doctrine demands; refactoring old legacy code can be extremely difficult; testing isn't as fully automated as XP would have it; and acute deadline pressure from customers can lead to cutting corners.
But when asked if he has reservations about any of the 12 core XP practices, Jensen says, "They are all good practices. The only thing you could possibly argue is whether you really can afford to do all of them all the time. But if there's any compromising, it's not intentional."
Adding Up the Savings
HOW PRODUCTIVITY IMPROVED
Sabre's effort to rewrite the GUIs for its AirServ software took off after the company migrated to XP practices; productivity improved by 42%.
Note: Sabre adjusted labor hours, normalizing for GUI complexity, learning-curve effects and other factors.
AirServ GUIs
Before XP (2001)
After XP (2002)
GUIs rewritten
87
91
Labor (hours)
23,531
17,276
Hours/GUI
270
190
Productivity gain: 42%
EXTREME QUALITY
Using XP cut defects dramatically for three software products. The biggest program, Profit Manager, to date has one-fifth the number of bugs found in the previous version.
"I told you so. I told you back in late June VMware was about to get its block knocked-off by..."
Read more...
"If you fail to plan a backup strategy, you can plan��on the failure of your data recovery efforts should the..."
Read more... Read more Software posts or See all Blogs
Is Microsoft's Golden Age over? What are Gates' most memorable quotes? Find out in Computerworld's complete coverage of the end of the Bill Gates era at Microsoft.
Computerworld Technology Briefing: An open-source path to optimal virtualization
Download this Technology Briefing now! (Source: Novell/IBM/Intel) Virtualization is about a lot more than just lowering total cost of ownership. In fact users that have taken an open source path to virtualization have realized the additional, mission-critical benefit of markedly reduced IT complexity, as well as a more flexible infrastructure that is easier to change to meet shifting, often unpredictable business requirements.
Download this executive briefing
Advance your BlackBerry(R) solution management know-how this July
Advance your BlackBerry(R) solution management know-how this July
BlackBerry Technical Seminar, register today!
Go to the webcast
Rapid application development, rapid results
Download this special report now! (Source: Intersystems) All too many businesses suffer from IT infrastructures that are a hodge-podge of disconnected databases and applications. What's needed is the ability rapidly develop connected applications under a unified service-oriented architecture. InterSystems Ensemble integration environment and Cache database are effective tools in answering this need, delivering a rapid ROI.
Download this white paper
White Papers
Read up on the latest ideas and technologies from companies that sell hardware, software and services.
Computerworld Technology Briefing: Meetings @ the Speed of Business For large organizations, Web conferencing gives a major boost to collaboration among far-flung offices. For smaller companies, experts say Web conferencing is no longer a luxury but a necessity for everything from webinars to customer presentations. But the real value lies in saving soft costs and in increases in productivity. Download this briefing
Subscribe to
Computerworld 
or
Other Software Stories
Download this Technology Briefing now!
Advance your BlackBerry(R) solution management know-how this July
Download this special report now!
Read up on the latest ideas and technologies from companies that sell hardware, software and services.
For large organizations, Web conferencing gives a major boost to collaboration among far-flung offices. For smaller companies, experts say Web conferencing is no longer a luxury but a necessity for everything from webinars to customer presentations. But the real value lies in saving soft costs and in increases in productivity.
This paper shows how Brocade StorageX enables tiers of storage across Windows and NetApp storage to align storage practices with organizational objectives. 