Computerworld - Sabre Airline Solutions had many years of experience with its AirFlite Profit Manager, a big modeling and forecasting package that many airlines use to wring more income out of flight schedules. Even so, Release 8 of the software was four months late in 2000 after final system testing turned up 300 bugs. The first customer found 26 more bugs in the first three days of its acceptance testing, and subsequent joint testing by Sabre and the customer uncovered an additional 200 defects.
"Our Sabre person on-site was as upset with the quality as the customer was," recalls Vinit Karandikar, a principal in Sabre's Airline Products Development group.
Sabre's experience with Release 8, which had almost 500,000 lines of code, isn't unusual. Software quality guru Watts Humphrey, a fellow at the Software Engineering Institute in Pittsburgh, estimates that commercial software typically ships with one to eight defects per 1,000 lines of code.
But a lot has changed in the development labs of Sabre Airline Solutions, part of Sabre Holdings Corp., a $2 billion air-travel systems company based in Southlake, Texas. The software development firm has embraced extreme programming (XP), a controversial development framework in which testing precedes coding and programmers work in pairs. Sabre Airline Solutions has 62 software products with 13 million lines of code, and it claims that XP has dramatically boosted both the productivity of its 300 developers and the quality of their work.
For example, fewer than 10 bugs surfaced in Release 10 of its Profit Manager software in the two months after it began shipping to airlines in December 2002. Now, 16 months later, just 100 defects have been found. Release 10 was written in Java, while Release 8 was written in C and C++. But Sabre says it was XP, not Java, that produced the dramatic quality improvements.
And the initial benefits from developing better code may be eclipsed by long-term cost savings, Karandikar says. For Release 10, Sabre assigned just three developers to support 13 customers, while Release 8 required 13 people to support 12 customers.
The evidence is anecdotal so far, but there are enough anecdotes at Sabre to make a compelling case for XP. In another project, the company converted the user interface of its AirServ airline cabin provisioning optimization system from C++ to Java for the Web, a two-year effort that required rewriting about 100 graphical user interface programs. Programmer productivity jumped 42%—as measured by the number of labor hours required for each screen—after the development team switched to XP halfway through the project.
In yet another project, the Host Access Tool, which provides a common application programming interface for accessing legacy host systems and has 15,000 lines of code, was written from scratch using XP practices and has remained bug free in the 20 months since it shipped. Similarly, Sabre's Peripheral Manager, which manages interaction between host systems and peripheral devices, has 28,000 lines of code and has had just four bugs show up in 15 months.
Establishing Best Practices
Sabre Airline Solutions adopted XP in 2001. With its new model, Sabre does iterative development in small, simple steps. The company uses two-week iterations, and customers see a new release every one to three months. Features, called "stories," are expressed in user terms and must be simple enough to be coded, tested and integrated in two weeks or less.
Automated unit tests (against the programmer's criteria) and broader acceptance tests (against customer requirements) must be passed at the end of each iteration before the next can begin. Unit and acceptance tests for each feature are written before the feature is coded. If a developer has trouble writing a test, he doesn't clearly understand the feature.
Actual coding is done in pairs by teams in open labs, promoting collective ownership of code, although individuals sometimes do the simplest tasks. Programmers are re-paired frequently, often every day or two. They sign up for the tasks they want to do and the person they want to pair with.
Every project team has an "XP coach" and an application subject-matter expert called the XP customer. The XP customer stays in or near the programming lab all or most of the time. He decides on and prioritizes product features, writes the stories for programmers and signs off on the results.
"Refactoring" code—rewriting it not to fix bugs or add features but to make it less redundant and more maintainable—is strongly encouraged. Sabre says the concept hardly existed at the company before XP because it was too difficult.
Free Insider Guide