March 29, 2004 (Computerworld) -- Sabre Airline Solutions had many years of experience with its AirFlite Profit Manager, a big modeling and forecasting package that many airlines use to wring more income out of flight schedules. Even so, Release 8 of the software was four months late in 2000 after final system testing turned up 300 bugs. The first customer found 26 more bugs in the first three days of its acceptance testing, and subsequent joint testing by Sabre and the customer uncovered an additional 200 defects.
"Our Sabre person on-site was as upset with the quality as the customer was," recalls Vinit Karandikar, a principal in Sabre's Airline Products Development group.
Sabre's experience with Release 8, which had almost 500,000 lines of code, isn't unusual. Software quality guru Watts Humphrey, a fellow at the Software Engineering Institute in Pittsburgh, estimates that commercial software typically ships with one to eight defects per 1,000 lines of code.
But a lot has changed in the development labs of Sabre Airline Solutions, part of Sabre Holdings Corp., a $2 billion air-travel systems company based in Southlake, Texas. The software development firm has embraced extreme programming (XP), a controversial development framework in which testing precedes coding and programmers work in pairs. Sabre Airline Solutions has 62 software products with 13 million lines of code, and it claims that XP has dramatically boosted both the productivity of its 300 developers and the quality of their work.
For example, fewer than 10 bugs surfaced in Release 10 of its Profit Manager software in the two months after it began shipping to airlines in December 2002. Now, 16 months later, just 100 defects have been found. Release 10 was written in Java, while Release 8 was written in C and C++. But Sabre says it was XP, not Java, that produced the dramatic quality improvements.
And the initial benefits from developing better code may be eclipsed by long-term cost savings, Karandikar says. For Release 10, Sabre assigned just three developers to support 13 customers, while Release 8 required 13 people to support 12 customers.
The evidence is anecdotal so far, but there are enough anecdotes at Sabre to make a compelling case for XP. In another project, the company converted the user interface of its AirServ airline cabin provisioning optimization system from C++ to Java for the Web, a two-year effort that required rewriting about 100 graphical user interface programs. Programmer productivity jumped 42%—as measured by the number of labor hours required for each screen—after the development team switched to XP halfway through the project.
In yet another project, the Host Access Tool, which provides a common application programming interface for accessing legacy host systems and has 15,000 lines of code, was written from scratch using XP practices and has remained bug free in the 20 months since it shipped. Similarly, Sabre's Peripheral Manager, which manages interaction between host systems and peripheral devices, has 28,000 lines of code and has had just four bugs show up in 15 months.
Establishing Best Practices
Sabre Airline Solutions adopted XP in 2001. With its new model, Sabre does iterative development in small, simple steps. The company uses two-week iterations, and customers see a new release every one to three months. Features, called "stories," are expressed in user terms and must be simple enough to be coded, tested and integrated in two weeks or less.
Automated unit tests (against the programmer's criteria) and broader acceptance tests (against customer requirements) must be passed at the end of each iteration before the next can begin. Unit and acceptance tests for each feature are written before the feature is coded. If a developer has trouble writing a test, he doesn't clearly understand the feature.
Actual coding is done in pairs by teams in open labs, promoting collective ownership of code, although individuals sometimes do the simplest tasks. Programmers are re-paired frequently, often every day or two. They sign up for the tasks they want to do and the person they want to pair with.
Every project team has an "XP coach" and an application subject-matter expert called the XP customer. The XP customer stays in or near the programming lab all or most of the time. He decides on and prioritizes product features, writes the stories for programmers and signs off on the results.
"Refactoring" code—rewriting it not to fix bugs or add features but to make it less redundant and more maintainable—is strongly encouraged. Sabre says the concept hardly existed at the company before XP because it was too difficult.
Brad Jensen, a senior vice president at Sabre Airline Solutions Image Credit: Reid Horn
Finally, simplicity is paramount. The simplest things are done first, and code is never made more complicated in order to accommodate an anticipated future need that may never materialize.
Brad Jensen, senior vice president for airline product development at Sabre, says the quality improvements come largely from XP's continuous testing and integration. "Every two weeks what you've completed has got to be production-ready," he says. "You code as you test. You actually write an automated unit test before you code the unit, so if bugs do creep in, you find out about it right then."
Damon Hougland, director of airline products and services, says pair programming is hard for some to swallow at first because it suggests that programming costs will double. But the method actually reduces cost, he says, because the extra time it takes to write a line of code is more than offset by the reduced effort required for testing, fixing bugs and maintaining the code.
And, because at least two people know every chunk of software, and because Sabre reassigns and re-pairs people frequently, there is always a backup on hand. "Everyone on the team works on every part of the system," Hougland says. "You have the weaker people paired with the stronger people, and business knowledge and coding knowledge are transferred very quickly."
Beyond XP
XP doesn't encompass all the practices that a software development organization should follow, Hougland says. "XP really focuses on what [programmers] do," he says. "It doesn't cover the traditional project management you have to do with customers, such as customer communications, and a lot of the testing we do is not covered in XP. A lot of people try XP and fail because they assume that XP will do everything for a development methodology."
Sabre doesn't yet follow XP 100%. XP doctrine says that both unit tests and acceptance tests should be automated. Unit tests at Sabre are conducted with the open-source JUnit testing tool, but many of the more complex acceptance tests are conducted with manual scripts. "We've struggled to automate them, and we have made some progress," Hougland says.
Jensen compares XP to the Capability Maturity Model for Software, saying XP covers many SW-CMM practices. The difference, he says, is that CMM tells organizations what they should do while XP is more oriented to saying how to do those things. "CMM is all about management processes; it's less about technical processes, the techniques of coding," says Jensen.
Jensen, who sponsored the adoption of XP at Sabre Airline Solutions, says he considered using other so-called agile programming methods, such as Agile RUP, a version of IBM's Rational Unified Process; Scrum, an iterative, user-oriented methodology; and Feature-Driven Development. "None of them are as fully developed as XP," he says.
Following extreme programming (XP) practices, Sabre Airline Solutions groups programmers into pairs that work closely together. They are frequently re-paired to broaden knowledge.
Jensen says he especially likes XP metrics because they come in user terminology rather than the technical jargon of some methodologies. "With XP, we are checking off features and stories that I can understand like a business person," he says. "There's a lot of increased communication between the customer and the development team because everything that's being prioritized is in terms the user can understand."
Jensen acknowledges that his developers deviate somewhat from pure XP practices. Sometimes the XP customer has to travel and can't be in his lab full time, as XP doctrine demands; refactoring old legacy code can be extremely difficult; testing isn't as fully automated as XP would have it; and acute deadline pressure from customers can lead to cutting corners.
But when asked if he has reservations about any of the 12 core XP practices, Jensen says, "They are all good practices. The only thing you could possibly argue is whether you really can afford to do all of them all the time. But if there's any compromising, it's not intentional."
Adding Up the Savings
HOW PRODUCTIVITY IMPROVED
Sabre's effort to rewrite the GUIs for its AirServ software took off after the company migrated to XP practices; productivity improved by 42%.
Note: Sabre adjusted labor hours, normalizing for GUI complexity, learning-curve effects and other factors.
AirServ GUIs
Before XP (2001)
After XP (2002)
GUIs rewritten
87
91
Labor (hours)
23,531
17,276
Hours/GUI
270
190
Productivity gain: 42%
EXTREME QUALITY
Using XP cut defects dramatically for three software products. The biggest program, Profit Manager, to date has one-fifth the number of bugs found in the previous version.
Mistakes such as putting down co-workers or burning bridges when you resign are surefire ways to darken your career prospects. Here's how to avoid them
Computerworld editors share stories of their first PCs, including some classics and some real clunkers -- then we ask readers to share their early-PC tales.
Download this Computerworld report, free for a limited time, compliments of HP. (Source: Computerworld) Faced with growing demands, immature tools and a confusing array of technologies, IT decision-makers have to make some strategic choices. Learn how to avoid the pitfalls in this Computerworld report, a $49.95 value, available free for a limited time, compliments of HP.
Download this executive briefing
Transformational Analytics: Virtualizing IT Environments
Download this white paper, free, compliments of CiRBA. (Source: CiRBA) The overwhelming complexity of the modern data center compounds the problem of how to safely virtualize IT environments. This paper provides an in-depth guide to analyzing complex environments for virtualization opportunities, particularly within production environments where stability, service levels and performance are of the upmost performance.
Download this white paper
Rapid, Widespread Adoption of CMMI at Lockheed Martin with Application Lifecycle Management
Rapid, Widespread Adoption of CMMI at Lockheed Martin with Application Lifecycle Management
Get this webcast now
Go to the webcast
White Papers
Read up on the latest ideas and technologies from companies that sell hardware, software and services.
AnalyzeAir software provides IT network professionals with the vision they need into the hidden world of RF, providing them with the ability to see the spectrum in a visible and intelligible format. AnalyzeAir software lets you see, monitor, analyze, and manage all the RF sources and wireless devices that influence your Wi-Fi network's performance and security, even if those devices are unauthorized or transient.
AnalyzeAir Trial Software v3.1 highlights the features found in AnalyzeAir Software using a set of saved spectrum files. Replay the data and experience the visibility that AnalyzeAir Wi-Fi Spectrum Analyzer provides. Note: The trial software is limited to a player version only. It does not communicate with an AnalyzeAir PC card so it does not collect actual spectrum data.
Subscribe to
Computerworld 
or
Other Development Stories
Download this Computerworld report, free for a limited time, compliments of HP.
Download this white paper, free, compliments of CiRBA.
Rapid, Widespread Adoption of CMMI at Lockheed Martin with Application Lifecycle Management
Read up on the latest ideas and technologies from companies that sell hardware, software and services.
Many of today's requirements management tool purchases are misguided: Application development and program management professionals often buy requirements management tools for the wrong reasons and select tools that are out of line with their needs. In this independent report, Forrester advises app dev organizations to be realistic about the problems that a requirements management tool can address, the level of tooling that they require, and their ability to build and maintain tool integrations.
