Evaluating Project Success, Failure -- and Everything in Between

According to the widely quoted "Chaos Chronicles," a staggering 66% of IT projects prove unsuccessful in some measure, whether they fail completely, exceed their allotted budget, aren't completed according to schedule or are rolled out with fewer features and functions than promised.

Clearly, IT project management could do with some improvement. But project managers themselves are often working so hard, and racing against such tight deadlines, that they simply don't have the time to pause and reflect on the efficacy of their project management practices.

Yet the key to improving the rate of project success may lie right underneath managers' noses in the form of comprehensive project retrospectives. Allotting time for the formal evaluation of a project's success, extracting the primary lessons from the project management process and making actionable recommendations for future projects (or for the future of the project in question) might very well, in the longer term, represent the most efficient use of managers' time and resources.

Graduate students in the master of science in the management of IT program at the University of Virginia's McIntire School of Commerce have, since 1999, collectively performed 72 IT project retrospectives at 57 organizations. These retrospectives have ranged from the examination of relatively small (several hundred thousand dollars) intrafirm projects to very large (over $100 million) mission-critical projects involving multiple external providers.

Each retrospective tells a unique story in its own right and provides a rich understanding of specific project management practices used within a specific time frame. But when viewed as a whole, these 72 projects offer the opportunity for considerable insight into project management at the macro level, revealing trends that may be generalized across a wide spectrum of applications and organizations.

Fundamentally, the project retrospectives showed success itself is difficult to define. In a typical group of stakeholders—i.e., the project manager, team members, product end users, project sponsor and top management—a project's success might, at any given moment, receive very different ratings.

Given this reality, it became clear that an evaluation of project success should include both process and outcome criteria. They used the following process-related criteria:

  1. Time: Did the project come in on schedule?
  2. Cost: Did the project come in according to budget?
  3. Product: Did the project result in a product of acceptable quality and meet other product-related specifications?

The three outcome-related criteria they used were:

  1. Use: Were the project's resultant products/services used by its intended constituents?
  2. Learning: Did the project increase stakeholder knowledge and better prepare the organization for future challenges?
  3. Value Did the project lead directly to the organization's improved efficiency or effectiveness? Common metrics include net present value (NPV), internal rate of return (IRR), economic value added (EVA) and the balanced scorecard.
1pixclear.gif

Project Success Criteria

Project Success Criteria

Interestingly, the retrospective data showed that the top three criteria (listed in descending order of importance) by which stakeholders judged a project's success were product, use and value; cost was ranked lowest overall. Although project managers and team members seemed to be more concerned with project process, users, not surprisingly, cared most about whether or not the project's outcome (a system, for example) could be used as intended.

Interestingly, the research also showed that project success might happen in some unexpected ways, and, likewise, failure might bring with it some surprising positives. In the case of IT project management, instances of such unexpected outcomes may be deemed "failed successes" (process success but outcome failure) or "successful failures" (process failure but outcome success). A successful failure might be completed on time and under budget, for instance, but fail to appeal to the intended constituency or add value to an organization. Conversely, a successful failure might cost more than expected and take longer than expected to complete—yet ultimately add value to an organization.

Given the critical role that project management plays in the field of IT, managers would be well advised to take some time to participate in the sometimes painful process of project retrospectives. Managers must recognize not only that the perspectives of many stakeholders must be considered, and on various dimensions, but also that virtually every project is characterized by some failures and some successes.

Project managers must strive to become adept at negotiating the trade-offs between process (time, cost and product) and outcome (use, learning and business value). The project charter should include negotiated success metrics, the project dashboard should enable real-time metrics, and the project retrospective should document the actual results, concluding with overall stakeholder satisfaction.

There is a lot to be gained from looking into the rearview mirror from time to time. As philosopher George Santayana so pithily remarked, "Those who cannot remember the past are condemned to repeat it."

Ryan Nelson is the director of the Center for the Management of Information Technology at the University of Virginia's McIntire School of Commerce in Charlottesville. He can be contacted at rrn2n@comm.virginia.edu.

Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon