Issue No.01 - January/February (2010 vol.27)
pp: 30-36
J. Laurenz Eveleens , VU University, Amsterdam
In 1994, Standish published the Chaos report that showed a shocking 16 percent project success. This and renewed figures by Standish are often used to indicate that project management of application software development is in trouble. However, Standish's definitions have four major problems. First, they're misleading because they're based solely on estimation accuracy of cost, time, and functionality. Second, their estimation accuracy measure is one-sided, leading to unrealistic success rates. Third, steering on their definitions perverts good estimation practice. Fourth, the resulting figures are meaningless because they average numbers with an unknown bias, numbers that are introduced by different underlying estimation processes. The authors of this article applied Standish's definitions to their own extensive data consisting of 5,457 forecasts of 1,211 real-world projects, totaling hundreds of millions of Euros. The Standish figures didn't reflect the reality of the case studies at all.
chaos report, standish group, forecasting, project success, software, software engineering
J. Laurenz Eveleens, "The Rise and Fall of the Chaos Report Figures", IEEE Software, vol.27, no. 1, pp. 30-36, January/February 2010, doi:10.1109/MS.2009.154
1. Chaos, tech. report, Standish Group Int'l, 1994.
2. R. Glass, "IT Failure Rates—70% or 10–15%," IEEE Software, May 2005, pp. 110–112.
3. R. Glass, "The Standish Report: Does It Really Describe a Software Crisis?" Comm. ACM, vol. 49, no. 8, 2006, pp. 15–16.
4. M. J⊘rgensen and K. Mol⊘kken, "How Large Are Software Cost Overruns? A Review of the 1994 Chaos Report," Information and Software Technology, vol. 48, no. 8, 2006, pp. 297–301.
5. D. Hartmann, "Interview: Jim Johnson of the Standish Group," 2006;
6. Chaos: A Recipe for Success, tech. report, Standish Group Int'l, 1999.
7. Extreme Chaos, tech. report, Standish Group Int'l, 2001.
8. B. Joy and K. Kennedy, Information Technology Research: Investing in Our Future, tech. report, President's Information Technology Advisory Committee, Feb. 1999.
9. B. Boehm, Software Engineering Economics, Prentice Hall, 1981.
10. T. De Marco, Controlling Software Projects, Prentice Hall, 1982.
11. T. Lister, "Becoming a Better Estimator—An Introduction to Using the EQF Metric,", 2002;
12. D. Garmus and D. Herron, Function Point Analysis—Measurement Practices for Successful Software Projects, Addison-Wesley, 2001.
13. T. Little, "Schedule Estimation and Uncertainty Surrounding the Cone of Uncertainty," IEEE Software, vol. 23, no. 3, 2006, pp. 48–54.
14. J. Johnson, "Standish: Why Were Project Failures Up and Cost Overruns Down in 1998?", 2006;
15. J.L. Eveleens and C. Verhoef, "Quantifying IT Forecast Quality," Science of Computer Programming, vol. 74, no. 11+12, 2009, pp. 934-988;
16. N. Zvegintzov, "Frequently Begged Questions and How to Answer Them," IEEE Software, vol. 20, no. 2, 1998, pp. 93–96.