This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Evidence-Based Guidelines for Assessment of Software Development Cost Uncertainty
November 2005 (vol. 31 no. 11)
pp. 942-954
Several studies suggest that uncertainty assessments of software development costs are strongly biased toward overconfidence, i.e., that software cost estimates typically are believed to be more accurate than they really are. This overconfidence may lead to poor project planning. As a means of improving cost uncertainty assessments, we provide evidence-based guidelines for how to assess software development cost uncertainty, based on results from relevant empirical studies. The general guidelines provided are: 1) Do not rely solely on unaided, intuition-based uncertainty assessment processes, 2) do not replace expert judgment with formal uncertainty assessment models, 3) apply structured and explicit judgment-based processes, 4) apply strategies based on an outside view of the project, 5) combine uncertainty assessments from different sources through group work, not through mechanical combination, 6) use motivational mechanisms with care and only if greater effort is likely to lead to improved assessments, and 7) frame the assessment problem to fit the structure of the relevant uncertainty information and the assessment process. These guidelines are preliminary and should be updated in response to new evidence.

[1] J. Ross and B.M. Staw, “Expo 86: An Escalation Prototype,” Administrative Science Quarterly, vol. 31, pp. 274-297, 1986.
[2] S. McConnell, Software Project Survival Guide. Microsoft Press, 1998.
[3] M.T. Pich, C.H. Loch, and A. De Meyer, “On Uncertainty, Ambiguity and Complexity In Project Management,” Management Science, vol. 48, pp. 1008-1023, 2002.
[4] B.A. Kitchenham, “Modeling Software Bidding Risks,” IEEE Trans. Software Eng., vol. 29, no. 6, pp. 542-554, 2003.
[5] M. Jørgensen and D.I.K. Sjøberg, “Impact of Effort Estimates on Software Project Work,” Information and Software Technology, vol. 43, no. 15, pp. 939-948, 2001.
[6] A. Huchzermeier and C.H. Loch, “Project Management under Risk: Using the Real Options Approach to Evaluate Flexibility in R&D,” Management Science, vol. 47, no. 1, pp. 85-101, 2001.
[7] D. Kahneman and D. Lovallo, “Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking,” Management Science, vol. 39, no. 1, pp. 17-31, 1993.
[8] M. Jørgensen, K.H. Teigen, and K. Moløkken, “Better Sure Than Safe? Overconfidence in Judgement Based Software Development Effort Prediction Intervals,” J. Systems and Software, vol. 70, nos. 1-2, pp. 79-93, 2004.
[9] L. Angelis and I. Stamelos, “A Simulation Tool for Efficient Analogy Based Cost Estimation,” Empirical Software Eng., vol. 5, no. 1, pp. 35-68, 2000.
[10] M. Jørgensen and D.I.K. Sjøberg, “An Effort Prediction Interval Approach Based on the Empirical Distribution of Previous Estimation Accuracy,” Information and Software Technology, vol. 45, no. 3, pp. 123-136, 2003.
[11] M. Jørgensen, “Regression Models of Software Development Effort Estimation Accuracy and Bias,” Empirical Software Eng., vol. 9, no. 4, pp. 297-314, 2004.
[12] W.S. Humphrey and N.D. Singpuwally, “Predicting (Individual) Software Productivity,” IEEE Trans. Software Eng., vol. 17, no. 2, pp. 196-207, 1991.
[13] C. Chapman and S. Ward, “Estimation and Evaluation of Uncertainty: A Minimalist First Pass Approach,” Int'l J. Project Management, vol. 18, no. 6, pp. 369-383, 2000.
[14] P.P. Dagum, et al., “Uncertain Reasoning and Forecasting,” Int'l J. Forecasting, vol. 11, no. 1, pp. 73-87, 1995.
[15] M. Elkjaer, “Stochastic Budget Simulation,” Int'l J. Project Management, vol. 18, no. 2, pp. 139-147, 2000.
[16] M.R. Duffey and J.R. van Dorp, “Risk Analysis for Large Engineering Projects: Modeling Cost Uncertainty for Ship Production Activities,” J. Eng. Valuation and Cost Analysis, vol. 2, no. 4, pp. 285-301, 1999.
[17] J. Hihn and H. Habib-Agahi, “Cost Estimation of Software Intensive Projects: A Survey of Current Practices,” Proc. Int'l Conf. Software Eng., pp. 276-287, 1991.
[18] T. Connolly and D. Dean, “Decomposed versus Holistic Estimates of Effort Required for Software Writing Tasks,” Management Science, vol. 43, no. 7, pp. 1029-1045, 1997.
[19] M. Jørgensen and K.H. Teigen, “Uncertainty Intervals versus Interval Uncertainty: An Alternative Method for Eliciting Effort Prediction Intervals in Software Development Projects,” Proc. Int'l Conf. Project Management (ProMAC), pp. 343-352, 2002.
[20] M. Jørgensen, “Top-Down and Bottom-Up Expert Estimation of Software Development Effort,” Information and Software Technology, vol. 46, no. 1, pp. 3-16, 2004.
[21] M. Alpert and H. Raiffa, “A Progress Report on the Training of Probability Assessors,” Judgment Under Uncertainty: Heuristics and Biases, D. Kahneman, et al., eds., pp. 294-305, 1982.
[22] D. Kahneman, PP. Slovic, and A. Tversky, Judgment Under Uncertainty: Heuristics and Biases. Cambridge, U.K.: Cambridge Univ. Press, 1982.
[23] A. Tversky and D. Kahneman, “Judgment Under Uncertainty: Heuristics and Biases,” Science, vol. 185, pp. 1124-1131, 1974.
[24] I. Yaniv and D.P.P. Foster, “Precision and Accuracy of Judgmental Estimation,” J. Behavioral Decision Making, vol. 10, pp. 21-32, 1997.
[25] S. Lichtenstein and B. Fischhoff, “Do Those Who Know More Also Know More About How Much They Know?” Organizational Behaviour and Human Decision Processes, vol. 20, no. 2, pp. 159-183, 1977.
[26] H.R. Arkes, “Overconfidence In Judgmental Forecasting,” Principles of Forecasting: A Handbook For Researchers and Practitioners, J.S. Armstrong, ed., pp. 495-415, 2001.
[27] NASA, Manager's Handbook for Software Development. NASA Software Eng. Laboratory, Greenbelt, Md.: Goddard Space Flight Center, 1990.
[28] B. Kitchenham, et al., “An Empirical Study of Maintenance and Development Estimation Accuracy,” J. Systems and Software, vol. 64, no. 1, pp. 57-77, 2002.
[29] S.M. Whitecotton, D.E. Sanders, and K.B. Norris, “Improving Predictive Accuracy with a Combination of Human Intuition and Mechanical Decision Aids,” Organizational Behaviour and Human Decision Processes, vol. 76, no. 3, pp. 325-348, 1998.
[30] T. Abdel-Hamid, “Investigating the Cost/Schedule Trade-Off in Software Development,” IEEE Software, vol. 7, no. 1, pp. 97-105, 1990.
[31] T.K. Abdel-Hamid and S.E. Madnik, “Impact of Schedule Estimation on Software Project Behavior,” IEEE Software, vol. 3, no. 4, pp. 70-75, 1986.
[32] K. Sengupta and T.K. Abdel-Hamid, “The Impact of Unreliable Information on the Management of Software Projects: A Dynamic Decision Perspective,” IEEE Trans. Systems, Man, and Cybernetics, vol. 26, no. 2, pp. 177-189, 1996.
[33] H. Otway and D. von Winterfeldt, “Expert Judgment in Risk Analysis and Management: Process, Context, and Pitfalls,” Risk Analysis, vol. 12, no. 1, pp. 83-93, 1992.
[34] M. Jørgensen, “Realism in Assessment of Effort Estimation Uncertainty: It Matters How You Ask,” IEEE Trans. Software Eng., vol. 30, no. 4, pp. 209-217, 2004.
[35] G. Gigerenzer, “Why the Distinction Between Single-Event Probabilities and Frequencies is Important for Psycology (and Vice Versa),” Subjective Probabilities, G. Wright and P.P. Ayton, eds., 1994.
[36] D. Kahneman and A. Tversky, “Variants of Uncertainty,” Cognition, vol. 11, pp. 143-157, 1982.
[37] M. Jørgensen and K. Moløkken, “Eliminating Overconfidence in Software Development Effort Estimates,” Proc. Int'l Conf. Product Focused Software Process Improvement, pp. 17-29, 2004.
[38] J.S. Armstrong, “Combining Forecasts,” Principles of Forecasting: A Handbook for Researchers and Practitioners, J.S. Armstrong, ed., pp. 417-440, 2001.
[39] M. Höst and C. Wohlin, “An Experimental Study of Individual Subjective Effort Estimations and Combinations of the Estimates,” Proc. Int'l Conf. Software Eng., pp. 332-339, 1998.
[40] S.G. MacDonell, “Software Source Code Sizing Using Fuzzy Logic Modeling,” Information and Software Technology, vol. 45, no. 7, pp. 389-404, 2003.
[41] I. Myrtveit and E. Stensrud, “A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models,” IEEE Trans. Software Eng., vol. 25, no. 4, pp. 510-525, 1999.
[42] M. Jørgensen and K. Moløkken, “Combination of Software Development Effort Prediction Intervals: Why, When and How?” Proc. 14th IEEE Conf. Software Eng. and Knowledge Eng. (SEKE '02), pp. 425-428, 2002.
[43] J.W. Taylor and D.W. Bunn, “Investigating Improvements in the Accuracy of Prediction Intervals for Combinations of Forecasts: A Simulation Study,” Int'l J. Forecasting, vol. 15, no. 3, pp. 325-339, 1999.
[44] R.C. Blattberg and S.J. Hoch, “Database Models and Managerial Intuition: 50 Percent Model + 50 Percent Manager,” Management Science, vol. 36, no. 8, pp. 887-899, 1990.
[45] G. Rowe and G. Wright, “Expert Opinions in Forecasting: The Role of the Delphi Process,” Principles of Forecasting: A Handbook for Researchers and Practitioners, J.S. Armstrong, ed., pp. 125-144, 2001.
[46] K. Moløkken-Østvold and M. Jørgensen, “Group Processes in Software Effort Estimation,” Empirical Software Eng., vol. 9, no. 4, pp. 315-334, 2004.
[47] I.L. Janis, Victims of Groupthink. Boston: Houghton Mifflin, 1972.
[48] N. Epley and T. Gilovich, “Just Going Along: Nonconscious Priming and Conformity to Social Pressure,” J. Experimental Social Psychology, vol. 35, pp. 578-589, 1999.
[49] N. Kogan and M.A. Wallach, Risk Taking: A Study In Cognition and Personality. New York: Rinehart and Winston, 1964.
[50] J.H. Davis, T. Kameda, and M.F. Stasson, “Group Risk Taking: Selected Topics,” Risk-Taking Behavior, J.F. Yates, ed., pp. 164-199, 1992.
[51] L.A. Maines, “An Experimental Examination of Subjective Forecast Combination,” Int'l J. Forecasting, vol. 12, no. 2, pp. 223-233, 1996.
[52] L.M. Taff, J.W. Borchering, and J.W.R. Hudgins, “Estimeetings: Development Estimates and a Front-End Process for a Large Project,” IEEE Trans. Software Eng., vol. 17, no. 8, pp. 839-849, 1991.
[53] J.A. Kernaghan and R.A. Cooke, “The Contribution of the Group Process to Successful Project Planning in R&D Settings,” IEEE Trans. Eng. Management, vol. 33, no. 3, pp. 134-140, 1986.
[54] S. Oskamp, “Overconfidence in Case-Study Judgments,” J. Consulting Psychology, vol. 29, no. 3, pp. 261-265, 1965.
[55] I.R. Newby-Clark, et al., “People Focus on Optimistic Scenarios and Disregard Pessimistic Scenarios When Predicting Task Completion Times,” J. Experimental Psychology: Applied, vol. 6, no. 3, pp. 171-182, 2000.
[56] C.R. Schwenk and R. Cosier, “Effects of the Expert, Devil's Advocate, and Dialectical Inquiry Methods on Prediction Performance,” Organizational Behaviour and Human Decision Processes, vol. 26, no. 3, pp. 409-424, 1980.
[57] R.A. Cosier and G.L. Rose, “Cognitive Conflict and Goal Conflict Effects on Task Performance,” Organizational Behaviour and Human Performance, vol. 19, no. 2, pp. 378-391, 1977.
[58] J.E. Sieber, “Effects of Decision Importance on Ability to Generate Warranted Subjective Uncertainty,” J. Personality and Social Psychology, vol. 30, no. 5, pp. 688-694, 1974.
[59] B.W. Pelham and E. Neter, “The Effect of Motivation of Judgment Depends on the Difficulty of the Judgment,” J. Personality and Social Psychology, vol. 68, no. 4, pp. 581-594, 1995.
[60] D.M. Grether and C.R. Plott, “Economic Theory of Choice and the Preference Reversal Phenomenon,” Am. Economic Rev., vol. 69, pp. 623-638, 1979.
[61] J.S. Lerner and P.P.E. Tetlock, “Accounting for the Effect of Accountability,” Psychological Bull., vol. 125, no. 2, pp. 255-275, 1999.
[62] C.F. Camerer and R.M. Hogarth, “The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework,” J. Risk and Uncertainty, vol. 19, no. 1, pp. 7-42, 1999.
[63] A.L. Lederer and J. Prasad, “Software Management and Cost Estimating Error,” J. Systems and Software, vol. 50, no. 1, pp. 33-42, 2000.
[64] K. Siegel-Jacobs and J.F. Yates, “Effects of Procedural and Outcome Accountability on Judgment Quality,” Organizational Behavior and Human Decision Processes, vol. 65, pp. 1-17, 1996.
[65] M. Jørgensen, “A Review of Studies on Expert Estimation of Software Development Effort,” J. Systems and Software, vol. 70, nos. 1-2, pp. 37-60, 2004.
[66] M.R. Lepper, D. Greene, and R.E. Nisbett, “Undermining Children's Intrinsic Interest with Extrinsic Rewards: A Test of the ‘Overjustification’ Hypothesis,” J. Personality and Social Psychology, vol. 28, pp. 129-137, 1973.
[67] R. Hagafors and B. Brehmer, “Does Having to Justify One's Judgments Change Nature of the Judgment Process?” Organizational Behaviour and Human Decision Processes, vol. 31, no. 2, pp. 223-232, 1983.
[68] S.C. Hora, J.A. Hora, and N.G. Dodd, “Assessment of Probability Distributions for Continous Random Variables: A Comparison of the Bisection and Fixed Value Methods,” Organizational Behavior and Human Decision Processes, vol. 51, pp. 223-232, 1992.
[69] P.P. Sedlmeier, Improving Statistical Reasoning: Theoretical Models and Practical Implications. Mahwah, N.J.: Erlbaum, 1999.
[70] K.H. Teigen and M. Jørgensen, “When 90 Percent Confidence Intervals are 50 Percent Certain: On the Credibility of Credible Intervals,” Applied Cognitive Psychology, 2005.
[71] T.D. Wilson and N. Brekke, “Mental Contamination and Mental Correction: Unwanted Influences on Judgments and Evaluation,” Psychological Bull., vol. 116, no. 1, pp. 117-142, 1994.
[72] S.A. Sloman and E. Al, “Frequency Illusions and Other Fallacies,” Organizational Behavior and Human Decisions Processes, vol. 91, pp. 296-309, 2003.
[73] S.C. Hora, J.A. Hora, and N.G. Dodd, “Assessment of Probability Distributions for Continous Random Variables: A Comparison of the Bisection and Fixed Value Methods,” Organizational Behavior and Human Decisions Processes, vol. 51, pp. 223-232, 1992.
[74] K.H. Teigen and M. Jørgensen, “When 90 Percent Confidence Intervals are 50 Percent Certain: On the Credibility of Credible Intervals,” Applied Cognitive Psychology, vol. 19, pp. 455-475, 2005.
[75] A. Winman, PP. Hanson, and PP. Juslin, “Subjective Probability Intervals: How to Reduce Overconfidence by Interval Estimation,” J. Experimental Psychology: Learning, Memory, and Cognition, vol. 30, pp. 1167-1175, 2004.

Index Terms:
Index Terms- Cost estimation, management, software psychology, uncertainty of software development cost.
Citation:
Magne J?rgensen, "Evidence-Based Guidelines for Assessment of Software Development Cost Uncertainty," IEEE Transactions on Software Engineering, vol. 31, no. 11, pp. 942-954, Nov. 2005, doi:10.1109/TSE.2005.128
Usage of this product signifies your acceptance of the Terms of Use.