This Article 
 Bibliographic References 
 Add to: 
Selecting Best Practices for Effort Estimation
November 2006 (vol. 32 no. 11)
pp. 883-895
Effort estimation often requires generalizing from a small number of historical projects. Generalization from such limited experience is an inherently underconstrained problem. Hence, the learned effort models can exhibit large deviations that prevent standard statistical methods (e.g., t-tests) from distinguishing the performance of alternative effort-estimation methods. The COSEEKMO effort-modeling workbench applies a set of heuristic rejection rules to comparatively assess results from alternative models. Using these rules, and despite the presence of large deviations, COSEEKMO can rank alternative methods for generating effort models. Based on our experiments with COSEEKMO, we advise a new view on supposed "best practices” in model-based effort estimation: 1) Each such practice should be viewed as a candidate technique which may or may not be useful in a particular domain, and 2) tools like COSEEKMO should be used to help analysts explore and select the best method for a particular domain.

[1] K. Lum, J. Powell, and J. Hihn, “Validation of Spacecraft Cost Estimation Models for Flight and Ground Systems,” Proc. Conf. Int'l Soc. Parametric Analysts (ISPA), Software Modeling Track, May 2002.
[2] M. Jorgensen, “A Review of Studies on Expert Estimation of Software Development Effort,” J. Systems and Software, vol. 70, nos.1-2, pp. 37-60, 2004.
[3] B. Boehm, Software Engineering Economics. Prentice Hall, 1981.
[4] B. Boehm, E. Horowitz, R. Madachy, D. Reifer, B.K. Clark, B. Steece, A.W. Brown, S. Chulani, and C. Abts, Software Cost Estimation with Cocomo II. Prentice Hall, 2000.
[5] B.B.S. Chulani, B. Clark, and B. Steece, “Calibration Approach and Results of the Cocomo II Post-Architecture Model,” Proc. Conf. Int'l Soc. Parametric Analysts (ISPA), 1998.
[6] S. Chulani, B. Boehm, and B. Steece, “Bayesian Analysis of Empirical Software Engineering Cost Models,” IEEE Trans. Software Eng., vol. 25, no. 4, July/Aug. 1999.
[7] C. Kemerer, “An Empirical Validation of Software Cost Estimation Models,” Comm. ACM, vol. 30, no. 5, pp. 416-429, May 1987.
[8] R. Strutzke, Estimating Software-Intensive Systems: Products, Projects and Processes. Addison Wesley, 2005.
[9] M. Shepperd and C. Schofield, “Estimating Software Project Effort Using Analogies,” IEEE Trans. Software Eng., vol. 23, no. 12, , Dec. 1997.
[10] T. Menzies, D. Port, Z. Chen, J. Hihn, and S. Stukes, “Validation Methods for Calibrating Software Effort Models,” Proc. Int'l Conf. Software Eng. (ICSE), rbanker/SE_XII.pdfhttp:/ /, 2005.
[11] Z. Chen, T. Menzies, and D. Port, “Feature Subset Selection Can Improve Software Cost Estimation,” Proc. PROMISE Workshop, Int'l Conf. Software Eng. (ICSE),, 2005.
[12] Z. Chen, T. Menzies, D. Port, and B. Boehm, “Finding the Right Data for Software Cost Modeling,” IEEE Software, Nov. 2005.
[13] “Certified Parametric Practioner Tutorial,” Proc. 2006 Int'l Conf. Int'l Soc. Parametric Analysts (ISPA), 2006.
[14] A. Miller, Subset Selection in Regression, second ed. Chapman & Hall, 2002.
[15] C. Kirsopp and M. Shepperd, “Case and Feature Subset Selection in Case-Based Software Project Effort Prediction,” Proc. 22nd SGAI Int'l Conf. Knowledge-Based Systems and Applied Artificial Intelligence, 2002.
[16] M. Jorgensen and K. Molokeen-Ostvoid, “Reasons for Software Effort Estimation Error: Impact of Respondent Error, Information Collection Approach, and Data Analysis Method,” IEEE Trans. Software Eng., vol. 30, no. 12, Dec. 2004.
[17] R. Park, “The Central Equations of the Price Software Cost Model,” Proc. Fourth COCOMO Users Group Meeting, Nov. 1988.
[18] R. Jensen, “An Improved Macrolevel Software Development Resource Estimation Model,” Proc. Fifth Conf. Int'l Soc. Parametric Analysts (ISPA), pp. 88-92, Apr. 1983.
[19] L. Putnam and W. Myers, Measures for Excellence. Yourdon Press Computing Series, 1992.
[20] V. Basili, F. McGarry, R. Pajerski, and M. Zelkowitz, “Lessons Learned from 25 Years of Process Improvement: The Rise and Fall of the NASA Software Engineering Laboratory,” Proc. 24th Int'l Conf. Software Eng. (ICSE '02), papers83.88.pdf, 2002.
[21] T. Jones, Estimating Software Costs. McGraw-Hill, 1998.
[22] J. Kliijnen, “Sensitivity Analysis and Related Analyses: A Survey of Statistical Techniques,” J. Statistical Computation and Simulation, vol. 57, nos. 1-4, pp. 111-142, 1997.
[23] D. Ferens and D. Christensen, “Calibrating Software Cost Models to Department of Defense Database: A Review of Ten Studies,” J.Parametrics, vol. 18, no. 1, pp. 55-74, Nov. 1998.
[24] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann, 1999.
[25] J.R. Quinlan, “Learning with Continuous Classes,” Proc. Fifth Australian Joint Conf. Artificial Intelligence, pp. 343-348, 1992.
[26] R. Kohavi and G.H. John, “Wrappers for Feature Subset Selection,” Artificial Intelligence, vol. 97, no. 1-2, pp. 273-324, 1997.
[27] M. Hall and G. Holmes, “Benchmarking Attribute Selection Techniques for Discrete Class Data Mining,” IEEE Trans. Knowledge and Data Eng., vol. 15, no. 6, pp. 1437-1447, Nov.-Dec. 2003.
[28] P. Cohen, Empirical Methods for Artificial Intelligence. MIT Press, 1995.
[29] I.H. Witten and E. Frank, Data Mining, second ed. Morgan Kaufmann, 2005.
[30] S. Stukes and D. Ferens, “Software Cost Model Calibration,” J.Parametrics, vol. 18, no. 1, pp. 77-98, 1998.
[31] S. Stukes and H. Apgar, “Applications Oriented Software Data Collection: Software Model Calibration Report TR-9007/549-1,” Management Consulting and Research, Mar. 1991.
[32] S. Chulani, B. Boehm, and B. Steece, “From Multiple Regression to Bayesian Analysis for Calibrating COCOMO II,” J. Parametrics, vol. 15, no. 2, pp. 175-188, 1999.
[33] H. Habib-agahi, S. Malhotra, and J. Quirk, “Estimating Software Productivity and Cost for NASA Projects,” J. Parametrics, pp. 59-71, Nov. 1998.
[34] T. Ho, J. Hull, and S. Srihari, “Decision Combination in Multiple Classifier Systems,” IEEE Trans Pattern Analysis and Machine Intelligence, vol. 16, no. 1, pp. 66-75, Jan. 1994.
[35] F. Provost and T. Fawcett, “Robust Classification for Imprecise Environments,” Machine Learning, vol. 42, no. 3, Mar. 2001.
[36] O.T. Yildiz and E. Alpaydin, “Ordering and Finding the Best of $k > 2$ Supervised Learning Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 3, pp. 392-402, Mar. 2006.
[37] L. Brieman, “Bagging Predictors,” Machine Learning, vol. 24, no. 2, pp. 123-140, 1996.

Index Terms:
Model-based effort estimation, COCOMO, deviation, data mining.
Tim Menzies, Zhihao Chen, Jairus Hihn, Karen Lum, "Selecting Best Practices for Effort Estimation," IEEE Transactions on Software Engineering, vol. 32, no. 11, pp. 883-895, Nov. 2006, doi:10.1109/TSE.2006.114
Usage of this product signifies your acceptance of the Terms of Use.