This Article 
 Bibliographic References 
 Add to: 
Comparing Software Prediction Techniques Using Simulation
November 2001 (vol. 27 no. 11)
pp. 1014-1022

Abstract—The need for accurate software prediction systems increases as software becomes much larger and more complex. A variety of techniques have been proposed; however, none has proven consistently accurate and there is still much uncertainty as to what technique suits which type of prediction problem. We believe that the underlying characteristics—size, number of features, type of distribution, etc.—of the data set influence the choice of the prediction system to be used. For this reason, we would like to control the characteristics of such data sets in order to systematically explore the relationship between accuracy, choice of prediction system, and data set characteristic. Also, in previous work, it has proven difficult to obtain significant results over small data sets. Consequently, it would be useful to have a large validation data set. Our solution is to simulate data allowing both control and the possibility of large (1,000) validation cases. In this paper, we compared four prediction techniques: regression, rule induction, nearest neighbor (a form of case-based reasoning), and neural nets. The results suggest that there are significant differences depending upon the characteristics of the data set. Consequently, researchers should consider prediction context when evaluating competing prediction systems. We also observed that the more “messy” the data and the more complex the relationship with the dependent variable, the more variability in the results. In the more complex cases, we observed significantly different results depending upon the particular training set that has been sampled from the underlying data set. This suggests that researchers will need to exercise caution when comparing different approaches and utilize procedures such as bootstrapping in order to generate multiple samples for training purposes. However, our most important result is that it is more fruitful to ask which is the best prediction system in a particular context rather than which is the “best” prediction system.

[1] M.J. Shepperd and C. Schofield, “Estimating Software Project Effort Using Analogies,” IEEE Trans. Software Eng., vol. 23, pp. 736-743, 1997.
[2] F. Niessink and H. van Vliet, “Predicting Maintenance Effort with Function Points,” Proc. Int'l Conf. Software Maintenance, 1997.
[3] L. Briand, T. Langley, and I. Wieczorek, “Using the European Space Agency Dataset: A Replicated Assessment and Comparison of Common Software Cost Modeling Techniques,” Proc. 22nd IEEE Int'l Conf. Software Eng., 2000.
[4] I. Myrtveit and E. Stensrud, "A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models," IEEE Trans. Software Engineering, vol. 25, no. 4, Jul./Aug. 1999, pp. 510-525.
[5] M.J. Shepperd, M.H. Cartwright, and G.F. Kadoda, “On Building Prediction Systems for Software Engineers,” Empirical Software Eng., vol. 5, pp. 175-182, 2000.
[6] B.A. Kitchenham and N.R. Taylor, “Software Cost Models,” ICL Technical J., vol. 4, pp. 73-102, 1984.
[7] J. Miller, “Can Results from Software Engineering Experiments be Safely Combined?,” Proc. IEEE Sixth Int'l Metrics Symp., 1999.
[8] L. Pickard, B. Kitchenham, and S. Linkman, “An Investigation Analysis Techniques for Software Datasets,” Proc. Sixth IEEE Int'l Software Metrics Symp., 1999.
[9] R. Gulezian, “Reformulating and Calibrating COCOMO,” J. Systems Software, vol. 16, pp. 235-242, 1991.
[10] P. Kok, B.A. Kitchenham, and J. Kirakowski, “The MERMAID Approach to Software Cost Estimation,” Esprit Technical Week, 1990.
[11] S.G. MacDonell, M.J. Shepperd, and P.J. Sallis, “Metrics for Database Systems: An Empirical Study,” Proc. Fourth IEEE Int'l Metrics Symp., 1997.
[12] SPSS for Windows—Base System User's Guide, Release 6., SPSS Inc., 1993.
[13] J. Neter, W. Wasserman, and M.H. Kutner, Applied Linear Statistical Models: Regression, Analysis of Variance, and Experimental Designs. Homewood, Ill: Irwin, 1985.
[14] J.R. Quinlan, C4.5: Programs for Machine Learning,San Mateo, Calif.: Morgan Kaufman, 1992.
[15] J.R. Quinlan, “Learning Decision Tree Classifiers,” ACM Computing Surveys, vol. 28, pp. 71-72, 1996.
[16] Clementine User Guide—Version 5, Integral Solutions Limited, 1998.
[17] M.A. de Almeida, H. Lounis, and W.L. Melo, “An Investigation on the Use of Machine Learned Models for Estimating Correction Costs,” Proc. IEEE Int'l Conf. Software Eng., 1998.
[18] R.W. Selby and A.A. Porter,“Learning from examples: Generation and evaluation of decision trees for software resource analysis,” IEEE Trans. Software Engineering, vol. 14, no. 12, pp. 1,743-1,757, Dec. 1988.
[19] B. Boehm, Software Engineering Economics, Prentice Hall, Upper Saddle River, N.J., 1981, pp. 533-535.
[20] A.J.C. Cowderoy and J.O. Jenkins, “Cost Estimation by Analogy as a Good Management Practice,” Software Engineering 88, 1988.
[21] T. Mukhopadhyay, S.S. Vicinanza, and M.J. Prietula, "Examining the Feasibility of a Case-Based Reasoning Model for Software Effort Estimation," MIS Quarterly, vol. 16, pp. 155-171, June, 1992.
[22] S. Vicinanza, M.J. Prietula, and T. Mukhopadhyay, “Case-Based Reasoning in Effort Estimation,” Proc. 11th Int'l Conf. Information Systems, 1990.
[23] D. Leake, “CBR in Context: The Present and the Future,” Case Based Reasoning: Experiences, Lessons and Future Directions, D. Leake, ed. Menlo Park: AAAI Press, pp. 1-35, 1996.
[24] G. Wittig and G. Finnie, “Estimating Software Development Effort with Connectionists Models,” Information&Software Technology, vol. 39, pp. 469-476, 1997.
[25] B. Efron and G. Gong, “A Leisurely Look at the Bootstrap, the Jackknife and Cross-Validation,” The Am. Statistician, vol. 37, pp. 36-48, 1983.

Index Terms:
Prediction system, simulation, machine learning, data set characteristic.
Martin Shepperd, Gada Kadoda, "Comparing Software Prediction Techniques Using Simulation," IEEE Transactions on Software Engineering, vol. 27, no. 11, pp. 1014-1022, Nov. 2001, doi:10.1109/32.965341
Usage of this product signifies your acceptance of the Terms of Use.