This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Simulation Study of the Model Evaluation Criterion MMRE
November 2003 (vol. 29 no. 11)
pp. 985-995
ASCII Text
x 
 
Tron Foss, Erik Stensrud, Barbara Kitchenham, Ingunn Myrtveit, "A Simulation Study of the Model Evaluation Criterion MMRE," IEEE Transactions on Software Engineering, vol. 29, no. 11, pp. 985-995, November, 2003.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2003.1245300,
author = {Tron Foss and Erik Stensrud and Barbara Kitchenham and Ingunn Myrtveit},
title = {A Simulation Study of the Model Evaluation Criterion MMRE},
journal ={IEEE Transactions on Software Engineering},
volume = {29},
number = {11},
issn = {0098-5589},
year = {2003},
pages = {985-995},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2003.1245300},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - A Simulation Study of the Model Evaluation Criterion MMRE
IS - 11
SN - 0098-5589
SP985
EP995
EPD - 985-995
A1 - Tron Foss,
A1 - Erik Stensrud,
A1 - Barbara Kitchenham,
A1 - Ingunn Myrtveit,
PY - 2003
KW - Mean magnitude of relative error
KW - software metrics
KW - simulation
KW - regression analysis
KW - prediction models
KW - software cost estimation
KW - software engineering
KW - empirical software engineering
KW - prediction accuracy.
VL - 29
JA - IEEE Transactions on Software Engineering
ER -
 

Abstract—The Mean Magnitude of Relative Error, MMRE, is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. One purpose of MMRE is to assist us to select the best model. In this paper, we have performed a simulation study demonstrating that MMRE does not always select the best model. Our findings cast some doubt on the conclusions of any study of competing software prediction models that used MMRE as a basis of model comparison. We therefore recommend not using MMRE to evaluate and compare prediction models. At present, we do not have any universal replacement for MMRE. Meanwhile, we therefore recommend using a combination of theoretical justification of the models that are proposed together with other metrics proposed in this paper.

[1] L. Angelis and I. Stamelos, A Simulation Tool for Efficient Analogy Based Cost Estimation Empirical Software Eng., vol. 5, pp. 35-68, 2000.
[2] L.C. Briand, V.R. Basili, and W.M. Thomas, "A Pattern Recognition Approach for Software Engineering Data Analysis," IEEE Trans. Software Eng., vol. 18, no. 11, pp. 931-942, 1992.
[3] L.C. Briand,V.R. Basili,, and C.J. Hetmanski,“Developing interpretable models with optimized set reduction for identifying high-risk software components,” IEEE Transactions on Software Engineering, vol. 19, no. 11, pp. 1,028-1,044, Nov. 1993.
[4] L.C. Briand, K. El-Emam, and F. Bomarius, “COBRA: A Hybrid Method for Software Cost Estimation, Benchmarking and Risk Assessment,” Proc. 20th Int'l Conf. Software Eng., pp. 390-399, 1998.
[5] L.C. Briand, K. El-Emam, and I. Wieczorek, A Case Study in Productivity Benchmarking: Methods and Lessons Learned Proc. Ninth European Software Control and Metrics Conf., pp. 4-14, 1998.
[6] L.C. Briand, K. El-Emam, and I. Wieczorek, Explaining the Cost of European Space and Military Projects Proc. 21st Int'l Conf. Software Eng., pp. 303-312, 1999.
[7] L.C. Briand, K. El-Emam, K. Maxwell, D. Surmann, and I. Wieczorek, An Assessment and Comparison of Common Cost Software Project Estimation Methods Proc. 21st Int'l Conf. Software Eng., pp. 313-322, 1999.
[8] L.C. Briand, T. Langley, and I. Wieczorek, A Replicated Assessment and Comparison of Common Software Cost Modeling Techniques Proc. Int'l Conf. Software Eng., pp. 377-386, 2000.
[9] L.C. Briand and D. Pfahl, Using Simulation for Assessing the Real Impact of Test Coverage on Defect Coverage IEEE Trans. Reliability, vol. 49, no. 1, pp. 60-70, 2000.
[10] L.C. Briand and I. Wieczorek, Resource Modeling in Software Engineering Encyclopedia of Software Eng., second ed., J. Marciniak, ed., Wiley, in press.
[11] A.F. Chalmers, What is This Thing Called Science? second ed. Buckingham: Open Univ. Press, 1982.
[12] The COCOMO II Suite,http://sunset.usc.edu/research/cocomosuite index.html, 2002.
[13] S.D. Conte, H.E. Dunsmore, and V.Y. Shen, Software Engineering Metrics and Models.Menlo Park, Calif.: Benjamin/Cummings, 1986.
[14] A. Cuelenaere, M. van Genuchten, and F. Heemstra, "Calibrating a Software Cost Estimation Model: Why and How," Information and Software Technology, vol. 20, no. 10, pp. 558-567, 1987.
[15] J.M. Desharnais, Analyse Statistique de la Productivite des Projects de Developpement en Informatique a Partir de la Technique des Points de Fonction Master's thesis, Univ. du Quebec a Montreal, 1989.
[16] J.J. Dolado, On the Problem of the Software Cost Function Information Software Technology, vol. 43, no. 1, pp. 61-72, 2001.
[17] Encyclopedia of Statistical Sciences, S. Kotz, N.L. Johnson, and C.B. Read, eds., New York: Wiley, 1982-1998.
[18] D.V. Ferens, Software Cost Estimation in the DoD Environment Am. Programmer, pp. 28-34, July 1996.
[19] G. Finnie and G. Wittig, “A Comparison of Software Effort Estimation Techniques: Using Function Points with Neural Networks, Case-Based Reasoning and Regression Models,” J. Systems and Software, vol. 39, pp. 281-289, 1997.
[20] T. Foss, I. Myrtveit, and E. Stensrud, MRE and Heteroscedasticity Proc. 12th European Software Control and Metrics Conf., pp. 157-164, 2001.
[21] T. Foss, I. Myrtveit, and E. Stensrud, A Comparison of LAD and OLS Regression for Effort Prediction of Software Projects Proc. 12th European Software Control and Metrics Conf., pp. 9-15, 2001.
[22] K. El-Emam, The Predictive Validity Criterion for Evaluating Binary Classifiers Proc. Second Int'l Software Metrics Symp., pp. 235-244, 1998.
[23] T. Foss, I. Myrtveit, and E. Stensrud, A Comparison of LAD and OLS Regression for Effort Prediction of Software Projects Proc. 12th European Software Control and Metrics Conf., pp. 9-15, 2001.
[24] A.R. Gray and S.G. MacDonell, Software Metrics Data Analysis Exploring the Relative Performance of Some Commonly Used Modeling Techniques Empirical Software Eng., vol. 4, pp. 297-316, 1999.
[25] D.N. Gujarati, Basic Econometrics, third ed. New York: McGraw-Hill, 1995.
[26] R. Jeffery, M. Ruhe, and I. Wieczorek, “Using Public Domain Metrics to Estimate Software Development Effort,” Proc. Seventh IEEE Int'l Metrics Symp., 2001.
[27] R. Jeffery and F. Walkerden, Analogy, Regression and Other Methods for Estimating Effort and Software Quality Attributes Proc. European Conf. Optimizing Software Development and Maintenance, pp. 37-46, 1999.
[28] R.L. Jenson and J.W. Bartley, Parametric Estimation of Programming Effort: An Object-Oriented Model J. Systems and Software, vol. 15, pp. 107-114, 1991.
[29] M. Jørgensen, “Experience with the Accuracy of Software Maintenance Task Effort Prediction Models,” IEEE Trans. Software Eng., vol. 21, no. 8, pp. 674–681, Aug. 1995.
[30] C. Kemerer, "An Empirical Validation of Software Cost Estimation Models," Comm. ACM, vol. 30, pp. 416-429, May 1987.
[31] B.A. Kitchenham, S.G. MacDonell, L.M. Pickard, and M.J. Shepperd, What Accuracy Statistics Really Measure IEE Proc. Software, vol. 148, no. 3, pp. 81-85, 2001.
[32] B.A. Kitchenham and K. Kansala, “Inter-Item Correlations Among Function Points,” Proc. First Int'l Software Metrics Symp., pp. 11-14, 1993.
[33] B. Kitchenham, "A Procedure for Analyzing Unbalanced Datasets," IEEE Trans. Software Eng., vol. 24, no. 4, Apr. 1998, pp. 278—300.
[34] A. Koutsoyiannis, Theory of Econometrics, second ed. London: MacMillan, 1977.
[35] S. Makridakis, S.C. Wheelwright, and R.J. Hyndman, Forecasting Methods and Applications, third ed. John Wiley&Sons Inc., 1998.
[36] R. Martin, "Evaluation of Current Software Costing Tools," Software Eng. Notes, vol. 13, no. 3, pp. 49-51, 1988.
[37] Minitab Statistical Software Release 13,www.minitab.com, 2000.
[38] Y. Miyazaki, M. Terakado, K. Ozaki, and H. Nozaki, “Robust Regression for Developing Software Estimation Models,” J. Systems and Software, vol. 27, pp. 3-16, 1994.
[39] T. Mukhopadhyay, S.S. Vicinanza, and M.J. Prietula, "Examining the Feasibility of a Case-Based Reasoning Model for Software Effort Estimation," MIS Quarterly, vol. 16, pp. 155-171, June, 1992.
[40] I. Myrtveit and E. Stensrud, "A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models," IEEE Trans. Software Engineering, vol. 25, no. 4, Jul./Aug. 1999, pp. 510-525.
[41] I. Myrtveit and E. Stensrud, “Benchmarking COTS Projects Using Data Envelopment Analysis,” Proc. METRICS'99, pp. 269-278, 1999.
[42] P. Nesi and T. Querci, "Effort Estimation and Prediction of Object-Oriented Systems," J. Systems and Software, to appear, 1998.
[43] L. Pickard, B. Kitchenham, and S.J. Linkman, Using Simulated Data Sets to Compare Data Analysis Techniques Used for Software Cost Modelling IEE Proc. Software, vol. 148, no. 6, pp. 165-174, Dec. 2001.
[44] J. Rosenberg, “Some Misconceptions About Lines of Code,” Proc. Fourth Int'l Software Metrics Symp., 1997.
[45] B. Samson, D. Ellison, and P. Dugard, Software Cost Estimation Using and Albus Perceptron (CMAC) Information and Software Technology, vol. 39, pp. 55-60, 1997.
[46] M. Shepperd and G. Kadoda, Using Simulation to Evaluate Predition Techniques Proc. Proc. Fifth Int'l Software Metrics Symp., pp. 349-359, 2001.
[47] M.J. Shepperd and C. Schofield, “Estimating Software Project Effort Using Analogies,” IEEE Trans. Software Eng., vol. 23, pp. 736-743, 1997.
[48] K. Srinivasan and D. Fisher, “Machine Learning Approaches to Estimating Software Development Effort,” IEEE Trans. Software Eng., vol. 21, no. 2, pp. 126–137, Feb. 1995.
[49] E. Stensrud and I. Myrtveit, “Human Performance Estimating with Analogy and Regression Models: An Empirical Validation,” Proc. METRICS'98, pp. 205-213, 1998.
[50] E. Stensrud, T. Foss, B. Kitchenham, and I. Myrtveit, A Further Empirical Investigation of the Relationship between MRE and Project Size Empirical Software Eng., 2002.
[51] K. Strike, K. El-Emam, and N. Madhavji, Software Cost Estimation with Incomplete Data IEEE Trans. Software Eng., vol. 27, no. 10, pp. 890-908, Oct. 2001.
[52] F. Walkerden and R. Jeffery, An Empirical Study of Analogy-based Software Effort Estimation Empirical Software Eng., vol. 4, no. 2, pp. 135-158, 1999.

Index Terms:
Mean magnitude of relative error, software metrics, simulation, regression analysis, prediction models, software cost estimation, software engineering, empirical software engineering, prediction accuracy.
Citation:
Tron Foss, Erik Stensrud, Barbara Kitchenham, Ingunn Myrtveit, "A Simulation Study of the Model Evaluation Criterion MMRE," IEEE Transactions on Software Engineering, vol. 29, no. 11, pp. 985-995, Nov. 2003, doi:10.1109/TSE.2003.1245300
Usage of this product signifies your acceptance of the Terms of Use.