This Article 
 Bibliographic References 
 Add to: 
A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models
July/August 1999 (vol. 25 no. 4)
pp. 510-525

Abstract—To have general validity, empirical results must converge. To be credible, an experimental science must understand the limitations and be able to explain the disagreements of empirical results. We describe an experiment to replicate previous studies which claim that estimation by analogy outperforms regression models. In the experiment, 68 experienced practitioners each estimated a project from a dataset of 48 industrial COTS projects. We applied two treatments, an analogy tool and a regression model, and we used the estimating performance when aided by the historical data as the control. We found that our results do not converge with previous results. The reason is that previous studies have used other datasets and partially different data analysis methods, and last but not least, the tools have been validated in isolation from the tool users. This implies that the results are sensitive to the experimental design: the characteristics of the dataset, the norms for removing outliers and other data points from the original dataset, the test metrics, significance levels, and the use of human subjects and their level of expertise. Thus, neither our results nor previous results are robust enough to claim any general validity.

[1] COCOMO II Model Definition Manual, version 1.4, Univ. of Southern California, , 1997.
[2] F.P. Brooks, Jr., "The Computer Scientist as Toolsmith II," Comm. ACM, Vol. 39, No. 3, Mar. 1996, pp. 61-68.
[3] IFPUG Function Point Counting Practices: Manual Release 4.0, Int'l Function Point Users' Group, Westerville, Ohio, 1994.
[4] M.J. Shepperd, C. Schofield, and B.A. Kitchenham, “Effort Estimation Using Analogy,” Proc. 18th Int'l Conf. Software Eng., 1996.
[5] M.J. Shepperd and C. Schofield, “Estimating Software Project Effort Using Analogies,” IEEE Trans. Software Eng., vol. 23, pp. 736-743, 1997.
[6] E. Stensrud and I. Myrtveit, “The Added Value of Estimation by Analogy—An Industrial Experiment,” Proc. The European Software Measurement Conf., FESMA'98, pp. 549–556, Antwerp, Belgium, May 1998.
[7] E. Stensrud and I. Myrtveit, “Human Performance Estimating with Analogy and Regression Models: An Empirical Validation,” Proc. METRICS'98, pp. 205-213, 1998.
[8] M. Zelkowitz and D. Wallace, “Experimental Models for Validating Technology,” Computer, vol. 31, no. 5, pp. 23–31, May 1998.

Index Terms:
Software cost estimation, commercial off-the-shelf (COTS) software projects, multivariate regression analysis, estimation by analogy, human performance, controlled experiment, enterprise resource planning (ERP) systems.
Ingunn Myrtveit, Erik Stensrud, "A Controlled Experiment to Assess the Benefits of Estimating with Analogy and Regression Models," IEEE Transactions on Software Engineering, vol. 25, no. 4, pp. 510-525, July-Aug. 1999, doi:10.1109/32.799947
Usage of this product signifies your acceptance of the Terms of Use.