Subscribe
Issue No.03 - May/June (2008 vol.34)
pp: 336-356
Syed W. Haider , The University of Texas at Dallas, Dallas
João W. Cangussu , The University of Texas at Dallas, Dallas
Kendra M.L. Cooper , The University of Texas at Dallas, Dallas
Ram Dantu , University of North Texas, Denton
Syed Haider , The University of Texas at Dallas, Dallas
ABSTRACT
An accurate prediction of the number of defects in a software product during system testing contributes not only to the management of the system testing process but also to the estimation of the product's required maintenance. Here, a new approach called ED^3M is presented that computes an estimate of the total number of defects in an ongoing testing process. ED^3M is based on estimation theory. Unlike many existing approaches the technique presented here does not depend on historical data from previous projects or any assumptions about the requirements and/or testers' productivity. It is a completely automated approach that relies only on the data collected during an ongoing testing process. This is a key advantage of the ED^3M approach, as it makes it widely applicable in different testing environments. Here, the ED^3M approach has been evaluated using five data sets from large industrial projects and two data sets from the literature. In addition, a performance analysis has been conducted using simulated data sets to explore its behavior using different models for the input data. The results are very promising; they indicate the ED^3M approach provides accurate estimates with as fast or better convergence time in comparison to well-known alternative techniques, while only using defect data as the input.
INDEX TERMS
Statistical methods, Testing and Debugging, Metrics/Measurement, Defect prediction, system testing, estimation theory
CITATION
Syed W. Haider, João W. Cangussu, Kendra M.L. Cooper, Ram Dantu, Syed Haider, "Estimation of Defects Based on Defect Decay Model: ED^{3}M", IEEE Transactions on Software Engineering, vol.34, no. 3, pp. 336-356, May/June 2008, doi:10.1109/TSE.2008.23
REFERENCES
 [1] V. Basili and B. Perricone, “Software Errors and Complexity: An Empirical Investigation,” Comm. ACM, vol. 27, no. 1, pp. 42-52, 1984. [2] B.T. Compton and C. Withrow, “Prediction and Control of ADA Software Defects,” J. Systems and Software, vol. 12, pp. 199-207, July 1990. [3] A.E. Ferdinand, “A Theory of System Complexity,” Int'l J. General Systems, vol. 1, pp. 19-33, 1974. [4] L.C. Briand, K.E. Emam, B.G. Freimut, and O. Laitenberger, “A Comprehensive Evaluation of Capture-Recapture Models for Estimating Software Defect Content,” IEEE Trans. Software Eng., vol. 26, pp. 518-540, June 2000. [5] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, “Locating where Faults Will Be,” Richard Tapia Celebration of Diversity in Computing Conf., pp. 48-50, Oct. 2005. [6] P. Zhang and A. Mockus, “On Measurement and Understanding of Software Development Processes,” Technical Report ALR-2002-048, Avaya Research Labs, Nov. 2002. [7] N. Fenton, M. Neil, W. Marsh, P. Hearty, D. Marquez, P. Krause, and R. Mishra, “Predicting Software Defects in Varying Development Lifecycles Using Bayesian Nets,” Information and Software Technology, vol. 49, pp. 32-43, Jan. 2007. [8] N. Fenton, P. Krause, and M. Neil, “Software Measurement: Uncertainty and Causal Modeling,” IEEE Software, vol. 19, no. 4, pp. 116-122, July-Aug. 2002. [9] J.-J. Gras, R. Gupta, and E. Pérez-Miñana, “Generating a Test Strategy with Bayesian Networks and Common Sense,” Proc. Testing: Academia and Industry Conf.—Practice and Research Techniques, pp. 29-40, Aug. 2006. [10] J. Musa, Software Reliability Engineering. McGraw-Hill, 1999. [11] A.L. Goel, “Software Reliability Models: Assumptions, Limitations, and Applicability,” IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1411-1423, Dec. 1985. [12] A. Leon-Garcia, Probability and Random Processes for Electrical Engineering, second ed. Addison-Wesley, May 1994. [13] S.M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice Hall PTR, 1993. [14] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, “A Formal Model for the Software Test Process,” IEEE Trans. Software Eng., vol. 28, no. 8, pp. 782-796, Aug. 2002. [15] D.E. Knuth, “The Errors of $T_{E}X$ ,” Software—Practice and Experience, vol. 19, no. 7, pp. 607-685, 1989. [16] S.R. Dalal and C.L. Mallows, “Some Graphical Aids for Deciding when to Stop Testing Software,” IEEE J. Selected Areas in Comm., vol. 8, pp. 169-175, Feb. 1990. [17] R. Jacoby and Y. Tohma, “The Hyper-Geometric Distribution Software Reliability Growth Model (HGDM): Precise Formulation and Applicability,” 1990. [18] W.K. Ehrlich, J.P. Stampfel, and J.R. Wu, “Application of Software Reliability Modeling to Product Quality and Test Process,” Proc. 12th Int'l Conf. Software Eng., pp. 108-116, Mar. 1990. [19] J.W. Cangussu, R.A. DeCarlo, and A.P. Mathur, “Using Sensitivity Analysis to Validate a State Variable Model of the Software Test Process,” IEEE Trans. Software Eng., vol. 29, no. 5, pp. 430-443, May 2003. [20] J.W. Cangussu, R.M. Karcich, R.A. DeCarlo, and A.P. Mathur, “Software Release Control Using Defect Based Quality Estimation,” Proc. 15th Int'l Symp. Software Reliability Eng., Nov. 2004. [21] K.O. John and D. Musa, “A Logarithmic Poisson Execution Time Model for Software Reliability Measurement,” Proc. Seventh Int'l Conf. Software Eng., pp. 230-238, Mar. 1984. [22] J.D. Musa, A. Iannino, and K. Okumoto, Software Reliability Measurement, Prediction, Application. McGraw-Hill, 1987. [23] A.P. Nikora, Computer Aided Software Reliability Estimation (CASRE) User's Guide, Open Channel Foundation, NASA, Jet Propulsion Laboratory, Mar. 2003. [24] G. Seber and C. Wild, Nonlinear Regression. John Wiley & Sons, 2003. [25] N.E. Fenton and M. Neil, “A Critique of Software Defect Prediction Models,” IEEE Trans. Software Eng., vol. 25, no. 5, pp.675-689, Sept.-Oct. 1999. [26] V.Y. Shen, T. Yu, S.M. Thebaut, and L.R. Paulsen, “Identifying Error-Prone Software—An Empirical Study,” IEEE Trans. Software Eng., vol. 11, no. 4, pp. 317-323, Apr. 1985. [27] K.H. Moeller and D. Paulish, “An Empirical Investigation of Software Fault Distribution,” Proc. First Int'l Software Metrics Symp., pp. 82-90, May 1993. [28] L. Hatton, “Re-Examining the Fault Density-Component Size Connection,” IEEE Software, vol. 14, no. 2, pp. 89-98, Mar.-Apr. 1997. [29] J.R. Gaffney, “Estimating the Number of Faults in Code,” IEEE Trans. Software Eng., vol. 10, pp. 459-464, July 1984. [30] M.D. Neil, “Statistical Modelling of Software Metrics,” PhD thesis, South Bank Univ. and Strathclyde Univ., 1992. [31] S. Henry and D. Kafura, “The Evaluation of Software System's Structure Using Quantitative Software Metrics,” Software—Practice and Experience, vol. 14, pp. 561-573, June 1984. [32] N. Ohlsson and H. Alberg, “Predicting Error-Prone Software Modules in Telephone Switches,” IEEE Trans. Software Eng., vol. 22, no. 12, pp. 886-894, Dec. 1996. [33] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, “Predicting the Location and Number of Faults in Large Software Systems,” IEEE Trans. Software Eng., vol. 31, no. 4, pp. 340-355, Apr. 2005. [34] R.M. Bell, T.J. Ostrand, and E.J. Weyuker, “Looking for Bugs in All the Right Places,” Proc. Int'l Symp. Software Testing and Analysis, pp. 61-72, July 2006. [35] F. Padberg, “A Fast Algorithm to Compute Maximum Likelihood Estimates for the Hypergeometric Software Reliability Model,” Proc. Second Asia-Pacific Conf. Quality Software, pp. 40-49, Dec. 2001. [36] F. Padberg, “Maximum Likelihood Estimates for the Hypergeometric Software Reliability Model,” Int'l J. Reliability, Quality, and Safety Eng., July 2002. [37] C. Bai, K.-Y. Cai, and T.Y. Chen, “An Efficient Defect Estimation Method for Software Defect Curves,” Proc. 27th Ann. Int'l Computer Software and Applications Conf., Nov. 2003. [38] P. Zhang and A. Mockus, “On Measurement and Understanding of Software Development Processes,” Technical Report ALR-2002-048, Avaya Research Labs, Nov. 2002. [39] P. Zhang, A. Mockus, and D. Weiss, “Understanding and Predicting Effort in Software Projects,” Proc. 25th Int'l Conf. Software Eng., pp. 274-284, 2003. [40] D. Satoh and S. Yamada, “Discrete Equations and Software Reliability Growth Models,” Proc. 12th Int'l Symp. Software Reliability Eng., pp. 176-184, Nov. 2001. [41] A. Onoma and T. Yamaura, “Practical Steps toward Quality Development,” IEEE Software, vol. 12, no. 4, pp. 68-77, July 1995. [42] R.-H. Hou, S.-Y. Kuo, and Y.-P. Chang, “Applying Various Learning Curves to Hyper-Geometric Distribution Software Reliability Growth Model,” Proc. Fifth Int'l Symp. Software Reliability Eng., pp. 18-24, Nov. 1994. [43] J.D. Musa, “Software Reliability Data,” technical report, Data and Analysis Center for Software, 1979. [44] R. AL-Ekram, “Software Reliability Growth Modeling and Prediction,” Technical Report ECE 750, Dept. of Computer Science, Univ. of Waterloo, Mar. 2002. [45] K.T.S.S. Gokhale and M.R. Lyu, “Model Validation Using Simulated Data,” Proc. First IEEE Workshop Application-Specific Software Eng. and Technology, pp. 22-27, Mar. 1998. [46] I. Burnstein, T. Suwannasart, and C.R. Carlson, “Developing a Testing Maturity Model: Part 1,” CrossTalk, J. Defense Software Eng., vol. 9, pp. 21-24, Aug. 1996. [47] I. Burnstein, T. Suwannasart, and C.R. Carlson, “Developing a Testing Maturity Model: Part 2,” CrossTalk, J. Defense Software Eng., vol. 9, pp. 19-26, Sept. 1996. [48] I. Burnstein, A. Homyen, T. Suwanassart, G. Saxena, and R. Grom, “A Testing Maturity Model for Software Test Process Assessment and Improvement,” Software Quality Professional, vol. 1, Sept. 1999. [49] D.G. Luenberger, Introduction to Dynamic Systems: Theory, Models and Applications. John Wiley & Sons, 1979.