The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.04 - April (2013 vol.39)
pp: 462-476
Tihana Galinac Grbac , Univesity of Rijeka, Rijeka
Per Runeson , Lund University, Lund
Darko Huljenić , Ericsson Nikola Tesla, Zagreb
ABSTRACT
Background: Software engineering is searching for general principles that apply across contexts, for example, to help guide software quality assurance. Fenton and Ohlsson presented such observations on fault distributions, which have been replicated once. Objectives: We aimed to replicate their study again to assess the robustness of the findings in a new environment, five years later. Method: We conducted a literal replication, collecting defect data from five consecutive releases of a large software system in the telecommunications domain, and conducted the same analysis as in the original study. Results: The replication confirms results on unevenly distributed faults over modules, and that fault proneness distributions persist over test phases. Size measures are not useful as predictors of fault proneness, while fault densities are of the same order of magnitude across releases and contexts. Conclusions: This replication confirms that the uneven distribution of defects motivates uneven distribution of quality assurance efforts, although predictors for such distribution of efforts are not sufficiently precise.
INDEX TERMS
Testing, Measurement, Context, Software, Software engineering, Complexity theory, Telecommunications, replication, Software fault distributions, software metrics, empirical research
CITATION
Tihana Galinac Grbac, Per Runeson, Darko Huljenić, "A Second Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems", IEEE Transactions on Software Engineering, vol.39, no. 4, pp. 462-476, April 2013, doi:10.1109/TSE.2012.46
REFERENCES
[1] C. Andersson and P. Runeson, "A Spiral Process Model for Case Studies on Software Quality Monitoring—Method and Metrics," Software Process Improvement Practice, vol. 12, no. 2, pp. 125-140, Mar./Apr. 2007.
[2] C. Andersson and P. Runeson, "A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems," IEEE Trans. Software Eng., vol. 33, no. 5, pp. 273-286, May 2007.
[3] V.R. Basili and B.T. Perricone, "Software Errors and Complexity: An Empirical Investigation," Comm. ACM, vol. 27, no. 1, pp. 42-52, Jan. 1984.
[4] B.T. Compton and C. Withrow, "Prediction and Control of ADA Software Defects," J. Systems Software, vol. 12, no. 3, pp. 199-207, July 1990.
[5] M. D'Ambros, M. Lanza, and R. Robbes, "Evaluating Defect Prediction Approaches: A Benchmark and an Extensive Comparison," Empirical Software Eng., vol. 17, nos. 4/5, pp. 531-577, Aug. 2012.
[6] G. Denaro and M. Pezzè, "An Empirical Evaluation of Fault-Proneness Models," Proc. 24th Int'l Conf. Software Eng., pp. 241-251, May 2002.
[7] T.T. Dinh-Trong and J.M. Bieman, "The FreeBSD Project: A Replication Case Study of Open Source Development," IEEE Trans. Software Eng., vol. 31, no. 6, pp. 481-494, June 2005.
[8] K. El Emam, S. Benlarbi, N. Goel, W. Melo, H. Lounis, and S.N. Rai, "The Optimal Class Size for Object-Oriented Software," IEEE Trans. Software Eng., vol. 28, no. 5, pp. 494-509, May 2002.
[9] M. English, C. Exton, I. Rigon, and B. Cleary, "Fault Detection and Prediction in an Open-Source Software Project," Proc. Fifth Int'l Conf. Predictor Models in Software Eng., pp. 17:1-17:11, May 2009.
[10] N.E. Fenton and N. Ohlsson, "Quantitative Analysis of Faults and Failures in a Complex Software System," IEEE Trans. Software Eng., vol. 26, no. 8, pp. 797-814, Aug. 2000.
[11] T. Galinac Grbac and D. Huljenić, "Defect Detection Effectiveness and Product Quality in Global Software Development," Proc. 12th Int'l Conf. Product-Focused Software Process Improvement, pp. 113-127, June 2011.
[12] O. Gómez, N. Juristo, and S. Vegas, "Replication Types in Experimental Disciplines," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng. and Measurement, 2010.
[13] L. Hatton, "Reexamining the Fault Density-Component Size Connection," IEEE Software, vol. 14, no. 2, pp. 89-97, Mar. 1997.
[14] J.M. Juran, Quality Control Handbook. McGraw-Hill, 1974.
[15] N. Juristo and S. Vegas, "The Role of Non-Exact Replications in Software Engineering Experiments," Empirical Software Eng., vol. 16, no. 3, pp. 295-324, June 2011.
[16] M. Kaâniche and K. Kanoun, "Reliability of a Commercial Telecommunications System," Proc. Seventh Int'l Symp. Software Reliability Eng., pp. 207-212, Oct./Nov. 1996.
[17] B.A. Kitchenham, "The Role of Replications in Empirical Software Engineering—A Word of Warning," Empirical Software Eng., vol. 13, no. 2, pp. 219-221, Apr. 2008.
[18] A.G. Koru, K. El Emam, D. Zhang, H. Liu, and D. Mathew, "Theory of Relative Defect Proneness," Empirical Software Eng., vol. 13, no. 5, pp. 473-498, Oct. 2008.
[19] A.G. Koru, D. Zhang, K. El Emam, and H. Liu, "An Investigation into the Functional Form of the Size-Defect Relationship for Software Modules," IEEE Trans. Software Eng., vol. 35, no. 2, pp. 293-304, Mar. 2009.
[20] A.G. Koru, D. Zhang, and H. Liu, "Modeling the Effect of Size on Defect Proneness for Open-Source Software," Proc. Third Int'l Workshop Predictor Models in Software Eng., pp. 115-124, May 2007.
[21] S. Lessmann, B. Baesens, C. Mues, and S. Pietsch, "Benchmarking Classification Models for Software Defect Prediction: A Proposed Framework and Novel Findings," IEEE Trans. Software Eng., vol. 34, no. 4, pp. 485-496, July 2008.
[22] Y.K. Malaiya and J. Denton, "Module Size Distribution and Defect Density," Proc. 11th Int'l Symp. Software Reliability Eng., pp. 62-71, Oct. 2000.
[23] T. Menzies, J. Greenwald, and A. Frank, "Data Mining Static Code Attributes to Learn Defect Predictors," IEEE Trans. Software Eng., vol. 33, no. 1, pp. 2-13, Jan. 2007.
[24] J. Miller, "Replicating Software Engineering Experiments: A Poisoned Chalice or the Holy Grail," Information Software Technology, vol. 47, no. 4, pp. 233-244, Mar. 2005.
[25] A. Mockus, R.T. Fielding, and J.D. Herbsleb, "Two Case Studies of Open Source Software Development: Apache and Mozilla," ACM Trans. Software Eng. Methodology, vol. 11, no. 3, pp. 309-346, July 2002.
[26] P. Mohagheghi and R. Conradi, "An Empirical Investigation of Software Reuse Benefits in a Large Telecom Product," ACM Trans. Software Eng. Methodology, vol. 17, no. 3, pp. 13:1-13:31, June 2008.
[27] J.C. Munson and T.M. Khoshgoftaar, "The Detection of Fault-Prone Programs," IEEE Trans. Software Eng., vol. 18, no. 5, pp. 423-433, May 1992.
[28] N. Ohlsson and H. Alberg, "Predicting Fault-Prone Software Modules in Telephone Switches," IEEE Trans. Software Eng., vol. 22, no. 12, pp. 886-894, Dec. 1996.
[29] T.J. Ostrand and E.J. Weyuker, "The Distribution of Faults in a Large Industrial Software System," Proc. ACM SIGSOFT Int'l Symp. Software Testing and Analysis, pp. 55-64, July 2002.
[30] K. Petersen and C. Wohlin, "Context in Industrial Software Engineering Research," Proc. Third Int'l Symp. Empirical Software Eng. and Measurement, pp. 401-404, Oct. 2009.
[31] L.M. Pickard, B.A. Kitchenham, and P. Jones, "Combining Empirical Results in Software Engineering," Information and Software Technology, vol. 40, no. 14, pp. 811-821, 1998.
[32] B. Robinson and P. Francis, "Improving Industrial Adoption of Software Engineering Research: A Comparison of Open and Closed Source Software," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng. and Measurement, pp. 21:1-21:10, Sept. 2010.
[33] P. Runeson, M.C. Ohlsson, and C. Wohlin, "A Classification Scheme for Studies on Fault-Prone Components," Proc. Third Int'l Conf. Product Focused Software Process Improvement, pp. 341-355, Sept. 2001.
[34] P. Runeson, M. Höst, A. Rainer, and B. Regnell, Case Study Research in Software Engineering: Guidelines and Examples. Wiley, 2012.
[35] E. Shihab, Z.M. Jiang, W.M. Ibrahim, B. Adams, and A.E. Hassan, "Understanding the Impact of Code and Process Metrics on Post-Release Defects: A Case Study on the Eclipse Project," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng. and Measurement, pp. 4:1-4:10, Sept. 2010.
[36] F.J. Shull, J.C. Carver, S. Vegas, and N. Juristo, "The Role of Replications in Empirical Software Engineering," Empirical Software Eng., vol. 13, no. 2, pp. 211-218, Apr. 2008.
[37] R.W. Selby and V.R. Basili, "Analyzing Error-Prone System Structure," IEEE Trans. Software Eng., vol. 17, no. 2, pp. 141-152, Feb. 1991.
[38] R.S. Chhillar and Nisha, "Empirical Analysis of Object-Oriented Design Metrics for Predicting High, Medium and Low Severity Faults Using Mallows," SIGSOFT Software Eng. Notes, vol. 36, no. 6, pp. 1-9, Nov. 2011.
[39] B. Turhan, T. Menzies, A.B. Bener, and J. Di Stefano, "On the Relative Value of Cross-Company and Within-Company Data for Defect Prediction," Empirical Software Eng., vol. 14, no. 5, pp. 540-578, Oct. 2009.
[40] S. Wu, Q. Wang, and Y. Yang, "Quantitative Analysis of Faults and Failures with Multiple Releases of Software," Proc. ACM/IEEE Second Int'l Symp. Empirical Software Eng. and Measurement, pp. 198-205, Oct. 2008.
[41] H. Zhang, A. Nelson, and T. Menzies, "On the Value of Learning from Defect Dense Components for Software Defect Prediction," Proc. Sixth Int'l Conf. Predictive Models in Software Eng., pp. 14:1-14:9, Sept. 2010.
[42] T. Zimmermann, N. Nagappan, H. Gall, E. Giger, and B. Murphy, "Cross-Project Defect Prediction: A Large Scale Experiment on Data vs. Domain vs. Process," Proc. Seventh Joint Meeting European Software Eng. Conf. and the ACM SIGSOFT Symp. Foundations of Software Eng., pp. 91-100, Aug. 2009.
[43] T. Zimmermann, R. Premraj, and A. Zeller, "Predicting Defects for Eclipse," Proc. Third Int'l Workshop Predictor Models in Software Eng., pp. 9:1-9:7, May 2007.
38 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool