The Community for Technology Leaders
RSS Icon
Issue No.04 - July/August (2009 vol.35)
pp: 484-496
Maggie Hamill , West Virginia University, Morgantown
Katerina Goševa-Popstojanova , West Virginia University, Morgantown
The benefits of the analysis of software faults and failures have been widely recognized. However, detailed studies based on empirical data are rare. In this paper, we analyze the fault and failure data from two large, real-world case studies. Specifically, we explore: 1) the localization of faults that lead to individual software failures and 2) the distribution of different types of software faults. Our results show that individual failures are often caused by multiple faults spread throughout the system. This observation is important since it does not support several heuristics and assumptions used in the past. In addition, it clearly indicates that finding and fixing faults that lead to such software failures in large, complex systems are often difficult and challenging tasks despite the advances in software development. Our results also show that requirement faults, coding faults, and data problems are the three most common types of software faults. Furthermore, these results show that contrary to the popular belief, a significant percentage of failures are linked to late life cycle activities. Another important aspect of our work is that we conduct intra- and interproject comparisons, as well as comparisons with the findings from related studies. The consistency of several main trends across software systems in this paper and several related research efforts suggests that these trends are likely to be intrinsic characteristics of software faults and failures rather than project specific.
Software faults and failures, fault location, fault types, software fault distribution, software reliability, empirical studies.
Maggie Hamill, Katerina Goševa-Popstojanova, "Common Trends in Software Fault and Failure Data", IEEE Transactions on Software Engineering, vol.35, no. 4, pp. 484-496, July/August 2009, doi:10.1109/TSE.2009.3
[1] C. Andersson and P. Runeson, “A Replicated Quantitative Analysis of Fault Distributions in Complex Software Systems,” IEEE Trans. Software Eng., vol. 33, no. 5, pp. 273-286, May 2007.
[2] V.R. Basili and B.T. Perricone, “Software Errors and Complexity: An Empirical Investigation,” Comm. ACM, vol. 27, no. 1, pp. 41-52, 1984.
[3] V.R. Basili, “The Role of Experimentation in Software Engineering: Past, Current, and Future,” Proc. 18th Int'l Conf. Software Eng., pp.442-449, 1996.
[4] B.W. Boehm, R.K. McClean, and D.B. Urfrig, “Some Experience with Automated Aids to the Design of Large Scale Reliable Software,” IEEE Trans. Software Eng., vol. 1, no. 1, pp. 125-133, Mar. 1975.
[5] B. Boehm and V.R. Basili, “Software Defect Reduction Top 10 List,” Computer, vol. 34, no. 1, pp. 135-137, Jan. 2001.
[6] R. Chillarege, I. Bhandari, J.K. Chaar, M.J. Halliday, D.S. Moebus, B.K. Ray, and M. Wong, “Orthogonal Defect Classification—A Concept for In-Process Measurements,” IEEE Trans. Software Eng., vol. 18, no. 11, pp. 943-956, Nov. 1992.
[7] J. Christmansson and R. Chillarege, “Generation of an Error Set That Emulates Software Faults Based on Field Data,” Proc. 26th Int'l Symp. Fault-Tolerant Computing, pp. 304-313, June 1996.
[8] J.A. Duraes and H.S. Madeira, “Emulation of Software Faults: A Field Data Study and a Practical Approach,” IEEE Trans. Software Eng., vol. 32, no. 11, pp. 849-867, Nov. 2006.
[9] A. Endres, “An Analysis of Errors and Their Causes in System Programs,” IEEE Trans. Software Eng., vol. 1, no. 2, pp. 140-149, June 1975.
[10] A. Endres and D. Rombach, A Handbook of Software and Systems Engneering: Empirical Observations, Laws and Theories. Pearson/Addison-Wesley, 2003.
[11] N.E. Fenton and M. Neil, “A Critique of Software Defect Prediction Models,” IEEE Trans. Software Eng., vol. 25, no. 5, pp.675-689, Sept./Oct. 1999.
[12] N. Fenton and N. Ohlsson, “Quantitative Analysis of Faults and Failures in a Complex Software System,” IEEE Trans. Software Eng., vol. 26, no. 8, pp. 797-814, Aug. 2000.
[13] K. Goševa-Popstojanova and K.S. Trivedi, “Architecture Based Approach to Quantitative Assessment of Software Systems,” Performance Evaluation, vol. 45, nos. 2/3, pp. 179-204, June 2001.
[14] K. Goševa-Popstojanova, M. Hamill, and R. Perugupalli, “Large Empirical Case Study of Architecture-Based Software Reliability,” Proc. 16th IEEE Int'l Symp. Software Reliability Eng., pp. 43-52, Nov. 2005.
[15] K. Goševa-Popstojanova, M. Hamill, and X. Wang, “Adequacy, Accuracy, Scalability, and Uncertainty of Architecture-Based Software Reliability: Lessons Learned from Large Empirical Case Studies,” Proc. 17th IEEE Int'l Symp. Software Reliability Eng., pp.505-514, Nov. 2006.
[16] W.S. Greenwell and J.C. Knight, “What Should Aviation Safety Incidents Teach Us?” Proc. 22nd Int'l Conf. Computer Safety, Reliability and Security, Sept. 2003.
[17] G. Holzmann, “Conquering Complexity,” Computer, vol. 40, no. 12, pp. 111-113, Dec. 2007.
[18] D. Jackson, M. Thomas, and L. Millett, Software for Dependable Systems: Sufficient Evidence? Nat'l Academies Press, 2007.
[19] M. Leszak, D. Perry, and D. Stoll, “Classification and Evaluation of Defect in a Project Retrospective,” J. Systems and Software, vol. 61, pp. 173-187, Apr. 2002.
[20] N. Leveson, Safeware System Safety and Computers. Addison-Wesley, 1995.
[21] R.R. Lutz and I.C. Mikulski, “Empirical Analysis of Safety Critical Anomalies during Operation,” IEEE Trans. Software Eng., vol. 30, no. 3, pp. 172-180, Mar. 2004.
[22] R.R. Lutz and I.C. Mikulski, “Ongoing Requirements Discovery in High Integrity Systems,” IEEE Software, vol. 21, no. 2, pp. 19-25, Mar./Apr. 2004.
[23] T.J. Ostrand and E.J. Weyuker, “The Distribution of Faults in a Large Industrial Software System,” Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 55-64, 2002.
[24] T.J. Ostrand, E.J. Weyuker, and R.M. Bell, “Where the Bugs Are,” Proc. ACM Int'l Symp. Software Testing and Analysis, 2004.
[25] W.D. Yu, “A Software Fault Prevention Approach in Coding and Root Cause Analysis,” Bell Labs Technical J., vol. 3, no. 2, pp. 3-21, Apr.-June 1998.
[26] sedefinitions.html, 2009.
3 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool