This Article 
 Bibliographic References 
 Add to: 
Software Fault Interactions and Implications for Software Testing
June 2004 (vol. 30 no. 6)
pp. 418-421
Dolores R. Wallace, IEEE Computer Society

Abstract—Exhaustive testing of computer software is intractable, but empirical studies of software failures suggest that testing can in some cases be effectively exhaustive. Data reported in this study and others show that software failures in a variety of domains were caused by combinations of relatively few conditions. These results have important implications for testing. If all faults in a system can be triggered by a combination of n or fewer parameters, then testing all n{\hbox{-}}\rm tuples of parameters is effectively equivalent to exhaustive testing, if software behavior is not dependent on complex event sequences and variables have a small set of discrete values.

[1] R. Brownlie, J. Prowse, and M.S. Phadke, Robust Testing of AT&T PMX/StarMail Using OATS AT&T Technical J., vol. 71, no. 3, pp. 41-47, May/June 1992.
[2] D.M. Cohen, S.R. Dalal, J. Parelius, and G.C. Patton, "The Combinatorial Design Approach to Automatic Test Generation," IEEE Software, vol. 13, no. 5, pp. 83-89, Sept. 1996.
[3] V.N. Nair, D.A. James, W.K. Erlich, and J. Zevallos, A Statistical Assessment of Some Software Testing Strategies and Application of Experimental Design Techniques Statistica Sinica, vol. 8, no. 1, pp. 165-184, 1998.
[4] D.R. Wallace and D.R. Kuhn, Failure Modes in Medical Device Software: An Analysis of 15 Years of Recall Data Int'l J. Reliability, Quality and Safety Eng., vol. 8, no. 4, 2001.
[5] D.R. Kuhn and M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing Proc. 27th NASA/IEEE Software Eng. Workshop, Dec. 2002.
[6] S.R. Dalal, A. Jain, N. Karunanithi, J.M. Leaton, C.M. Lott, G.C. Patton, and B.M. Horowitz, Model-Based Testing in Practice Proc. Int'l Conf. Software Eng., 1999.
[7] B. Smith, M.S. Feather, and N. Muscettola, Challenges and Methods in Testing the Remote Agent Planner Proc. Fifth Int'l Conf. Artificial Intelligence Planning Systems, 2000.
[8] J. Pan, The Dimensionality of Failures A Fault Model for Characterizing Software Robustness Proc. Int'l Symp. Fault-Tolerant Computing, June 1999.
[9] M. Jones, What Really Happened on Mars Pathfinder Rover RISKS Digest, vol. 19, no. 49, Dec. 1997.
[10] J.L. Lions, Ariane 5, Flight 501, Report of the Inquiry Board European Space Agency, July 1996.
[11] G. Slabodkin, Software Glitches Leave Navy Smart Ship Dead in the Water Government Computer News, July 1998.
[12] B. Stevens, L. Moura, and E. Mendelsohn, Lower Bounds for Transversal Covers Design, Codes, and Cryptography, vol. 15, pp. 279-299, 1998.
[13] The CRC Handbook of Combinatorial Designs. C.J. Colbourn and J.H. Dinitz, eds., CRC Press, 1996.
[14] A.W. Williams and R.L. Probert, A Measure for Component Interaction Test Coverage Proc. ACS/IEEE Int'l Conf. Computer Systems and Applications (AICCSA 2001), pp. 304-311, June 2001.
[15] C. Cheng, A. Dumitrescu, and P. Schroeder, Generating Small Test Suites for Non-Uniform Instances Third Int'l Conf. Quality Software, Nov. 2003.
[16] P.E. Ammann, P.E. Black, and W. Majurski, Using Model Checking to Generate Tests from Specifications Proc. Second IEEE Int'l Conf. Formal Eng. Methods (ICFEM '98), pp. 46-54, Dec. 1998.
[17] M.R. Blackburn, R.D. Busser, A.M. Nauman, and R. Chandramouli, Model Based Approach to Security Test Automation Proc. Quality Week, June 2001.
[18] K.C. Tai and Y. Lie, A Test Generation Strategy for Pairwise Testing IEEE Trans. Software Eng., vol. 28, no. 1, pp. 109-111, 2002.
[19] A.W. Williams and R.L. Probert, Formulation of the Interaction Test Coverage Problem as an Integer Program Proc. TestCom Conf., pp. 283-298, 2002.
[20] M.B. Cohen, C.J. Colbourn, P.B. Gibbons, and W.B. Mugridge, Constructing Test Suites for Interaction Testing Proc. 25th Int'l Conf. Software Eng. (ICSE 2003), pp. 38-48, May 2003.
[21] D.M. Cohen, S.R. Dalal, M.L. Fredman, and G.C. Patton, The AETG System: An Approach to Testing Based on Combinatorial Design IEEE Trans. Software Eng., vol. 23, no. 7, pp. 437-443, July 1997.

Index Terms:
Statistical methods, testing strategies, test design.
D. Richard Kuhn, Dolores R. Wallace, Albert M. Gallo, "Software Fault Interactions and Implications for Software Testing," IEEE Transactions on Software Engineering, vol. 30, no. 6, pp. 418-421, June 2004, doi:10.1109/TSE.2004.24
Usage of this product signifies your acceptance of the Terms of Use.