This Article 
 Bibliographic References 
 Add to: 
An Empirical Comparison of Software Fault Tolerance and Fault Elimination
February 1991 (vol. 17 no. 2)
pp. 173-182

The authors compared two major approaches to the improvement of software-software fault elimination and software fault tolerance-by examination of the fault detection (and tolerance, where applicable) of five techniques: run-time assertions, multiversion voting, functional testing augmented by structural testing, code reading by stepwise abstraction, and static data-flow analysis. The focus was on characterizing the sets of faults detected by the techniques and on characterizing the relationships between these sets of faults. Two categories of questions were investigated: (1) comparison between fault elimination and fault tolerance techniques and (2) comparisons among various testing techniques. The results provide information useful for making decisions about the allocation of project resources, show strengths and weaknesses of the techniques studies, and indicate directions for future research.

[1] "Glossary of software engineering terminology," ANSI-IEEE Standard 729-1983, 1983.
[2] T. Anderson, P. A. Barrett, D. N. Halliwell, and M. R. Moulding, "Software fault tolerance: An evaluation,"IEEE Trans. Software Eng., vol. SE-11, no. 12, pp. 1502-1510, Dec. 1985.
[3] T. Anderson and P. A. Lee,Fault Tolerance: Principles and Practice. Englewood Cliffs, NJ: Prentice-Hall, 1981.
[4] A. Avizienis and J. P. J. Kelly, "Fault tolerance by design diversity: Concepts and experiments,"Computer, vol. 17, pp. 67-80, Aug. 1984.
[5] V. R. Basili and R. W. Selby, "Comparing the effectiveness of software testing strategies,"IEEE Trans. Software Eng., vol. SE-13, no. 12, pp. 1278-1296, Dec. 1987.
[6] P. G. Bishop, D. G. Esp, M. Barnes, P. Humphreys, G. Dahl, and J. Lahti, "PODS--A project on diverse software,"IEEE Trans. Software Eng., vol. SE-12, no. 9, pp. 929-940, 1986.
[7] S. S. Brilliant, "Testing software using multiple versions," Ph.D. dissertation, Univ. Virginia, Charlottesville, Sept. 1987.
[8] S. S. Brilliant, J. C. Knight, and N. G. Leveson, "The consistent comparison problem in multiversion software,"IEEE Trans. Software Eng., vol. 15, no. 11, pp. 1481-1485, Nov. 1989.
[9] S. S. Brilliant, J. C. Knight, and N. G. Leveson, "Analysis of faults in anN-version software experiment,"IEEE Trans. Software Eng., vol. 16, no. 2, pp. 238-247, Feb. 1990.
[10] J. E. Brunelle and D. E. Eckhardt, "Fault tolerant software: Experiment with the SIFT operating system," inProc. AIAA Computers in Aerospace V Conf., Oct. 1985, pp. 355-360.
[11] L. Chen and A. Avizienis, "N-version programming: A fault tolerance approach to the reliability of software," inProc. Eighth Int. Symp. Fault-Tolerant Computing, Toulouse, France, June 1978, pp. 3-9.
[12] A. W. Dobieski, "Modeling tactical military operations,"Quest, pp. 1-25, Spring 1979.
[13] L. D. Fosdick and L. J. Osterweil, "Data flow analysis in software reliability,"ACM Comput. Surveys, vol. 8, no. 3, pp. 306-330, Sept. 1976.
[14] P. Frankl and E. Weyuker, "Data flow testing in the presence of unexecutable paths," inProc. Workshop Software Testing, Banff, Alta., Canada, July 1986, pp. 4-13.
[15] M. R. Girgis and M. R. Woodward, "An experimental comparison of the error exposing ability of program testing criteria," inProc. Software Testing Workshop, Banff, Alta., Canada, 1986, pp. 64-73.
[16] W. C. Hetzel, "An experimental analysis of program verification methods," Ph.D. dissertation, Univ. North Carolina at Chapel Hill, 1976.
[17] W. E. Howden, "Functional testing and design abstractions,"J. Syst. Software, vol. 1, pp. 307-313, 1980.
[18] W. E. Howden, "A survey of static analysis methods,"Tutorial: Software Testing and Validation Techniques. New York: IEEE Press, 1981, pp. 101-115.
[19] J. Knight and N. Leveson, "An experimental evaluation of the assumption of independence in multiversion programming,"IEEE Trans. Software Eng., vol. SE-12, no. 1, pp. 96-109, Jan. 1986.
[20] J. C. Knight and N. G. Leveson, "An empirical study of failure probabilities in multiversion software," inProc. Sixteenth Int. Symp. Fault-Tolerant Computing, Vienna, Austria, July 1986, pp. 165-170.
[21] Personal communication with Larry Yount of Honeywell/Sperry Avionics and Robert Hall and Michael Dewalt of the FAA.
[22] N. G. Leveson, S. S. Cha, J. C. Knight, and T. J. Shimeall, "The use of self checks and voting in software error detection: An empirical study,"IEEE Trans. Software Eng., vol. 16, no. 4, pp. 432-443, Apr. 1990.
[23] R. C. Linger, H. D. Mills, and B. I. Witt,Structured Programming Theory and Practice. Reading. MA: Addison-Wesley, 1979.
[24] P. M. Melliar-Smith and R. L. Schwartz, "Formal specification and mechanical verification of SIFT: A fault-tolerant flight-control system,"IEEE Trans. Comput., vol. C-31, no. 7, pp. 616-630, July 1982.
[25] G. J. Myers, "A controlled experiment in program testing and code walk-throughs/inspections,"Commun. ACM, pp. 760-768, Sept. 1978.
[26] C. V. Ramamoorthy, Y. K. Mok, E. B. Bastani, G. H. Chin, and K. Suzuki, "Application of a methodology for the development and validation of reliable process control software,"IEEE Trans. Software Eng., vol. SE-7, no. 6, pp. 537-555, Nov. 1981.
[27] C. V. Ramamoorthy and S. F. Ho, "Testing large software with automated software evaluation systems,"IEEE Trans. Software Eng., vol. SE-1, no. 1, pp. 46-58, Mar. 1975.
[28] F. Saglietti and W. Ehrenberger, "Software diversity--Some considerations about its benefits and its limitations," inProc. Safecomp '86, Sarlat, France, Oct. 1986.
[29] K. S. Tso, A. Avizienis, and J. P. J. Kelly, "Error recovery in multiversion software development," inProc. Safecomp '86, Sarlat, France, Oct. 1986, pp. 43-50.

Index Terms:
software reliability; software fault tolerance; fault elimination; run-time assertions; multiversion voting; functional testing; structural testing; code reading; stepwise abstraction; static data-flow analysis; project resources; fault tolerant computing; program testing; software reliability
T.J. Shimeall, N.G. Leveson, "An Empirical Comparison of Software Fault Tolerance and Fault Elimination," IEEE Transactions on Software Engineering, vol. 17, no. 2, pp. 173-182, Feb. 1991, doi:10.1109/32.67598
Usage of this product signifies your acceptance of the Terms of Use.