This Article 
 Bibliographic References 
 Add to: 
The Use of Self Checks and Voting in Software Error Detection: An Empirical Study
April 1990 (vol. 16 no. 4)
pp. 432-443

The results of an empirical study of software error detection using self checks and N-version voting are presented. Working independently, each of 24 programmers first prepared a set of self checks using just the requirements specification of an aerospace application, and then each added self checks to an existing implementation of that specification. The modified programs were executed to measure the error-detection performance of the checks and to compare this with error detection using simple voting among multiple versions. The analysis of the checks revealed that there are great differences in the ability of individual programmers to design effective checks. It was found that some checks that might have been effective failed to detect an error because they were badly placed, and there were numerous instances of checks signaling nonexistent errors. In general, specification-based checks alone were not as effective as specification-based checks combined with code-based checks. Self checks made it possible to identify faults that had not been detected previously by voting 28 versions of the program over a million randomly generated inputs. This appeared to result from the fact that the self checks could examine the internal state of the executing program, whereas voting examines only final results of computations. If internal states had to be identical in N-version voting systems, then there would be no reason to write multiple versions.

[1] T. Anderson, P. A. Barrett, D. N. Halliwell, and M. R. Moulding, "An evaluation of software fault tolerance in a practical system," inDig. Papers FTCS-15: Fifteenth Annu. Symp. Fault-Tolerant Computing, Ann Arbor, MI, June 1985, pp. 140-145.
[2] T. Anderson and P. A. Lee,Fault Tolerance: Principles and Practice, Englewood Cliffs, NJ: Prentice-Hall International, 1981.
[3] D. M. Andrews, "Using executable assertions for testing and fault tolerance," inProc. Ninth Int. Symp. Fault-Tolerant Computer Systems, June 1979, pp. 102-105.
[4] A. Avizienis and J. P. J. Kelly, "Fault tolerance by design diversity: concepts and experiments,"Computer, vol. 17, no. 8, pp. 67-80, Aug. 1984.
[5] J. P. Benson and S. H. Saib, "A software quality assurance experiment," inProc. Software Quality and Assurance Workshop, Nov. 1978, pp. 87-91.
[6] P. G. Bishop, D. G. Esp, M. Barnes, P. Humphreys, G. Dahl, and J. Lahti, "PODS--A project on diverse software,"IEEE Trans. Software Eng., vol. SE-12, no. 9, pp. 929-940, 1986.
[7] S. S. Brilliant, "Testing software using multiple versions," Ph.D. dissertation, Univ. Virginia, Charlottesville, Sept. 1987.
[8] S. S. Brilliant, J. C. Knight, and N. G. Leveson, "Analysis of faults in anN-version software experiment,"IEEE Trans. Software Eng., vol. 16, no. 2, pp. 238-247, Feb. 1990.
[9] S. S. Brilliant, J. C. Knight, and N. G. Leveson, "The consistent comparison problem inN-version software,"IEEE Trans. Software Eng., vol. 15, no. 11, pp. 1481-1485, Nov. 1989.
[10] L. Chen and A. Avizienis, "N-version programming: A fault-tolerance approach to reliability of software operation," inDig. Papers FTCS-8: Eighth Annu. Symp. Fault Tolerant Computing, Toulouse, France, June 1978, pp. 3-9.
[11] J. R. Dunham, "Software errors in experimental systems having ultra-reliability requirements," inDig. Papers FTCS-16: Sixteenth Annu. Symp. Fault-Tolerant Computing, Vienna, Austria, July 1986, pp. 158-164.
[12] L. Gmeiner and U. Voges, "Software diversity in reactor protection system: An experiment," inProc. IFAC Workshop SAFECOMP '79, 1979, pp. 75-79.
[13] D. Gries,The Science of Programming. New York: Springer-Verlag, 1981.
[14] A. L. Hopkins,et al., "FTMP--A highly reliable fault-tolerant multiprocessor for aircraft,"Proc. IEEE, vol. 66, pp. 1221-1239, Oct. 1978.
[15] J. P. J. Kellyet al., "Multi-version software development," inProc. IFAC Workshop Safecomp '86, Sarlat, France, Oct. 1986, pp. 43-49.
[16] J. Knight and N. Leveson, "An experimental evaluation of the assumption of independence in multiversion programming,"IEEE Trans. Software Eng., vol. SE-12, no. 1, pp. 96-109, Jan. 1986.
[17] J. C. Knight and N. G. Leveson, "An empirical study of failure probabilities in multiversion software," inDig. Papers FTCS-16: Sixteenth Annu. Symp. Fault-Tolerant Computing, Vienna, Austria, July 1986, pp. 165-170.
[18] N. G. Leveson and P. R. Harvey, "Analyzing software safety,"IEEE Trans. Software Eng., vol. SE-9, no. 5, pp, 569-579, Sept. 1983.
[19] N. G. Leveson, and T. J. Shimeall, "Safety assertions for processcontrol systems," inDig. Papers FTCS-13: Thirteenth Annu. Symp. Fault-Tolerant Computing, Milan, Italy, June 1983, pp. 236-240.
[20] H. Partsch and R. Steinbrüggen, "Program transformation systems,"ACM Comput. Surveys, vol. 15, no. 3, pp. 199-236, Sept. 1983.
[21] C. V. Ramamoorthy, Y. K. Mok, E. B. Bastani, G. H. Chin, and K. Suzuki, "Application of a methodology for the development and validation of reliable process control software,"IEEE Trans. Software Eng., vol. SE-7, no. 6, pp. 537-555, Nov. 1981.
[22] B. Randell, "System structure for software fault-tolerance,"IEEE Trans. Software Eng., vol. SE-1, no. 2, pp. 220-232, June 1975.
[23] F. Saglietti and W. Ehrenberger, "Software diversity--Some considerations about its benefits and its limitations," inProc. Safecomp '86, Sarlat, France, Oct. 1986, pp. 27-34.
[24] R. D. Schlichting and F.B. Schneider, "Fail-stop processors: An approach to designing fault-tolerant computing systems,"ACM Trans. Comput. Syst., vol. 1, no. 3, pp. 222-238, Aug. 1983.
[25] K. R. Scott, J. W. Gault, D. F. McAllister, and J. Wiggs, "Experimental validation of six fault tolerant software reliability models," inDig. Papers FTCS-14: Fourteenth Annu. Symp. Fault-Tolerant Computing, Kissemmee, NY, 1984, pp. 102-107.
[26] T. J. Shimeall and N. G. Leveson, "An empirical comparison of software fault tolerance and fault elimination," inProc. Second Workshop Software Test. Verification Analysis, Banff, Canada, July 1988.
[27] T. J. Shimeall and N. G. Leveson, "An empirical comparison of software fault tolerance and fault elimination," Naval Postgraduate School, Monterey, CA, Tech. Rep. NPS52-89-D47, July 1989.
[28] L. G. Stucki, "New directions in automated tools for improving software quality," inCurrent Trends in Programming Methodology-- volume II: Program Validation. Englewood Cliffs, NJ: Prentice-Hall, 1977, pp. 80-111.
[29] J. H. Wensleyet al., "SIFT, the design and analysis of a fault-tolerant computer for aircraft control,"Proc. IEEE, vol. 66, pp. 1240- 1254, Oct, 1978.

Index Terms:
self checks; voting; software error detection; N-version voting; requirements specification; code-based checks; fault tolerant computing; software reliability.
N.G. Leveson, S.S. Cha, J.C. Knight, T.J. Shimeall, "The Use of Self Checks and Voting in Software Error Detection: An Empirical Study," IEEE Transactions on Software Engineering, vol. 16, no. 4, pp. 432-443, April 1990, doi:10.1109/32.54295
Usage of this product signifies your acceptance of the Terms of Use.