This Article 
 Bibliographic References 
 Add to: 
Modeling the Effects of Combining Diverse Software Fault Detection Techniques
December 2000 (vol. 26 no. 12)
pp. 1157-1167

Abstract—The software engineering literature contains many studies of the efficacy of fault finding techniques. Few of these, however, consider what happens when several different techniques are used together. We show that the effectiveness of such multitechnique approaches depends upon quite subtle interplay between their individual efficacies and dependence between them. The modeling tool we use to study this problem is closely related to earlier work on software design diversity. The earliest of these results showed that, under quite plausible assumptions, it would be unreasonable even to expect software versions that were developed “truly independently” to fail independently of one another. The key idea here was a “difficulty function” over the input space. Later work extended these ideas to introduce a notion of “forced” diversity, in which it became possible to obtain system failure behavior better even than could be expected if the versions failed independently. In this paper, we show that many of these results for design diversity have counterparts in diverse fault detection in a single software version. We define measures of fault finding effectiveness and of diversity and show how these might be used to give guidance for the optimal application of different fault finding procedures to a particular program. We show that the effects upon reliability of repeated applications of a particular fault finding procedure are not statistically independent—in fact, such an incorrect assumption of independence will always give results that are too optimistic. For diverse fault finding procedures, on the other hand, things are different: Here it is possible for effectiveness to be even greater than it would be under an assumption of statistical independence. We show that diversity of fault finding procedures is, in a precisely defined way, “a good thing” and should be applied as widely as possible. The new model and its results are illustrated using some data from an experimental investigation into diverse fault finding on a railway signalling application.

[1] E.N. Adams, “Optimizing Preventive Maintenance of Software Products,” IBM J. Research and Development, vol. 28, no. 1, pp. 2–14, 1984.
[2] V. Basili and S. Green, "Software Process Evolution at the SEL," IEEE Software, pp. 58-66, 1994.
[3] V.R. Basili and R.W. Selby, “Comparing the Effectiveness of Software Testing Strategies,” IEEE Trans. Software Eng., vol. 13, pp. 1,278-1,296, 1987.
[4] D.E. Eckhardt et al., "An Experimental Evaluation of Software Redundancy as a Strategy for Improving Reliability," IEEE Trans. Software Eng., vol. 17, no. 7, 1991, pp. 692-702.
[5] D.E. Eckhardt and L.D. Lee, “A Theoretical Basis of Multiversion Software Subject to Coincident Errors,” IEEE Trans. Software Eng., vol. 11, pp. 1,511–1,517, 1985.
[6] P. Frankl, D. Hamlet, B. Littlewood, and L. Strigini, “Choosing a Testing Method to Deliver Reliability,” Proc. 19th Int'l Conf. Software Eng., pp. 68-78, May 1997.
[7] P.G. Frankl and S.N Weiss,“An experimental comparison of the effectiveness of branch testing and data flow testing,” IEEE Transactions on Software Engineering, vol. 19, no. 8, Oct. 1993, pp. 774-787.
[8] R.B. Grady, Practical Software Metrics for Project Management and Process Improvement, Prentice Hall, Englewood Cliffs, N.J., 1992.
[9] J.C. Knight and N.G. Leveson, "An Experimental Evaluation of the Assumption of Independence in Multiversion Programming," IEEE Trans. Software Eng., Vol. 12, No. 1, 1986, pp. 96-109.
[10] Bev Littlewood and Douglas R. Miller, "Conceptual Modeling of Coincident Failures in Multiversion Software," IEEE Transactions on Software Engineering, vol. 15, p. 1,596, Dec. 1989.
[11] J.C. Rouquet and P.J. Traverse., “Safe and Reliable Computing on Board the Airbus and ATR Aircraft,” Proc. Safecomp: Fifth IFAC Workshop Safety of Computer Control Systems, 1986.
[12] T.J. Shimeall and N.G. Leveson, “An Empirical Comparison of Software Fault Tolerance and Fault Elimination,” IEEE Trans. Software Eng., vol. 17, no. 2, pp. 173–182, 1991.
[13] N.M. Shryane, S.J. Westerman, C.M. Crawshaw, G.R.J. Hockey, and J. Sauer, “Task Analysis for the Investigation of Human Error in Safety-Critical Software Design: A Convergent Methods Approach,” Ergonomics, vol. 41, no. 11, pp. 1,719–1,736, 1998.
[14] S.J. Westerman, N.M. Shryane, C.M. Crawshaw, and G.R.J. Hockey., “Engineering Cognitive Diversity,” Proc. Fifth Safety Critical Systems Symp., 1997.
[15] S.J. Westerman, N.M. Shryane, C.M. Crawshaw, G.R.J. Hockey, and C.W. Wyatt-Millington., “Cognitive Diversity: A Structured Approach to Trapping Human Error,” Proc. Safecomp '95: 14th Int'l Conf. Computer Safety, Reliability, and Security, 1995.

Index Terms:
Software fault, fault removal, software reliability, testing, reliability growth, diversity.
Bev Littlewood, Peter T. Popov, Lorenzo Strigini, Nick Shryane, "Modeling the Effects of Combining Diverse Software Fault Detection Techniques," IEEE Transactions on Software Engineering, vol. 26, no. 12, pp. 1157-1167, Dec. 2000, doi:10.1109/32.888629
Usage of this product signifies your acceptance of the Terms of Use.