This Article 
 Bibliographic References 
 Add to: 
Estimation and Enhancement of Real-Time Software Reliability Through Mutation Analysis
May 1992 (vol. 41 no. 5)
pp. 550-558

A simulation-based method for obtaining numerical estimates of the reliability of N-version, real-time software is proposed. An extended stochastic Petri net is used to represent the synchronization structure of N versions of the software, where dependencies among versions are modeled through correlated sampling of module execution times. The distributions of execution times are derived from automatically generated test cases that are based on mutation testing. Since these test cases are designed to reveal software faults, the associated execution times and reliability estimates are likely to be conservative. Experimental results using specifications for NASA's planetary lander control software suggest that mutation-based testing could hold greater potential for enhancing reliability than the desirable but perhaps unachievable goal of independence among N versions. Nevertheless, some support for N-version enhancement of high-quality, mutation-tested code is also offered. Mutation analysis could also be valuable in the design of fault-tolerant software systems.

[1] P. Ammann and J. Knight, "Data diversity: An approach to software fault-tolerance," inProc. 17th Int. Symp. Fault-Tolerant Comput. (FTCS- 17), Pittsburgh, PA, July, 1987, pp. 122-126.
[2] A. Avizienis, "TheN-version approach to fault-tolerant software,"IEEE Trans. Software Eng., vol. SE-11, pp. 1491-1501, 1985.
[3] G. Balbo, S. Bruell, and S. Ghanta, "Combining queueing networks and generalized stochastic Petri nets for the solution of complex models of system behavior,"IEEE Trans. Comput., vol. 37, 1988.
[4] T. A. Budd, "Mutation analysis of program test data," Ph.D. thesis, Yale Univ., New Haven CT, 1980.
[5] T. A. Budd and D. Angluin, "Two notions of correctness and their relation to testing,"Acta Informatica, vol. 18, no. 1, pp. 31-45, Nov. 1982.
[6] R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Offutt, "An extended overview of the Mothra software testing environment," inProc. Second Workshop Software Testing, Verification and Analysis, Banff, Alberta, July 1988, pp. 142-151.
[7] R. A. DeMillo, R. J. Lipton, and F. G. Sayward, "Hints on test data selection: Help for the practicing programmer,"IEEE Comput. Mag., vol. 11, Apr. 1978.
[8] R. A. DeMillo and A. J. Offutt, "Experimental results of automaticaIly generated adequate test sets," inProc. Sixth Annu. Pacific Northwest Software Quality Conf., Portland OR, Sept. 1988, Lawrence and Craig, pp. 209-151.
[9] R. A. DeMillo and A. J. Offutt, "Constraint-based automatic test data generation,"IEEE Trans. Software Eng., vol. 17, pp. 900-910, Sept. 1991.
[10] J. Dugan, A. Bobbio, G. Ciardo, and K. Trivedi, "The design of a unified package for the solution of stochastic petri net models," inProc. Int. Workshop Timed Petri Nets, Torino, Italy, 1985.
[11] J. B. Dugan, K. S. Trivedi, R. M. Geist, and V. F. Nicola, "Extended stochastic petri nets: Applications and analysis," inProc. 10th Int. Symp. Comput. Performance (PERFORMANCE 84), Dec. 1984, pp. 507-520.
[12] D. Eckhardt and L. Lee, "A theoretical basis for the analysis of multiversion software subject to coincident errors,"IEEE Trans. Software Eng., vol. SE-11, 1985.
[13] M. R. Girgis and M. R. Woodward, "An experimental comparison of the error exposing ability of program testing criteria," inProc. Workshop Software Testing, July 1986, IEEE Computer Society Press, pp. 64-73.
[14] M. A. Holliday and M. K. Vernon, "A generalized timed Petri net model for performance analysis,"IEEE Trans. Software Eng., vol. SE-13, no. 12, pp. 1297-1310, Dec. 1987.
[15] J. Kelly and A. Avizienis, "A specification-oriented multi-version software experiment," inProc. FTCS-13, Milan, Italy, 1983.
[16] K. N. King and A. J. Offutt, "A Fortran language system for mutation-based software testing,"Software Practice and Experience, to be published.
[17] J. Knight and N. Leveson, "An experimental evaluation of the assumption of independence in multiversion programming,"IEEE Trans. Software Eng., vol. SE-12, no. 1, pp. 96-109, Jan. 1986.
[18] J. Knight and N. Leveson, "An empirical study of failure probabilities in multiversion software," inProc. FTCS-16, Vienna, Austria, 1986.
[19] B. Littlewood and D. Miller, "A conceptual model of multiversion software," inProc. FTCS-17, Pittsburgh, PA, 1987.
[20] M. Ajmone Marsan, G. Balbo, and G. Conte, "A class of generalized stochastic Petri nets for the performance evaluation of multiprocessor systems,"ACM Trans. Comput. Syst., vol. 2, pp. 93-122, May 1984.
[21] M. K. Molloy, "Performance analysis using stochastic Petri nets,"IEEE Trans. Comput., vol. C-31, pp. 913-917, 1982.
[22] J. L. Peterson,Petri Net Theory and the Modeling of Systems. Englewood Cliffs, NJ: Prentice-Hall, 1981.
[23] Y.-B. Shieh, D. Ghosal, and S. Tripathi, "Modeling of fault-tolerant techniques in hierarchical systems," inProc. FTCS-19, Chicago, IL, 1989.
[24] K. S. Trivedi,Probability and Statistics with Reliability, Queueing and Computer Science Applications. Englewood Cliffs, NJ: Prentice-Hall, 1982.
[25] B. Withers, D. Rich, D. Lowman, and R. Buckland, "Software requirements: Guidance and control software development specification," NASA Contractor Rep. 182058, NASA Langley Research Center, June 1990.
[26] T. Yoneda, K. Nakade, and Y. Tohma, "A fast timing verification method based on the independence of units," inProc. FTCS-19, Chicago, IL, 1989.

Index Terms:
simulation; NASA; real-time software reliability; mutation analysis; numerical estimates; stochastic Petri net; synchronization structure; dependencies; correlated sampling; module execution times; mutation testing; software faults; planetary lander control software; mutation-tested code; fault-tolerant software systems; computational complexity; fault tolerant computing; Petri nets; software reliability.
R. Geist, A.J. Offutt, F.C. Harris, Jr., "Estimation and Enhancement of Real-Time Software Reliability Through Mutation Analysis," IEEE Transactions on Computers, vol. 41, no. 5, pp. 550-558, May 1992, doi:10.1109/12.142681
Usage of this product signifies your acceptance of the Terms of Use.