This Article 
 Bibliographic References 
 Add to: 
The Effect of Imperfect Error Detection on Reliability Assessment via Life Testing
February 1994 (vol. 20 no. 2)
pp. 142-148

Measurement of software reliability by life testing involves executing the software on large numbers of test cases and recording the results. The number of failures observed is used to bound the failure probability even if the number of failures observed is zero. Typical analyses assume that all failures that occur are observed, but, in practice, failures occur without being observed. In this paper, we examine the effect of imperfect error detection, i.e. the situation in which a failure of the software may not be observed. If a conventional analysis associated with life testing is used, the confidence in the bound on the failure probability is optimistic. Our results show that imperfect error detection does not necessarily limit the ability of life testing to bound the probability of failure to the very low values required in critical systems. However, we show that the confidence level associated with a bound on failure probability cannot necessarily be made as high as desired, unless very strong assumptions are made about the error detection mechanism. Such assumptions are unlikely to be met in practice, and so life testing is likely to be useful only for situations in which very high confidence levels are not required.

[1] S. S. Brilliant, "Testing software using multiple versions," Ph.D. dissertation, Univ. Virginia, Charlottesville, Sept. 1987.
[2] R. W. Butler, G. B. Finelli, "The infeasibility of the experimental quantification of life-critical software reliability,"Proc. ACM SIGSOFT'91: Software For Critical Systems, New Orleans, LA, 1991, pp. 66-76.
[3] R. H. Cobb, H. D. Mills, "Engineering software under statistical quality control,"IEEE Software, vol. 16, pp. 44-54, Nov. 1990.
[4] Federal Aviation Administration, "System design analysis," Advisory Circular AC-25.1309-1, U.S. Department of Transportation, Sept. 7, 1982.
[5] N. G. Leveson, P. R. Harvey, "Analyzing software safety,"IEEE Trans. Software Eng., vol. SE-9, no. 5, pp. 569-579, Sept. 1983.
[6] R. J. Lipton, "New directions in testing,"Interface '90, East Lansing, MI, May 1990.
[7] D. R. Miller, "The role of statistical modeling and inference in software quality assurance,"CSR Workshop on Software Certification, Gatwick, England, Oct. 1988.
[8] D. R. Miller, "Making statistical inferences about software reliability," NASA Contractor Rep. 4197, Dec. 1988.
[9] K. W. Miller, L. J. Morell, R. E. Noonan, S. K. Park, D. M. Nicol, B. W. Murrill, J. M. Voas, "Estimating the probability of failure when testing reveals no failures,"IEEE Trans. Software Eng., vol. 18, pp. 33-43, Jan. 1992.
[10] J. D. Musaet al., Software Reliability Measurement, Prediction, Application. New York: McGraw-Hill International, 1987.
[11] D.L. Parnas, A.J. van Shouwen, and S.P. Kwan, "Evaluation of Safety-Critical Software,"Comm. ACM, Vol. 33, No. 6, June 1990, pp. 636-648.
[12] D. P. Siewiorek, R. S. Swarz,The Theory and Practice of Reliable System Design. Bedford, MA: Digital Press, 1982.
[13] T. A. Thayer, M. Lipow, E. C. Nelson,Software Reliability. North Holland, 1978.
[14] E. J. Weyuker, "On testing non-testable programs,"Comput. J., vol. 25, no. 4, Nov. 1982.

Index Terms:
software reliability; error detection; life testing; probability; imperfect error detection; software reliability assessment; life testing; test cases; failure probability; unobserved failures; test oracles; critical systems; confidence level; software testing
P.E. Ammann, S.S. Brilliant, J.C. Knight, "The Effect of Imperfect Error Detection on Reliability Assessment via Life Testing," IEEE Transactions on Software Engineering, vol. 20, no. 2, pp. 142-148, Feb. 1994, doi:10.1109/32.265635
Usage of this product signifies your acceptance of the Terms of Use.