This Article 
 Bibliographic References 
 Add to: 
Estimating the Probability of Failure When Testing Reveals No Failures
January 1992 (vol. 18 no. 1)
pp. 33-43

Formulas for estimating the probability of failure when testing reveals no errors are introduced. These formulas incorporate random testing results, information about the input distribution; and prior assumptions about the probability of failure of the software. The formulas are not restricted to equally likely input distributions, and the probability of failure estimate can be adjusted when assumptions about the input distribution change. The formulas are based on a discrete sample space statistical model of software and include Bayesian prior assumptions. Reusable software and software in life-critical applications are particularly appropriate candidates for this type of analysis.

[1] G. Becker and L. Camarinopoulos, "A Bayesian estimation method for the failure rate of a possibly correct program,"IEEE Trans. Software Eng., vol. 16, pp. 1307-1310, Nov. 1990.
[2] S. Brocklehurst, P. Y. Chan, B. Littlewood, and J. Snell, "Recalibrating software reliability models,"IEEE Trans. Software Eng., vol. 16, pp. 458-470, Apr. 1990.
[3] J. R. Brown and M. Lipow, "Testing for software reliability," inProc. Int. Conf. Reliable Software, Los Angeles, CA, Apr. 1975, pp. 518-527.
[4] R. C. Cheung, "A user-oriented software reliability model,"IEEE Trans. Software Eng., vol. SE-6, pp. 118-125, Mar. 1980.
[5] J. W. Duran and S. C. Ntafos, "An evaluation of random testing,"IEEE Trans. Software Eng., vol. SE-10, pp. 438-444, July 1984.
[6] W. Feller,An Introduction to Probability Theory and its Applications, Vol. I, 3rd ed. New York: Wiley, 1970.
[7] R. G. Hamlet, "Probable correctness theory,"Inform. Proc. Lett., pp. 17-25, Apr. 1987.
[8] R. Hamlet and R. Taylor, "Partition testing does not inspire confidence," inProc. 2nd Workshop on Software Testing, Verification, and Analysis, July 1988, pp. 206-215.
[9] B. Jeng and E. Weyuker, "Some observations on partition testing," inProc. ACM SIGSOFT'89: 3rd Symp. on Software Testing, Analysis, and Verification (TAV3), Dec. 1989, pp. 38-47.
[10] M. Lenz, H. A. Schmid, and P. F. Wolf, "Software reuse through building blocks,"IEEE Software, vol. 4, pp. 34-42, July 1987.
[11] Y. Levendel, "Reliability analysis of large software systems: defect data modeling,"IEEE Trans. Software Eng., vol. 16, pp. 141-152, Feb. 1990.
[12] B. Littlewood and J. Verrall, "A Bayesian reliability model with a stochastically monotone failure rate,"IEEE Trans. Rel., vol. R-23, pp. 108-114, June 1974.
[13] D. R. Miller, "Making statistical inferences about software reliability," NASA Contractor Rep. No. 4197, Dec. 1988.
[14] H. D. Mills, "The new math of computer programming,"CACM, vol. 18, no. 1, pp. 43-48, Jan. 1975.
[15] B. W. Murrill and L. J. Morell, "Error flow testing," Dept. Computer Sci., College of William and Mary, Williamsburg, VA, Tech. Rep. WM-90-1, Feb. 1990.
[16] J. D. Musaet al., Software Reliability Measurement, Prediction, Application. New York: McGraw-Hill International, 1987.
[17] J. Neyman, "Outline of a theory of statistical estimation based on the classical theory of probability,"Phil. Trans. Roy. Soc., London A., vol. 236, p. 333, 1937.
[18] C. V. Ramamoorthy and F. B. Bastini, "Software reliability-status and perspectives,"IEEE Trans. Software Eng., vol. SE-8, pp. 354-370, July 1982.
[19] G. J. Schick and R. W. Wolverton, "An analysis of competing software reliability models,"IEEE Trans. Software Eng., vol. SE-4, pp. 104-120, Mar. 1978.
[20] Special Committee 152, "Software considerations in airborne system and equipment certification," Radio Tech. Commission for Aeronautics, Washington, DC, DO-178A, Mar. 1985.
[21] T. A. Thayer, M. Lipow, and E. C. Nelson,Software Reliability(TRW Series of Software Techn., vol. 2). New York: North-Holland, 1978.
[22] J. M. Voas and L. J. Morell, "Fault sensitivity analysis (PIA) applied to computer programs," Dept. Computer Sci., College of William and Mary, Williamsburg, VA, Tech. Rep. 89-4, Dec. 1989.
[23] S. N. Weiss, "What to compare when comparing test data adequacy criteria,"Sofrware Eng. Notes, vol. 14, no. 6, pp. 42-49, Oct. 1989.
[24] E. J. Weyuker and B. Jeng, "Analyzing partition testing strategies,"IEEE Trans. Software Eng., vol. 17, pp. 703-711, July 1991.
[25] S. N. Weiss and E. J. Weyuker, "An extended domain-based model of software reliability,"IEEE Trans. Software Eng., vol. 14, pp. 1512-1524, Oct. 1988.

Index Terms:
failure probability estimation; formulas; random testing results; input distribution; prior assumptions; failure estimate; discrete sample space statistical model; Bayesian prior assumptions; life-critical applications; Bayes methods; probability; program testing
K.W. Miller, L.J. Morell, R.E. Noonan, S.K. Park, D.M. Nicol, B.W. Murrill, M. Voas, "Estimating the Probability of Failure When Testing Reveals No Failures," IEEE Transactions on Software Engineering, vol. 18, no. 1, pp. 33-43, Jan. 1992, doi:10.1109/32.120314
Usage of this product signifies your acceptance of the Terms of Use.