This Article 
 Bibliographic References 
 Add to: 
Optimal Test Distributions for Software Failure Cost Estimation
March 1995 (vol. 21 no. 3)
pp. 219-228
In this article, we generalize the input-domain based software reliability measures by Nelson and by Weiss and Weyuker, introducing expected failure costs under the operational distribution as a measure for software unreliability. This approach incorporates in the reliability concept a distinction between different degrees of failure severity. It is shown how to estimate the proposed quantity by means of random testing, using the Importance Sampling technique from Rare Event Simulation. A test input distribution that yields an unbiased estimator with minimum variance is determined. The practical application of the presented method is outlined, and a detailed numerical example is given.

[1] B. Beizer, Software Testing Techniques, second ed. Boston, Mass.: Int'l Thomson Computer Press, 1990.
[2] B. Boehm,“Software risk management,”inProc. ESEC,Warwick, U.K., Sept. 1989, pp. 1–19.
[3] R. W. Butler and G. B. Finelli,“The infeasibility of quantifying the reliability of life-critical software,”IEEE Trans. Software Eng., vol. 19, pp. 3–12, Jan. 1993.
[4] L. Devroye,Non-Uniform Random Variate Generation. New York: Springer, 1986.
[5] W. D. Ehrenberger,“Combining probabilistic and deterministic verification efforts,”inProc. SAFECOMP '92,H. Frey, Ed. Oxford, England: Pergamon, 1992, pp. 299–304.
[6] J. S. Gourlay,“A mathematical framework for the investigation of testing,”IEEE Trans. Software Eng., vol. SE-9, pp. 686–709, Nov. 1983.
[7] W. J. Gutjahr,“Probabilistic software test efficiency measures and optimal test data selection,”in preparation.
[8] R.G. Hamlet, “Probable Correctness Theory,” Information Processing Letters, vol. 25, pp. 17–25, Apr. 1987.
[9] R. G. Hamlet,“Theoretical comparison of testing methods,”inProc. Third Symp. Testing, Anal., Verification, Key West, FL, Dec. 1989, pp. 28–37.
[10] M. H. Kalos and P. A. Whitlock,Monte Carlo Methods. New York: Wiley, 1986, vol. I.
[11] H. A. Linstone and M. Turoff,The Delphi Method. Reading, PA: Addison-Wesley, 1975.
[12] K. W. Milleret al.,“Estimating the probability of failure when testing reveals no failures,”IEEE Trans. Software Eng., vol. 18, pp. 33–43, Jan. 1992.
[13] P. G. Moore,The Business of Risk. Cambridge, England: Cambridge University Press, 1993.
[14] E. C. Nelson,“A statistical basis for software reliability assessment,”TRW-SS-73-03, Mar. 1973.
[15] E. C. Nelson,“Estimating software reliability from test data,”Microelectron., Reliability, vol. 17, pp. 67–74, 1978.
[16] C. V. Ramamoorthy and F. B. Bastani,“Software reliability—Status and perspectives,”IEEE Trans. Software Eng., vol. SE-8, pp. 354–371, July 1982.
[17] R. W. Selby, V. R. Basili, and F. T. Baker,“Cleanroom software development: An empirical investigation,”IEEE Trans. Software Eng., vol. SE-13, pp. 1027–1037, Sept. 1987.
[18] S. A. Sherer,Software Failure Risk.New York: Plenum, 1992.
[19] M. Smithson,Ignorance and Uncertainty. New York: Springer, 1988.
[20] Th. A. Thayer, M. Lipow, and E. C. Nelson,Software Reliability. Amsterdam, The Netherlands: North-Holland, 1978.
[21] S. N. Weiss and E. J. Weyuker,“An extended domain–based model of software reliability,”IEEE Trans. Software Eng., vol. 14, pp. 1512–1524, Oct. 1988.
[22] E. J. Weyuker and T. J. Ostrand,“Theories of program testing and the application of revealing subdomains,”IEEE Trans. Software Eng., vol. SE-6, pp. 236–245, May 1980.
[23] E. J. Weyuker, S. N. Weiss, and D. Hamlet,“Comparison of program testing strategies,”inProc. Symp. Testing, Anal., Verification(Victoria, B.C.), Oct. 1991, pp. 1–10.

Index Terms:
Input distributions, input-domain-based models, reliability estimation, risk assessment, software failure costs, software reliability, software testing
Walter J. Gutjahr, "Optimal Test Distributions for Software Failure Cost Estimation," IEEE Transactions on Software Engineering, vol. 21, no. 3, pp. 219-228, March 1995, doi:10.1109/32.372149
Usage of this product signifies your acceptance of the Terms of Use.