|
| This Article | ||
| ||
| Share | ||
| Bibliographic References | ||
| Add to: | ||
 Digg  Furl  Spurl  Blink  Simpy  Google  Del.icio.us  Y!MyWeb | ||
| Search | ||
| ||
| ASCII Text | x | ||
| P.E. Ammann, S.S. Brilliant, J.C. Knight, "The Effect of Imperfect Error Detection on Reliability Assessment via Life Testing," IEEE Transactions on Software Engineering, vol. 20, no. 2, pp. 142-148, February, 1994. | |||
| BibTex | x | ||
| @article{ 10.1109/32.265635, author = {P.E. Ammann and S.S. Brilliant and J.C. Knight}, title = {The Effect of Imperfect Error Detection on Reliability Assessment via Life Testing}, journal ={IEEE Transactions on Software Engineering}, volume = {20}, number = {2}, issn = {0098-5589}, year = {1994}, pages = {142-148}, doi = {http://doi.ieeecomputersociety.org/10.1109/32.265635}, publisher = {IEEE Computer Society}, address = {Los Alamitos, CA, USA}, } | |||
| RefWorks Procite/RefMan/Endnote | x | ||
| TY - JOUR JO - IEEE Transactions on Software Engineering TI - The Effect of Imperfect Error Detection on Reliability Assessment via Life Testing IS - 2 SN - 0098-5589 SP142 EP148 EPD - 142-148 A1 - P.E. Ammann, A1 - S.S. Brilliant, A1 - J.C. Knight, PY - 1994 KW - software reliability; error detection; life testing; probability; imperfect error detection; software reliability assessment; life testing; test cases; failure probability; unobserved failures; test oracles; critical systems; confidence level; software testing VL - 20 JA - IEEE Transactions on Software Engineering ER - | |||
Measurement of software reliability by life testing involves executing the software on large numbers of test cases and recording the results. The number of failures observed is used to bound the failure probability even if the number of failures observed is zero. Typical analyses assume that all failures that occur are observed, but, in practice, failures occur without being observed. In this paper, we examine the effect of imperfect error detection, i.e. the situation in which a failure of the software may not be observed. If a conventional analysis associated with life testing is used, the confidence in the bound on the failure probability is optimistic. Our results show that imperfect error detection does not necessarily limit the ability of life testing to bound the probability of failure to the very low values required in critical systems. However, we show that the confidence level associated with a bound on failure probability cannot necessarily be made as high as desired, unless very strong assumptions are made about the error detection mechanism. Such assumptions are unlikely to be met in practice, and so life testing is likely to be useful only for situations in which very high confidence levels are not required.
[1] S. S. Brilliant, "Testing software using multiple versions," Ph.D. dissertation, Univ. Virginia, Charlottesville, Sept. 1987.
[2] R. W. Butler, G. B. Finelli, "The infeasibility of the experimental quantification of life-critical software reliability,"Proc. ACM SIGSOFT'91: Software For Critical Systems, New Orleans, LA, 1991, pp. 66-76.
[3] R. H. Cobb, H. D. Mills, "Engineering software under statistical quality control,"IEEE Software, vol. 16, pp. 44-54, Nov. 1990.
[4] Federal Aviation Administration, "System design analysis," Advisory Circular AC-25.1309-1, U.S. Department of Transportation, Sept. 7, 1982.
[5] N. G. Leveson, P. R. Harvey, "Analyzing software safety,"IEEE Trans. Software Eng., vol. SE-9, no. 5, pp. 569-579, Sept. 1983.
[6] R. J. Lipton, "New directions in testing,"Interface '90, East Lansing, MI, May 1990.
[7] D. R. Miller, "The role of statistical modeling and inference in software quality assurance,"CSR Workshop on Software Certification, Gatwick, England, Oct. 1988.
[8] D. R. Miller, "Making statistical inferences about software reliability," NASA Contractor Rep. 4197, Dec. 1988.
[9] K. W. Miller, L. J. Morell, R. E. Noonan, S. K. Park, D. M. Nicol, B. W. Murrill, J. M. Voas, "Estimating the probability of failure when testing reveals no failures,"IEEE Trans. Software Eng., vol. 18, pp. 33-43, Jan. 1992.
[10] J. D. Musaet al., Software Reliability Measurement, Prediction, Application. New York: McGraw-Hill International, 1987.
[11] D.L. Parnas, A.J. van Shouwen, and S.P. Kwan, "Evaluation of Safety-Critical Software,"Comm. ACM, Vol. 33, No. 6, June 1990, pp. 636-648.
[12] D. P. Siewiorek, R. S. Swarz,The Theory and Practice of Reliable System Design. Bedford, MA: Digital Press, 1982.
[13] T. A. Thayer, M. Lipow, E. C. Nelson,Software Reliability. North Holland, 1978.
[14] E. J. Weyuker, "On testing non-testable programs,"Comput. J., vol. 25, no. 4, Nov. 1982.