This Article 
 Bibliographic References 
 Add to: 
Coverage Estimation Using Statistics of the Extremes for When Testing Reveals No Failures
January 2002 (vol. 51 no. 1)
pp. 3-12

The existing classes of fault coverage models require an a priori distribution for collected data in their analysis. Using these models, analyses can be performed using various assumed distributions. The assumed distributions may not accurately reflect the behavior of the collected data and, as a result, the coverage values predicted by the models may be inaccurate, especially if testing yields little or no failure data. Since the occurrence of an uncovered fault in an ultra-dependable system is a rare event, then statistics of the extremes can be used to quantify uncoverage estimates in such systems. Statistics of the extremes provides for an analysis of rare event data without requiring any a priori knowledge of its distribution. It classifies most distributions into one of three asymptotic families; that is, in the limit, most distributions converge to one of three forms. Using statistics of the extremes, a coverage model is developed for when testing reveals no failures. From this model, the number of fault injection experiments required to demonstrate that a desired coverage level can be met is derived, as is the probability that this coverage estimate can be met.

[1] A. Ang and W. Tang, Probability Concepts in Engineering Planning and Design, vol. II, chapter 4. New York: John Wiley&Sons, 1984.
[2] J. Arlat et al., "Fault Injection for Dependability Validation: A Methodology and Some Applications," IEEE Trans. Software Eng., Feb. 1990, pp. 166-182.
[3] J. Arlat, Y. Crouzet, and J.C. Laprie, “Fault Injection for the Experimental Validation of Fault Tolerance,” Proc. Esprit Conf., pp. 791-805, Nov. 1991.
[4] T. Arnold, “The Concept of Coverage and Its Effect on the Reliability Model of Repairable Systems,” IEEE Trans. Computers, vol. 22, no. 3, Mar. 1973.
[5] A. Benso, M. Rebaudengo, I. Impagliazzo, and P. Marmo, Fault-List Collapsing for Fault Injection Experiments Proc. Ann. Reliability&Maintainability Symp. (RAMS '98), pp. 383-388, 1998.
[6] W. G. Bouricius, W. C. Carter, and P. R. Schneider,“Reliability modeling techniques for self-repairing computer systems,”inProc. 24th Nat. Conf., ACM, 1969, pp. 295–309.
[7] C. Cassandras, Discrete Event Systems: Modeling and Performance Analysis, Appendix I. Boston: Aksen Assoc. and Richard D. Irwin, 1993.
[8] E. Castillo, Extreme Value Theory in Engineering, chapter 3. San Diego, Calif.: Academic Press, 1988.
[9] E. Castillo, E. Alvarez, A. Cobe, and T. Herrero, “An Expert System for the Analysis of Extreme Value Problems,” Dept. of Applied Math. and Computational Sciences, Univ. of Cantabria, Santander, Spain, 1993.
[10] E. Castillo and J. Sarabia, “Engineering Analysis of Extreme Value Data: Selection of Models,” J. Waterway, Port, Coastal. and Ocean Eng., vol. 118, no. 2, pp. 129-146, Mar./Apr. 1992.
[11] C. Constantinescu, “Using Multi-Stage&Stratified Sampling for Inferring Fault Coverage Probabilities,” IEEE Trans. Reliability, vol. 44, no. 4, pp. 632-639, 1995.
[12] M. Cukier, J. Arlat, and D. Powell, “Frequentist and Bayesian Estimations with Stratified Fault Injection,” Proc. Sixth IFIP Int'l Working Conf. Dependable Computing for Critical Applications (DCCA-6), pp. 38-57, Mar. 1997.
[13] M. Cukier, D. Powell, and J. Arlat, “Coverage Estimation Methods for Stratified Fault-Injection,” IEEE Trans. Computers, vol. 48, no. 7, pp. 707-723, July 1999.
[14] W. Daehn, “Fault Injection Using Small Fault Samples,” J. Electronic Testing: Theory and Applications, vol. 2, pp. 191-203, 1991.
[15] J.L. Devore, Probability and Statistics for Engineering and the Sciences, fourth ed. Boston: Duxbury Press, 1995.
[16] J. B. Dugan and K. S. Trivedi,“Coverage modeling for dependability analysis of fault-tolerant systems,” IEEE Trans. on Computers, vol. 38, no. 6, pp. 775-787, June 1989.
[17] J. Güthoff and V. Sieh, Combining Software-Implemented and Simulation-Based Fault Injection into a Single Fault Injection Method Proc. 25th Int'l Symp. Fault-Tolerant Computing (FTCS-25), pp. 196-206, 1995.
[18] E. Gumbel, Statistics of the Extremes. New York: Columbia Press, 1958.
[19] B.W. Johnson, Design and Analysis of Fault-Tolerant Digital Systems, pp. 394-402. Reading, Mass.: Addison-Wesley, June 1989.
[20] L. Kaufman, J. Dugan, and B. Johnson, “Using Statistics of the Extremes for Software Reliability Analysis,” IEEE Trans. Reliability, vol. 48, no. 3, pp. 292-299, Sept. 1999.
[21] J. Lambert, N. Matalas, C. Ling, Y. Haimes, and D. Li, “Selection of Probability Distributions in Characterizing Risk of Extreme Events,” Risk Analysis, vol. 14, no. 5, pp. 731-742, May 1994.
[22] J.C. Laprie, “Dependable Computing and Fault Tolerance: Concepts and Terminology,” Proc. 15th Ann. Int'l Symp. Fault-Tolerant Computing, pp. 2-11, June 1985.
[23] J.C. Laprie, Dependability: Basic Concepts and Terminology—In English, French, German, and Japanese. Vienna: Springer-Verlag, 1992.
[24] A. Papoulis, Random Variables and Stochastic Processes. New York: McGraw-Hill, 1991.
[25] D. Powell, E. Martins, J. Arlat, and Y. Crouzet, “Estimators for Fault Tolerance Coverage Evaluation,” Proc. 23rd Int'l Symp. Fault-Tolerant Computing (FTCS-23), pp. 228-237, Toulouse, France, 1993 (extended version in IEEE Trans. Computers, vol. 44, no. 2, pp. 347-366, Feb. 1995).
[26] D. Powel, E. Martins, J. Arlat, and Y. Crouzet, “Estimators for Fault Tolerance Coverage Evaluation,” IEEE Trans. Computers, vol. 44, no. 2, pp. 261-274, Feb. 1995.
[27] S. Ross, Introduction to Probability Models. New York: Academic Press, 1985.
[28] D. Smith, B. Johnson, N. Andrianos, and J. Profeta III, “A Variance Reduction Technique via Fault Expansion for Fault Coverage Estimation,” IEEE Trans. Reliability, vol. 46, no. 3, pp. 366-374, Sept. 1997.
[29] D.T. Smith et al., "A Method to Determine Equivalent Fault Classes for Permanent and Transient Faults," Proc. Ann. Reliability and Maintainability Symp., IEEE, 1995, pp. 418-424.
[30] D. Smith, B.W. Johnson, and J.A. Profeta III, “System Dependability Evaluation via a Fault List Generation Algorithm,” IEEE Trans. Computers, vol. 45, no. 8, pp. 974-979, Aug. 1996.
[31] T.K. Tsai, M.-C. Hsueh, H. Zhao, Z. Kalbarczyk, and R.K. Iyer, “Stress-Based and Path-Based Fault Injection,” IEEE Trans. Computers, vol. 48, no. 11, pp. 1183-1201, Nov. 1999.
[32] W. Wang, K.S. Trivedi, B.V. Shah, and J.A. Profeta III, “The Impact of Fault Expansion on the Interval Estimate for Fault Detection Coverage,” Proc. 24th Int'l Symp. Fault-Tolerant Computing (FTCS-24), pp. 330-337, Austin, Tex., June 1994.
[33] W. Weibull, “A Statistical Theory of the Strength of Materials,” Ingeniörs Vetenskaps Akademian Handlingar, no. 151, Generalstabens Litografiska Anstalts, Stockholm, 1939.

Index Terms:
Coverage, rare events, statistics of the extremes.
Lori M. Kaufman, Barry W. Johnson, Joanne Bechta Dugan, "Coverage Estimation Using Statistics of the Extremes for When Testing Reveals No Failures," IEEE Transactions on Computers, vol. 51, no. 1, pp. 3-12, Jan. 2002, doi:10.1109/12.980013
Usage of this product signifies your acceptance of the Terms of Use.