This Article 
 Bibliographic References 
 Add to: 
Reliability Estimation During Prototyping of Knowledge-Based Systems
June 1995 (vol. 7 no. 3)
pp. 378-390

Abstract—Many knowledge based systems are designed and built with little attention paid to the reliability of the output. In this paper, we present an approach, using partitioning of both the knowledge base and the input space, that allows for the measurement of the reliability during any program increment in a rapid prototyping development cycle. Before presenting the approach, we formalize the problem using concepts from general systems theory and then describe our three objectives: 1) measurement of the reliability of the knowledge-based system at the current program increment, 2) prediction of the reliability of the future system, and 3) support for design decisions.

Finally, we apply our approach to a design-aiding knowledge-based system for the selection of materials under various climatic conditions. The design-aiding knowledge-based system is used by U.S. Army personnel in the development of equipment to be used by the U.S. Army in various regions of the world. We find that the current system, containing 40 rules, has a reliability of approximately 0.85. However, more importantly, we have discovered the rules that led to many of the failures.

[1] A.A. Abdel-Ghaly,P.Y. Chan,, and B. Littlewood,“Evaluation of competing software reliability predictions,” IEEE Trans. on Software Engineering, vol. 12, no. 9, 1986.
[2] M.R. Anderberg,“Cluster analysis for applications,” Probability and Mathematical Statistics, Z.W. Birnbaum and E. Lukacs, eds., Academic Press, Inc., New York, N.Y., 1973.
[3] J. Bader et al., “Practical engineering of knowledge-based systems,” Information and Software Technology, vol. 30, no. 5, 1988.
[4] L. Breiman et al., Classification and Regression Trees, The Wadsworth Statistics/Probability Series, Wadsworth&Brooks, Monterey, Calif., 1984.
[5] D.E. Brown,“A method for obtaining software reliability measures during development,” IEEE Trans. Reliability, vol. 36, no. 5, 1987.
[6] D.E. Brown and B.G. Duren,“Conflicting information integration for decision support,” Decision Support Systems, vol. 2, no. 4, 1986.
[7] D.E. Brown,C.L. Huntley,, and P.J. Garvey,“Clustering of homogeneous subsets,” Pattern Recognition Letters, vol. 12, no. 7, 1991.
[8] A.G. Colombo and D. Constantini,“Ground-hypotheses for beta distribution as Bayesian prior,” IEEE Trans. Reliability, vol. 29, no. 1, 1980.
[9] M.R. Garey and D.S. Johnson,“Computers and intractability: A guide to the theory of NP-completeness,” Mathematical Sciences, V. Klee, ed., W.H. Freeman and Co., New York, N.Y., 1979.
[10] J. Gaschnig et al., “Evaluation of expert systems: issues and case studies,” Building Expert Systems, F. Hayes-Roth, ed., Addison-Wesley, Reading, Mass., 1983.
[11] F. Golshani,“Rule-based expert systems,” Knowledge Engineering, Vol. 1: Fundamentals, H. Adeli, ed., McGraw-Hill, New York, N.Y., 1990.
[12] C.J.R. Green and M.M. Keyes,“Verification and validation of expert systems,” WESTEX 87, 1987.
[13] J.A Hartigan,Clustering Algorithms, John Wiley and Sons, New York, N.Y., 1975.
[14] F. Hayes-Roth,D.A. Waterman,, and D.B. Lenat,Building Expert Systems, Addison-Wesley, Reading, Mass., 1983.
[15] S. Hayward,“A structured development methodology for expert systems,” KBS 86, London, 1986.
[16] F. Hickman,“Knowledge acquisition: The key to success for commercial expert systems,” KBS 86, London, 1986.
[17] A.K. Jain and R.C. Dubes, Algorithms for Clustering Data. Englewood Cliffs, N.J.: Prentice Hall, 1988.
[18] Z. Jelinski and P. Moranda,“Software reliability research,” Statistical Computer Performance Evaluation, W. Freiberger, ed., Academic Press, 1972.
[19] N.L. Johnson and S. Kotz,Continuous Univariate Distributions, Wiley-Interscience, John Wiley and Sons, New York, N.Y., 1970.
[20] L. Kaufman and P.J. Rousseeuw,Finding Groups in Data: An Introduction to Cluster Analysis, Wiley Series in Probability and Mathematical Statistics, John Wiley and Sons, New York, N.Y., 1990.
[21] N.E. Lane,“Global issues in evaluation of expert systems,” IEEE Int’l Conf. Systems, Man, and Cybernetics,Atlanta, Ga., 1986.
[22] J. Laski,“Testing in the program development cycle,” Software Engineering J., vol. 4, no. 2, 1989.
[23] D.R. Martinez and M.G. Sobol,“Systems analysis techniques for the implementation of expert systems,” Information and Software Technology, vol. 30, no. 2, 1988.
[24] J.F. Meyer,“Reliable Design of Software,” Rational Fault Analysis, R. Saeks and S.R. Liberty, eds., Marcel Dekker, New York, N.Y., 1977.
[25] S.D. Moitra,“Skewness and the beta distribution,” J. Operational Research Soc., vol. 41, no. 10, 1990.
[26] J.D. Musa,“A theory of software reliability and its application,” IEEE Trans. Software Engineering, vol. 1, no. 3, 1975.
[27] E.C. Nelson,“Estimating software reliability from test data,” Microelectronics and Reliability, vol. 17, no. 1, 1978.
[28] A. Newell and H.A. Simon,“Computer science as empirical inquiry: Symbols and search,” Comm. ACM, vol. 19, 1976.
[29] D. Partridge.,Artificial Intelligence: Applications in the Future of Software Engineering, Ellis Horwood Series in Artificial Intelligence, J. Campbell, ed., Ellis-Horwood, Chichester, West Sussex, England, 1986.
[30] E. Parzen,“On the estimation of a probability density function and mode,” Annals of Mathematical Statistics, vol. 33, 1962.
[31] J.J. Pomykalski,Knowledge-Based System Design Enhancement Through Reliability Measurement, Ph.D. dissertation, Dept. of Systems Engineering,Univ. of Virginia, Jan. 1994.
[32] H. Raiffa and R. Schlaifer,Applied Statistical Decision Theory, MIT Press, Cambridge, Mass., 1961.
[33] J.W. Ratcliff and D.E. Metzener,“Pattern matching: The Gestalt approach,” Dr. Dobb’s J., 1988.
[34] M.L. Shooman.,“Software reliability: Measurement and models,, Proc. Ann. Reliability and Maintainability Symp.,Washington, D.C., 1975.
[35] M.L. Shooman,Software Engineering: Design, Reliability, and Management, McGraw-Hill, 1983.
[36] A.I. Wasserman,“Information system design methodology,” J. Am. Soc. of Information Sciences, vol. 31, no. 1, 1980.
[37] D.A. Waterman,A Guide to Expert Systems, Addison-Wesley, Reading, Mass., 1986.
[38] H. Weiler,“The use of incomplete beta functions for prior distributions in binomial sampling,” Technometrics, vol. 7, no. 3, 1965.
[39] S.M. Weiss and C.A. Kulikowski,A Practical Guide to Designing Expert Systems, Rowman&Allanheld, Totowa, N.J., 1984.

Index Terms:
Knowledge-based systems, software reliability modeling, rapid prototyping, reliability prediction, knowledge base design, knowledge base development.
Donald E. Brown, James J. Pomykalski, "Reliability Estimation During Prototyping of Knowledge-Based Systems," IEEE Transactions on Knowledge and Data Engineering, vol. 7, no. 3, pp. 378-390, June 1995, doi:10.1109/69.390245
Usage of this product signifies your acceptance of the Terms of Use.