This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Verification and Validation of Knowledge-Based Systems
January/February 1999 (vol. 11 no. 1)
pp. 202-212

Abstract—Knowledge-based systems (KBS) are being used in many applications areas where their failures can be costly because of the losses in services, property, or even life. To ensure their reliability and dependability, it is therefore important that these systems are verified and validated before they are deployed. This paper provides perspectives on issues and problems that impact the verification and validation (V&V) of KBS. Some of the reasons V&V of KBS is difficult are presented. The paper also provides an overview of different techniques and tools that have been developed for performing V&V activities. Finally, some of the research issues that are relevant for future work in this field are discussed.

[1] Validation, Verification, and Test of Knowledge-Based Systems, M. Ayel and J.P. Laurent, eds., John Wiley and Sons, Chichester, U.K., 1991.
[2] M. Ayel and J.-P. Laurent, "SACCOSYCOJET: Two Different Ways of Verifying Knowledge-Based Systems," Validation, Verification and Test of Knowledge Based Systems, M. Ayel and J.-P. Laurent, eds. Chichester, U.K.: John Wiley&Sons, 1991.
[3] F.B. Bastani and I.R. Chen, "Assessment of Reliability of Artificial Intelligence Programs," Proc. IEEE Conf. Tools for Artificial Intelligence, pp. 753-759, 1990.
[4] V. Barker and D. O'Connor, "Expert Systems for Configuration at Digital: XCON and Beyond," Comm. ACM, Vol. 32, No. 3, Mar. 1989, pp. 298-318.
[5] B. Beizer, Software Testing Techniques, Van Nostrand Reinhold, New York, 1990.
[6] N. Botten, "Complex Knowledge Base Verification Using Matrices," Lecture Notes in Artificial Intelligence, F. Belli and F.J. Radermacher, eds., pp. 225-235, Springer-Verlag, Berlin, 1992.
[7] B.G. Buchanan, "Artificial Intelligence As An Experimental Science," J.E. Fetzer, ed., Aspects of Artificial Intelligence, Kluwer, pp. 209-250, 1988.
[8] C.L. Chang, J.B. Combs, and R.A. Stachowitz, "A Report on the Expert Systems Validation Associate (EVA)," Expert Systems with Applications, vol. 1, pp. 217-230, 1990.
[9] E. Charles and O. Dubois, "MELODIA: Logical Methods for Checking Knowledge Bases," Validation, Verification and Test of Knowledge Based Systems, M. Ayel and J.-P. Laurent, eds. Chichester, U.K.: John Wiley, 1991.
[10] B.J. Cragun and H.J. Steudel, "A Decision-Table-Based Processor for Checking Completeness and Consistency in Rule-Based Expert Systems," Int'l J. Man-Machine Studies, vol. 26, pp. 633-648, 1987.
[11] S. Craw, "Judging Knowledge Base Quality," Validation, Verification, and Test of Knowledge-Based Systems, M. Ayel and J.P. Laurent, ed., pp. 207-219, John Wiley and Sons, Chichester, U.K., 1991.
[12] Expert Systems With Applications, special issue on verification and validation of knowledge-based systems, C. Culbert, ed., vol. 1, no. 3, 1990.
[13] L. De Raedt, G. Sablon, and M. Bruynooghe, "Using Interactive Concept Learning for Knowledge-Base Validation and Verification," Validation, Verification, and Test of Knowledge-Based Systems, M. Ayel and J.P. Laurent, ed., pp. 177-190, John Wiley and Sons, Chichester, U.K., 1991.
[14] A. Ginsberg, S. Weiss, and P. Politakis, "SEEK2: A Generalized Approach to Automatic Knowledge Base Refinement," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 367-374, 1985.
[15] A. Ginsberg, "Knowledge-Base Reduction: A New Approach to Checking Knowledge Bases for Inconsistency and Redundancy," Proc. Seventh Nat'l Conf. Artificial Intelligence, pp. 585-589, 1988.
[16] A. Ginsberg and K. Williamson, "Inconsistency and Redundancy Checking for Quasi-First-Order-Logic Knowledge Bases," Int'l J. Expert Systems, vol. 6, no. 3, pp. 321-340 1993.
[17] Validating and Verifying Knowledge-Based Systems, U.G. Gupta, ed., IEEE CS Press, Los Alamitos, Calif., 1991.
[18] D. Hamilton, K. Kelley, and C. Culbert, "State-of-the-Practice in Knowledge-Based System Verification and Validation," Expert Systems With Applications, vol. 3, pp. 403-410, 1991.
[19] F. Hayes-Roth and N. Jacobstein, "The State of Knowledge-Based Systems," Comm. ACM, Mar. 1994, pp. 27-39.
[20] H. Huang, W.T. Tsai, and S. Subramanian, "Generalized Program Slicing for Software Maintenance," Proc. Software Eng. and Knowledge Eng., pp. 261-268, 1996.
[21] IEEE Std. 610.12-1990, Standard Glossary of Software Engineering Terminology, IEEE Press, Piscataway, N.J., 1990.
[22] S. Kirani, I.A. Zualkernan, and W.T. Tsai, "Evaluation of Expert System Testing Methods," Comm. ACM, vol. 37, no. 11, pp. 71-81, 1994.
[23] P. Lafon, "A Descriptive Model of Predicates for Verifying Production Systems," Validation, Verification, and Test of Knowledge-Based Systems, M. Ayel and J.P. Laurent, eds., John Wiley and Sons, Chichester, U.K., pp. 149-162, 1991.
[24] T.J. Lydiard, "Overview of Current Practice and Research Initiatives for the Verification and Validation of KBS," Knowledge Eng. Rev., vol. 7, no. 2, pp. 101-113, 1992.
[25] H. Marathe, T.K. Ma, and C.C. Liu, "An Algorithm for Identification of Relations Among Rules," Proc. IEEE Int'l Workshop Tools for Artificial Intelligence, pp. 360-366, 1989.
[26] D. Marr, Vision, W.H. Freeman, New York, 1982.
[27] P. Meseguer and A. Verdaguer, "Verification of Multi-Level Rule-Based Expert Systems: Theory and Practice," Int'l J. Expert Systems, vol. 6, no. 2, pp. 163-192, 1993.
[28] L.A. Miller, "Recommended Guidelines for V&V of Various Kinds of Systems at Various Lifecycle Phases," Proc. AAAI Workshop Validation and Verification of Knowledge-Based Systems, pp. 1-9, 1994.
[29] J. Musa, A. Iannino, and K. Okumoto, Software Reliability: Measurement, Prediction, Application, McGraw-Hill, New York, 1987.
[30] D.L. Nazareth, “Issues in the Verification of Knowledge in Rule-Based Systems” Int'l J. Man-Machine Studies, pp. 255-271, 1989.
[31] T.A. Nguyen, W.A. Perkins, T.J. Laffey, and D. Pecora, “Knowledge Base Verification,” AI Magazine, pp. 69–75, Summer 1987.
[32] R.M. O'Keefe and D.E. O'Leary, "Expert System Verification and Validation: A Survey and Tutorial," Artificial Intelligence Rev., vol. 7, pp. 3-42, 1993.
[33] D.E. O'Leary, "Design, Development, and Validation of Expert Systems: A Survey of Developers," Validation, Verification, and Test of Knowledge-Based Systems, M. Ayel and J.P. Laurent, eds., John Wiley and Sons, Chichester, U.K., pp. 3-19, 1991.
[34] Int'l J. Intelligent Systems, special issue on verification and validation of intelligent systems: five years of AAAI workshops," D.E. O'Leary, ed., vol. 9, nos. 8-9, 1994.
[35] "Validation and Verification of Knowledge-Based Systems," E. Plaza, ed., IEEE Expert, vol. 8, pp. 45-81, 1993.
[36] A. Preece, R. Shinghal, and A. Batarekh, "Verifying Expert Systems: A Logical Framework and A Practical Tool," Expert Systems with Applications, vol. 5, nos. 2-3, pp. 421-436, 1992.
[37] Int'l J. Expert Systems A. Preece and C. Suen, eds., special issues on verification and validation of knowledge-based systems, vol. 6, nos. 2-3, 1993.
[38] R.S. Pressman, Software Eng.: A Practitioner's Approach, McGraw-Hill, New York, 1987.
[39] M.C. Rousset, "On the Consistency of Knowledge Bases: The COVADIS System," Proc. Eighth European Conf. Artificial Intelligence, pp. 79-84, 1988.
[40] J. Rushby, "Quality Measures and Assurance for Artificial Intelligence Software," NASA Contractor Report 4187, Oct. 1988.
[41] J. Rushby, "Formal Verification of Artificial Intelligence Software," NASA Contractor Report 181827, Feb. 1989.
[42] S. Subramanian, W.T. Tsai, and S. Kirani, "Hierarchical Data Flow Analysis for OO Programs," J. Object-Oriented Programming, vol. 7, no. 2, pp. 36-46, 1994.
[43] W.T. Tsai and I.A. Zualkernan, "Towards a Unified Framework for Testing Expert Systems," Proc. Int'l Conf. Software Eng. and Knowledge Eng., 1990.
[44] W.T. Tsai, I.A. Zualkernan, and S. Kirani, "Pragmatic Testing Methods for Expert Systems," Int'l J. Artificial Intelligence Tools, vol. 2, no. 2, pp. 181-217, 1993.
[45] D.C. Wilkins and B.G. Buchanan, "On Debugging Rule Sets When Reasoning Under Uncertainty," Proc. Fourth Nat'l Conf. Artificial Intelligence, pp. 448-454, 1986.
[46] D. Zhang and D. Nguyen, "A Tool for Knowledge Base Verification," N. Bourbakis, ed., Knowledge Eng. Shells—Systems and Techniques, Advanced Series on Artificial Intelligence, vol. 2, World Scientific, pp. 455-486, 1993.
[47] D. Zhang and D. Nguyen, “Prepare: A Tool for Knowledge Base Verification,” IEEE Trans. Knowledge and Data Eng., vol. 6, no. 6, pp. 983–989, Dec. 1994.
[48] N. Zlatareva and A. Preece, "State of the Art in Automated Validation of Knowledge-Based Systems," Expert Systems with Applications, vol. 7, no. 2, pp. 151-167, 1994.
[49] I.A. Zualkernan, W.T. Tsai, P.E. Johnson, and J.H. Moller, "Utility of Knowledge-Level Specifications," Proc. Fourth Ann. Artificial Intelligence and Advanced Computer Technology Conf., pp. 79-85, 1988.

Index Terms:
Expert systems, knowledge-based systems, verification, validation, reliability, dependability, testing.
Citation:
Wei-Tek Tsai, Rama Vishnuvajjala, Du Zhang, "Verification and Validation of Knowledge-Based Systems," IEEE Transactions on Knowledge and Data Engineering, vol. 11, no. 1, pp. 202-212, Jan.-Feb. 1999, doi:10.1109/69.755629
Usage of this product signifies your acceptance of the Terms of Use.