This Article 
 Bibliographic References 
 Add to: 
Symbolic Interpretation of Artificial Neural Networks
May/June 1999 (vol. 11 no. 3)
pp. 448-463

Abstract—Hybrid Intelligent Systems that combine knowledge-based and artificial neural network systems typically have four phases involving domain knowledge representation, mapping of this knowledge into an initial connectionist architecture, network training, and rule extraction, respectively. The final phase is important because it can provide a trained connectionist architecture with explanation power and validate its output decisions. Moreover, it can be used to refine and maintain the initial knowledge acquired from domain experts. In this paper, we present three rule-extraction techniques. The first technique extracts a set of binary rules from any type of neural network. The other two techniques are specific to feedforward networks, with a single hidden layer of sigmoidal units. Technique 2 extracts partial rules that represent the most important embedded knowledge with an adjustable level of detail, while the third technique provides a more comprehensive and universal approach. A rule-evaluation technique, which orders extracted rules based on three performance measures, is then proposed. The three techniques area applied to the iris and breast cancer data sets. The extracted rules are evaluated qualitatively and quantitatively, and are compared with those obtained by other approaches.

[1] R. Andrews, J. Diederich, and A. Tickle, "A Survey and Critique of Techniques for Extracting Rules from Trained Artificial Neural Networks," Knowledge-Based Systems, vol. 8, no. 6, pp. 373-389, Dec. 1995.
[2] R. Andrews and S. Geva, "Inserting and Extracting Knowledge from Constrained Error Backpropagation Networks," Proc. Sixth Australian Conf. Neural Networks,Sydney, Australia, 1995.
[3] R.K. Brayton, G.D. Hachtel, C.T. McMullen, and A.L. Sangiovanni-Vincintelli, Logic Minimization Algorithms for VLSI Synthesis.Boston: Kluwer Academic, 1984.
[4] T. Caelli and W. Bischof, "The Role of Machine Learning in Building Image Interpretation Systems," J. Artificial Intelligence and Pattern Recognition, vol. 11, p. 143-168, 1996.
[5] R. Challo, R.A. McLauchlan, D.A. Clark, and S.I. Omar, "A Fuzzy Neural Hybrid System," Proc. IEEE Int'l Conf. Neural Networks, vol. 3, pp. 1,654-1,657,Orlando, Fla., 1994.
[6] M.W. Craven and J.W. Shavlik, "Using Sampling and Queries to Extract Rules from Trained Neural Networks," Machine Learning: Proc. 11th Int'l Conf., pp. 37-45, 1994.
[7] L.M. Fu, "Knowledge-Based Connectionism for Revising Domain Theories," IEEE Trans. Systems, Man, and Cybernetics, vol. 23, no. 1, pp. 173-182, 1993.
[8] L.M. Fu, Neural Networks in Computer Intelligence.McGraw-Hill, 1994.
[9] J. Ghosh and K. Tumer, "Structural Adaptation and Generalization in Supervised Feed-Forward Networks," J. Artificial Neural Networks, vol. 1, no. 4, pp. 431-458, 1994.
[10] C.L. Giles et al., "Extracting and Learning an Unknown Grammar with Recurrent Neural Networks," S.J. Hanson, J.E. Moody, and R.P. Lippmann, eds., Advances in Neural Information Processing Systems—4.San Mateo, Calif.: Morgan Kaufmann, 1992.
[11] C.W. Glover, M. Silliman, M. Walker, and P. Spelt, "Hybrid Neural Network and Rule-Based Pattern Recognition System Capable of Self-Modification," Proc. SPIE, Application of Artificial Intelligence, vol. 8, pp. 290-300, 1990.
[12] J.A. Hendler, "Marker-Passing Over Microfeatures: Towards a Hybrid Symbolic/Connectionist Model," Cognitive Science, vol. 13, pp. 79-106, 1989.
[13] S. Horikawa, T. Furuhashi, and Y. Uchikawa, "On Fuzzy Modeling Using Fuzzy Neural Networks with Back-Propagation Algorithm," IEEE Trans. Neural Networks, vol. 3, no. 5, pp. 801-806, Sept. 1992.
[14] P. Howes and N. Crook, "Rule Extraction from Neural Networks," R. Andrews and J. Diederich, eds., Rules and Networks: Proc. Rule Extraction from Trained Artificial Neural Networks Workshop, pp. 60-67, Queensland Univ. of Technology, Neurocomputing Research Center, Apr. 1996.
[15] R.A. Jacobs, M.I. Jordan, S.J. Nowlan, and G.E. Hinton, "Adaptive Mixtures of Local Experts," Neural Computation, vol. 3, pp. 78-88, 1991.
[16] R. Kerber, "ChiMerge: Discretization of Numeric Attributes," Proc. 10th Nat'l Conf. Artificial Intelligence AAAI, pp. 123-128, July 1992.
[17] R.H. Klinkenberg and D.C. St. Clair, "Rule Set Quality Measures for Inductive Learning Algorithms," Intelligent Eng. Systems Through Artificial Neural Networks ANNIE, vol. 6, pp. 161-168. ASME Press, Nov. 1996.
[18] C.T. Lin and C.S.G. Lee, "Neural-Network-Based Fuzzy Logic Control and Decision System," IEEE Trans. Computers, vol. 40, no. 12, pp. 1,320-1,326, Dec. 1991.
[19] H. Liu and R. Setiono, "Chi2: Feature Selection and Discretization of Numeric Attributes," Proc. Seventh Int'l Conf. Tools with Artificial Intelligence, pp. 388-391, Nov. 1995.
[20] J.J. Mahoney and R.J. Mooney, "Combining Connectionist and Symbolic Learning to Refine Certainty Factor Rule Bases," Connection Science, vol. 5, nos. 3-4, pp. 339-364, 1993.
[21] O.L. Mangasarian and H.W. Wolberg, "Cancer Diagnosis via Linear Programming," SIAM News, vol. 23, no. 5, pp. 1-18, 1990.
[22] C. McMillan, M.C. Mozer, and P. Smolensky, "The Connectionist Scientist Game: Rule Extraction and Refinement in a Neural Network," Proc. 13th Ann. Conf. Cognitive Science Soc., 1991.
[23] W. Mendenhall, Introduction to Probability and Statistics, fifth ed. Wadsworth, 1979.
[24] P.M. Murphy and D.W. Aha, "UCI Repository of Machine Learning Database," technical report, Dept. of Computer Science, Univ. of California, 1992.
[25] C.W. Omlin and C.L. Giles, "Extraction of Rules from Discrete-Time Recurrent Neural Networks," Neural Networks, vol. 9, no. 1, pp. 41-52, 1996.
[26] D.W. Optiz and J.W. Shavlik, "Heuristically Expanding Knowledge-Based Neural Network," Proc. 13th Int'l Joint Conf. Artificial Intelligence, pp. 512-517, 1993.
[27] D. Ourston and R.J. Mooney, "Changing the Rules: A Comprehensive Approach to Theory Refinement," Proc. Eighth Nat'l Conf. Artificial Intelligence, pp. 815-820. AAAI Press, 1990.
[28] J.R. Quinlan, C4.5: Programs for Machine Learning,San Mateo, Calif.: Morgan Kaufman, 1992.
[29] S. Ramachandran, "Theory Refinement of Bayesian Networks with Hidden Variables," PhD thesis, Dept. of Computer Science, Univ. of Texas at Austin, Dec. 1997.
[30] R. Ruddel and A. Sangiovanni-Vincentelli, "Espresso-MV: Algorithms for Multiple-Valued Logic Minimization," Proc. Cust. Int'l Circ. Conf.,Portland, Ore., May 1985.
[31] K. Saito and R. Nakano, "Medical Diagnostic Expert System Based on PDP Model," Proc. IEEE Int'l Conf. Neural Networks, vol. 1, pp. 255-262, 1988.
[32] J.S. Schlimmer, "Concept Acquisition Through Representational Adjustment," PhD thesis, Dept. of Information and Science, Univ. of California at Irvine, May 1996.
[33] S. Sestito and T. Dillon, "Automated Knowledge Acquisition of Rules with Continuously Valued Attributes," Proc. 12th Int'l Conf. Expert Systems and Their Applications (AVIGNON), pp. 645-656, May 1995.
[34] R. Setiono, "Extracting Rules from Pruned Neural Networks for Breast Cancer Diagnosis," Artificial Intelligence in Medicine, vol. 8, no. 1, pp. 37-51, Feb. 1996.
[35] R. Setiono and H. Liu, "Understanding Neural Networks via Rule Extraction," Proc. 14th Int'l Joint Conf. Artificial Intelligence (IJCAI), pp. 480-485, 1995.
[36] R. Setiono and H. Liu, "Symbolic Representation of Neural Networks," Computer, pp. 71-77, Mar. 1996.
[37] N.E. Sharkey and A.J.C. Sharkey, "Understanding Catastrophic Interference in Neural Networks," Technical Report CS-94-4, Dept. of Computer Science, Sheffield, U.K., 1994.
[38] I. Taha, "A Hybrid Intelligent Architecture for Revising Domain Knowledge," PhD thesis, Military Technical College, Cairo, 1997.
[39] I. Taha and J. Ghosh, "A Hybrid Intelligent Architecture for Refining Input Characterization and Domain Knowledge," Proc. World Congress on Neural Networks (WCNN), vol. 2, pp. 284-287, July 1995.
[40] I. Taha and J. Ghosh, "Hybrid Intelligent Architecture and Its Application to Water Reservoir Control," Int'l J. Smart Eng. Systems, vol. 1, pp. 59-75, 1997.
[41] H. Takagi and I. Hayashi, "NN-Driven Fuzzy Reasoning," J.C. Bezdek and S.K. Pal, eds., Fuzzy Models for Pattern Recognition, pp. 496-512. IEEE Press, 1992.
[42] E. Tazaki and N. Inoue, "A Generation Methods for Fuzzy Rules Using Neural Networks with Planar Lattice Architecture," Proc. IEEE Int'l Conf. Neural Networks, vol. 3, pp. 1,743-1,748,Orlando, Fla., 1994.
[43] S.B. Thrun, J. Bala, E. Bloedorn, B. Cheng, I. Bratko, S. Dzeroski, K. De-Jong, S. Fahlman, D. Fisher, R. Hamann, K. Kaufman, S. Keller, I. Kononenko, J. Kreuziger, R. Michalski, T. Mitchell, P. Pachowicz, Y. Reich, H. Vafaie, K. Van de Welde, W. Wenzel, J. Wnek, and J. Zhang, "The Monk's Problem: A Performance Comparison of Different Learning Algorithms," Technical Report CMU-CS-91-197, Carnegie Mellon Univ., Dec. 1990.
[44] A.B. Tickle, M. Orlowski, and J. Diederich, "DEDEC: A Methodology for Extracting Rules from Trained Artificial Neural Networks," R. Andrews and J. Diederich, eds., Rules and Networks: Proc. Rule Extraction from Trained Artificial Neural Networks Workshop, pp. 90-102, Neurocomputing Research Center, Queensland Univ. of Tech nology, Apr. 1996.
[45] G.G. Towell and J.W. Shavlik, "The Extraction of Refined Rules from Knowledge-Based Neural Networks," Machine Learning, vol. 13, no. 1, pp. 71-101, 1993.
[46] G.G. Towell and J.W. Shavlik, "Knowledge-Based Artificial Neural Networks," Artificial Intelligence, vol. 70, nos. 1-2, pp. 119-165, 1994.
[47] G.G. Towell, J.W. Shavlik, and M.O. Noordwier, "Refinement of Approximate Domain Theories by Knowledge-Based Artificial Neural Network," Proc. Eighth Nat'l Conf. Artificial Intelligence, pp. 861-866, 1990.
[48] A.B. Tickle, R. Andrews, M. Golea, and J. Diederich, "The Truth Will Come to Light: Directions and Challenges in Extracting the Knowledge Embedded within Trained Artificial Neural Networks," IEEE Trans. Neural Networks, vol. 9, no. 6, pp. 1,057-1,068, 1998.
[49] K. McGarry, S. Wertmer, and J. MacIntyre, "Hybrid Neural Systems: From Simple Coupling to Fully Integrated Neural Networks," Neural Computing Surveys, vol. 2, pp. 62-93, 1999.

Index Terms:
Rule extraction, hybrid systems, knowledge refinement, neural networks, rule evaluation.
Ismail A. Taha, Joydeep Ghosh, "Symbolic Interpretation of Artificial Neural Networks," IEEE Transactions on Knowledge and Data Engineering, vol. 11, no. 3, pp. 448-463, May-June 1999, doi:10.1109/69.774103
Usage of this product signifies your acceptance of the Terms of Use.