The Community for Technology Leaders
RSS Icon
Issue No.01 - January (2008 vol.20)
pp: 26-39
UCS, a sUpervised Classifier System, is an accuracy-based evolutionary learning classifier system for data mining classification tasks. UCS works through two stages: exploration and exploitation. During the exploration phase, a population of rules is evolved in order to represent a complete solution of the target problem. The exploitation phase is responsible for applying the rule-based knowledge obtained in the first phase when the system is exposed to unseen data. The representation of a rule in UCS as a univariate classification rule can be easily seen in a symbolic form, which is easy for a human to understand and comprehend (i.e. expressive power). However, the system may generate a large number of rules to cover the input space. Artificial neural networks normally provide a more compact and accurate representation. However, it is not a straightforward task to understand the network. In this paper, we propose a novel way to incorporate neural networks into UCS. The approach offers a good compromise between compactness, expressiveness, and accuracy. By using a simple artificial neural network as the classifier's action, we obtain smaller/compact population size, better generalization, while maintaining a reasonable level of expressiveness. We also apply negative correlation learning (NCL) during the training of the resultant neural network ensemble. NCL is shown to improve the generalization of the ensemble.
Rule-based processing, Representations (procedural and rule-based), Learning, Knowledge modeling
Hai H. Dam, Hussein A. Abbass, Chris Lokan, Xin Yao, "Neural-Based Learning Classifier Systems", IEEE Transactions on Knowledge & Data Engineering, vol.20, no. 1, pp. 26-39, January 2008, doi:10.1109/TKDE.2007.190671
[1] H.A. Abbass, M. Towsey, and G.D. Finn, “C-Net: A Method for Generating Non-Deterministic and Dynamic Multivariate Decision Trees,” Knowledge and Information Systems, vol. 3, no. 2, pp.184-197, 2001.
[2] E. Bernadó-Mansilla and J.M. Garrell-Guiu, “Accuracy-Based Learning Classifier Systems: Models, Analysis, and Applications to Classification Tasks,” Evolutionary Computation, vol. 11, no. 3, pp. 209-238, 2003.
[3] E. Bernadó-Mansilla, X. Llorà, and J.M. Garrell-Guiu, “XCS and GALE: A Comparative Study of Two Learning Classifier Systems with Six Other Learning Algorithms on Classification Tasks,” Proc. Fourth Int'l Workshop Learning Classifier Systems (IWLCS '01), short version published in Proc. Genetic and Evolutionary Computation Conf. (GECCO '01), pp. 337-341, 2001.
[4] G. Brown, “Diversity in Neural Network Ensembles,” PhD dissertation, School of Computer Science, Univ. of Birmingham, 2004.
[5] G. Brown, J. Wyatt, R. Harris, and X. Yao, “Diversity Creation Methods: A Survey and Categorisation,” Information Fusion, vol. 6, no. 1, pp. 5-20, 2005.
[6] L. Bull, “On Using Constructivism in Neural Classifier Systems,” Parallel Problem Solving from Nature, 7, pp. 558-567, 2002.
[7] L. Bull and T. O'Hara, “Accuracy-Based Neuro and Neuro-Fuzzy Classifier Systems,” Proc. Genetic and Evolutionary Computation Conf. (GECCO '02), pp. 905-911, 2002.
[8] M.V. Butz, “Rule-Based Evolutionary Online Learning Systems: Learning Bounds, Classification, and Prediction,” PhD dissertation, Univ. of Illinois at Urbana–Champaign, 2004.
[9] M.V. Butz, “Kernel-Based, Ellipsoidal Conditions in the Real-Valued XCS Classifier System,” Proc. Conf. Genetic and Evolutionary Computation (GECCO '05), pp. 1835-1842, 2005.
[10] H.H. Dam, H.A. Abbass, and C. Lokan, “Be Real! XCS with Continuous Valued Inputs,” Proc. Eighth Int'l Workshop Learning Classifier Systems (IWLCS '05), 2005.
[11] P. Duell, I. Fermin, and X. Yao, “Speciation Techniques in Evolved Ensembles with Negative Correlation Learning,” IEEE Congress on Evolutionary Computation, pp. 16-21, 2006.
[12] D.E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley, 1989.
[13] S. Haykin, Neural Networks, a Comprehensive Foundation, second ed. Prentice Hall, 1999.
[14] J.H. Holland, Adaptation in Natural and Artificial Systems. Univ. of Michigan Press, 1975, republished by the MIT Press, 1992.
[15] J.H. Holland, “Properties of the Bucket Brigade,” Proc. First Int'l Conf. Genetic Algorithms, pp. 1-7, 1985.
[16] J.H. Holland, “Escaping Brittleness: The Possibilities of General-Purpose Learning Algorithms Applied to Parallel Rule-Based Systems,” Machine Learning, an Artificial Intelligence Approach, Mitchell, Michalski, and Carbonell, eds., vol.2, chapter 20, pp.593-623. Morgan Kaufmann, 1986.
[17] K. Hornik, M. Stinchcombe, and H. White, “Multilayer Feedforward Networks Are Universal Approximators,” Neural Network, vol. 2, no. 5, pp. 359-366, 1989.
[18] J. Hurst and L. Bull, “A Self-Adaptive Neural Learning Classifier System with Constructivism for Mobile Robot Control,” Parallel Problem Solving from Nature–PPSN VIII, pp. 942-951, Springer, 2004.
[19] M.M. Islam, X. Yao, and K. Murase, “A Constructive Algorithm for Training Cooperative Neural Network Ensembles,” IEEE Trans. Neural Networks, vol. 14, no. 4, pp. 820-834, 2003.
[20] T. Kovacs, “What Should a Classifier System Learn and How Should We Measure It,” J. Soft Computing, vol. 6, nos. 3-4, pp. 171-182, June 2002.
[21] P.L. Lanzi and A. Perrucci, “Extending the Representation of Classifier Conditions Part I: From Messy Coding to S-Expressions,” Proc. Genetic and Evolutionary Computation Conf., W.Banzhaf, J.Daida, A.E. Eiben, M.H. Garzon, V. Honavar, M.Jakiela, and R.E. Smith, eds., vol. 1, pp. 337-344, 1999.
[22] P.L. Lanzi and A. Perrucci, “Extending the Representation of Classifier Conditions Part II: From Messy Coding to S-Expressions,” Proc. Genetic and Evolutionary Computation Conf., W.Banzhaf, J. Daida, A.E. Eiben, M.H. Garzon, V. Honavar, M.Jakiela, and R.E. Smith, eds., vol. 1, pp. 345-352, 1999.
[23] W.P. Lincoln and J. Skrzypek, “Synergy of Clustering Multiple Back Propagation Networks,” Advances in Neural Information Processing Systems 2, pp. 650-659, 1990.
[24] Y. Liu, “Negative Correlation Learning and Evolutionary Design of Neural Network Ensembles,” PhD dissertation, Univ. College, Univ. of New South Wales, Australian Defence Force Academy, 1999.
[25] Y. Liu and X. Yao, “Negatively Correlated Neural Networks Can Produce Best Ensembles,” Australian J. Intelligent Information Processing Systems, vol. 4, no. 3/4, pp. 176-185, 1997.
[26] Y. Liu, X. Yao, and T. Higuchi, “Evolutionary Ensembles with Negative Correlation Learning,” IEEE Trans. Evolutionary Computation, vol. 4, no. 4, pp. 380-387, 2000.
[27] R. McKay and H. Abbass, “Anti-Correlation: A Diversity Promoting Mechanism in Ensemble Learning,” Australian J. Intelligence Information Processing Systems, vol. 7, no. 3/4, pp. 139-149, 2001.
[28] D. Newman, S. Hettich, C. Blake, and C. Merz, “UCI Repository of Machine Learning Databases,” Dept. of Information and Computer Science, Univ. of California at Irvine, http://www.ics. , 1998.
[29] N.J. Nilsson, Learning Machines: Foundations of Trainable Pattern-Classifying Systems. McGraw Hill, 1965.
[30] S. Quartz and T. Sejnowski, “The Neural Basis of Cognitive Development: A Constructivist Manifesto,” Brain and Behavioral Sciences, vol. 20, no. 4, pp. 537-596, 1997.
[31] K. Shafi, H. Abbass, and W. Zhu, “The Role of Early Stopping and Population Size in XCS for Intrusion Detection,” Proc. Sixth Int'l Conf. Simulated Evolution and Learning (SEAL '06), pp. 50-57, 2006.
[32] A. Sharkey, “On Combining Artificial Neural Nets,” Connection Science, vol. 8, pp. 299-313, 1996.
[33] M. Skurichina, L. Kuncheva, and R. Duin, “Bagging and Boosting for the Nearest Mean Classifier: Effects of Sample Size on Diversity and Accuracy,” Proc. Int'l Workshop Multiple Classifier Systems, pp. 62-71, 2002.
[34] S.F. Smith, “A Learning System Based on Genetic Adaptive Algorithms,” PhD dissertation, Univ. of Pittsburgh, 1980.
[35] S.W. Wilson, “Classifier Fitness Based on Accuracy,” Evolutionary Computation, vol. 3, no. 2, pp. 149-175, 1995.
[36] S.W. Wilson, “Generalization in the XCS Classifier System,” Proc. Third Ann. Conf. Genetic Programming, J.R. Koza, W. Banzhaf, K.Chellapilla, K. Deb, M. Dorigo, D.B. Fogel, M.H. Garzon, D.E.Goldberg, H. Iba, and R. Riolo, eds., pp. 665-674, 1998.
[37] S.W. Wilson, “Get Real! XCS with Continuous-Valued Inputs,” Learning Classifier Systems, from Foundations to Applications, P.Lanzi, W. Stolzmann, and S. Wilson, eds., LNAI 1813, pp.209-219, Springer, 2000.
[38] S.W. Wilson, “Mining Oblique Data with XCS,” Proc. Third Int'l Workshop (IWLCS '00), P.L. Lanzi, W. Stolzmann, and S.W.Wilson,eds., pp. 158-174, 2001.
22 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool