Subscribe

Issue No.03 - March (2014 vol.26)

pp: 652-666

Dominik Fisch , University of Kassel, Kassel

Edgar Kalkowski , University of Kassel, Kassel

Bernhard Sick , University of Kassel, Kassel

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TKDE.2013.20

ABSTRACT

If knowledge such as classification rules are extracted from sample data in a distributed way, it may be necessary to combine or fuse these rules. In a conventional approach this would typically be done either by combining the classifiers' outputs (e.g., in form of a classifier ensemble) or by combining the sets of classification rules (e.g., by weighting them individually). In this paper, we introduce a new way of fusing classifiers at the level of parameters of classification rules. This technique is based on the use of probabilistic generative classifiers using multinomial distributions for categorical input dimensions and multivariate normal distributions for the continuous ones. That means, we have distributions such as Dirichlet or normal-Wishart distributions over parameters of the classifier. We refer to these distributions as hyperdistributions or second-order distributions. We show that fusing two (or more) classifiers can be done by multiplying the hyperdistributions of the parameters and derive simple formulas for that task. Properties of this new approach are demonstrated with a few experiments. The main advantage of this fusion approach is that the hyperdistributions are retained throughout the fusion process. Thus, the fused components may, for example, be used in subsequent training steps (online training).

INDEX TERMS

Probabilistic logic, Bayesian methods, Covariance matrix, Knowledge engineering, Training, Data mining, Coordinate measuring machines,Bayesian techniques, Knowledge fusion, classifier fusion, probabilistic classifier, generative classifier, data mining

CITATION

Dominik Fisch, Edgar Kalkowski, Bernhard Sick, "Knowledge Fusion for Probabilistic Generative Classifiers with Data Mining Applications",

*IEEE Transactions on Knowledge & Data Engineering*, vol.26, no. 3, pp. 652-666, March 2014, doi:10.1109/TKDE.2013.20REFERENCES

- [1] D. Fisch, M. Jänicke, E. Kalkowski, and B. Sick, "Learning from Others: Exchange of Classification Rules in Intelligent Distributed Systems,"
Artificial Intelligence, vol. 187-188, pp. 90-114, 2012.- [2] C.M. Bishop,
Pattern Recognition and Machine Learning. Springer, 2006.- [3] D. Fisch, B. Kühbeck, B. Sick, and S.J. Ovaska, "So Near and Yet So Far: New Insight into Properties of Some Well-Known Classifier Paradigms,"
Information Sciences, vol. 180, no. 18, pp. 3381-3401, 2010.- [4] N. Bouguila, "Hybrid Generative/Discriminative Approaches for Proportional Data Modeling and Classification,"
IEEE Trans. Knowledge and Data Eng., vol. 24, no. 12, pp. 2184-2202, Dec. 2012.- [5] T.M. Hospedales, S. Gong, and T. Xiang, "Finding Rare Classes: Active Learning with Generative and Discriminative Models,"
IEEE Trans. Knowledge and Data Eng., vol. 25, no. 2, pp. 374-386, Feb. 2013.- [6] D. Fisch, T. Gruber, and B. Sick, "SwiftRule: Mining Comprehensible Classification Rules for Time Series Analysis,"
IEEE Trans. Knowledge and Data Eng., vol. 23, no. 5, pp. 774-787, May 2011.- [7] J. Sander and J. Beyerer, "Fusion Agents—Realizing Bayesian Fusion via a Local Approach,"
Proc. IEEE Int'l Conf. Multisensor Fusion and Integration for Intelligent Systems, pp. 249-254, 2006.- [8] A. Makarenko and H. Durrant-Whyte, "Decentralized Data Fusion and Control in Active Sensor Networks,"
Proc. Seventh Int. Conf. Information Fusion, pp. 479-486, 2004.- [9] O. Punska, "Bayesian Approaches to Multi-Sensor Data Fusion," master's thesis, Dept. of Eng., Univ. of Cambridge, 1999.
- [10] H. Durrant-Whyte and T. Henderson, "Multisensor Data Fusion,"
Springer Handbook of Robotics, B. Siciliano and O. Khatib, eds. chapter 25, pp. 585-610, Springer, 2008.- [11] L. Iocchi, N. Monekosso, D. Nardi, M. Nicolescu, P. Remagnino, and M. Valera, "Smart Monitoring of Complex Public Scenes,"
Proc. Assoc. for the Advancement of Artificial Intelligence (AAAI) Fall Symp., 2011.- [12] P.K. Atrey and M.S. Kankanhalli, "Probability Fusion for Correlated Multimedia Streams,"
Proc. ACM Int'l Conf. Multimedia, pp. 408-411, 2004.- [13] A. Barreiro, S. Liu, N. Namachchivaya, P. Sauer, and R. Sowers, "Data Assimilation in the Detection of Vortices,"
Applications of Nonlinear Dynamics, Series Understanding Complex Systems, V. In, P. Longhini, and A. Palacios, eds., pp. 47-59, Springer, 2009.- [14] L.I. Kuncheva, "A Theoretical Study on Six Classifier Fusion Strategies,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 281-286, Feb. 2002.- [15] M. Zhang, H. Song, S. Lv, Y. Li, X. Yu, and J. Bao, "Research on the Multi-Sensors Information Fusion Technique Based on the Neural Networks and Its Application,"
Proc. Int'l Workshop Knowledge Discovery and Data Mining, pp. 93-96, 2009.- [16] B. Verma and A. Rahman, "Cluster Oriented Ensemble Classifier: Impact of Multi-Cluster Characterisation on Ensemble Classifier Learning,"
IEEE Trans. Knowledge and Data Eng., vol. 24, no. 4, pp. 605-618, Apr. 2011.- [17] X. Ceamanos, B. Waske, J.A. Benediktsson, J. Chanussot, M. Fauvel, and J.R. Sveinsson, "A Classifier Ensemble Based on Fusion of Support Vector Machines for Classifying Hyperspectral Data,"
Int'l J. Image and Data Fusion, vol. 1, no. 4, pp. 293-307, 2010.- [18] P. Gray et al., "KRAFT: Knowledge Fusion from Distributed Databases and Knowledge Bases,"
Proc. Eighth Int. Workshop Database and Expert Systems Applications, pp. 682-691, 1997.- [19] K. ying Hui and P. Gray, "Constraint and Data Fusion in a Distributed Information System,"
Proc. 16th British Nat'l Conf. Databases: Advances in Databases, pp. 181-182, 1998.- [20] K. ying Hui, "Knowledge Fusion and Constraint Solving in a Distributed Environment," PhD dissertation, Dept. of Computing Science, Univ. of Aberdeen, 2000.
- [21] G. Pavlin, P. De Oude, M. Maris, J. Nunnink, and T. Hood, "A Multi Agent Systems Approach to Distributed Bayesian Information Fusion,"
Information Fusion, vol. 11, no. 3, pp. 267-282, 2010.- [22] E. SantosJr., J. Wilkinson, and E. Santos, "Bayesian Knowledge Fusion,"
Proc. 22nd Int'l FLAIRS Conf., pp. 559-564, 2009.- [23] Y. Wang, B. Wu, and J. Hu, "A Semantic Knowledge Fusion Method Based on Topic Maps,"
Proc. Workshop Intelligent Information Technology Application, pp. 74-76, 2007.- [24] H. Lu and B. Feng, "An Intelligent Topic Map-Based Approach to Detecting and Resolving Conflicts for Multi-Resource Knowledge Fusion,"
Information Technology J., vol. 8, no. 8, pp. 1242-1248, 2009.- [25] A. Smirnov, M. Pashkin, N. Chilov, and T. Levashova, "KSNET-Approach to Knowledge Fusion from Distributed Sources,"
Computing and Informatics, vol. 22, no. 2, pp. 105-142, 2003.- [26] O. Buchtala and B. Sick, "Techniques for the Fusion of Symbolic Rules in Distributed Organic Systems,"
Proc. IEEE Mountain Workshop Adaptive and Learning Systems, pp. 85-90, 2006.- [27] C.S.R. Fraser, L.F. Bertuccelli, H.-L. Choi, and J.P. How, "A Hyperparameter-Based Approach for Consensus under Uncertainties,"
Proc. Am. Control Conf., pp. 3192-3197, 2010.- [28] A.G. Foina, J. Planas, R.M. Badia, and F.J. Ramirez-Fernandez, "P-means, A Parallel Clustering Algorithm for a Heterogeneous Multi-Processor Environment,"
Proc. Int'l Conf. High Performance Computing and Simulation, pp. 239-248, 2011.- [29] Y. Li, K. Zhao, X. Chu, and J. Liu, "Speeding Up k-Means Algorithm by GPUs,"
Proc. 10th IEEE Int'l Conf. Computer and Information Technology, pp. 115-122, 2010.- [30] C.-T. Chu, S.K. Kim, Y.-A. Lin, Y. Yu, G. Bradski, A.Y. Ng, and K. Olukotun, "Map-Reduce for Machine Learning on Multicore,"
Proc. Advances in Neural Information Processing Systems, pp. 281-288, 2006.- [31] D. Fisch, S.J. Ovaska, E. Kalkowski, and B. Sick, "In Your Interest - Objective Interestingness Measures for a Generative Classifier,"
Proc. Third Int'l Conf. Agents and Artificial Intelligence, pp. 414-423, 2011.- [32] R.O. Duda, P.E. Hart, and D.G. Stork,
Pattern Classification. John Wiley & Sons, 2001.- [33] D. Fisch, F. Kastl, and B. Sick, "Novelty-Aware Attack Recognition - Intrusion Detection with Organic Computing Techniques,"
Proc. Seventh IFIP TC 10 Working Conf. Distributed, Parallel and Biologically Inspired Systems, pp. 242-253, 2010.- [34] D. Fisch and B. Sick, "Training of Radial Basis Function Classifiers with Resilient Propagation and Variational Bayesian Inference,"
Proc. Int'l Conf. Neural Networks, pp. 838-847, 2009.- [35] D. Fisch, "Intelligente Technische Systeme Mit Der Fähigkeit Zum Kollaborativen Wissenserwerb," PhD dissertation, Dept. of Electrical Eng. and Computer Science, Univ. of Kassel, 2012.
- [36] L.L. Cam and G. Yang,
Asymptotics in Statistics: Some Basic Concepts, second ed. Springer, 2000.- [37] K. Fukunaga,
Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1990.- [38] "UCL, UCL/MLG Elena Database," http://www.ucl.ac.be/mlgindex.php?page=Elena , 2007.
- [39] A. Frank and A. Asuncion, "UCI Machine Learning Repository," http://archive.ics.uci.eduml, 2010.
- [40] B. Kaluža, V. Mirchevska, E. Dovgan, M. Luštrek, and M. Gams, "An Agent-Based Approach to Care in Independent Living,"
Proc. First Int'l Joint Conf. Ambient Intelligence, pp. 177-186, 2010.- [41] D. Fisch, M. Jänicke, E. Kalkowski, and B. Sick, "Techniques for Knowledge Acquisition in Dynamically Changing Environments - Special Issue on Self-Adaptive Systems,"
ACM Trans. Autonomous and Adaptive Systems, vol. 7, no. 1, pp. 1-25, 2012. |