This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
On Combining Classifiers
March 1998 (vol. 20 no. 3)
pp. 226-239

Abstract—We develop a common theoretical framework for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions—the sum rule—outperforms other classifier combinations schemes. A sensitivity analysis of the various schemes to estimation errors is carried out to show that this finding can be justified theoretically.

[1] K.M. Ali and M.J. Pazzani, "On the Link Between Error Correlation and Error Reduction in Decision Tree Ensembles," Technical Report 95-38, ICS-UCI, 1995.
[2] S.C. Bagui and N.R. Pal, "A Multistage Generalization of the Rank Nearest Neighbor Classification Rule," Pattern Recognition Letters, vol. 16, no. 6, pp. 601-614, 1995.
[3] J. Cao, M. Ahmadi, and M. Shridhar, "Recognition of Handwritten Numerals With Multiple Feature and Multistage Classifier," Pattern Recognition, vol. 28, no. 2, pp. 153-160, 1995.
[4] C. Sung-Bae and J.H. Kim, “Combining Multiple Neural Networks by Fuzzy Integral for Robust Classification,” IEEE Trans. Systems, Man,and Cybernetics, vol. 25, no. 2, pp. 380-384, 1995.
[5] S.B. Cho and J.H. Kim, "Multiple Network Fusion Using Fuzzy Logic," IEEE Trans. Neural Networks, vol. 6, no. 2, pp. 497-501, 1995.
[6] D.A. Denisov and A.K. Dudkin, "Model-Based Chromosome Recognition Via Hypotheses Construction/Verification," Pattern Recognition Letters, vol. 15, no. 3, pp. 299-307, 1994.
[7] H. El-Shishini, M.S. Abdel-Mottaleb, M. El-Raey, and A. Shoukry, "A Multistage Algorithm for Fast Classification of Patterns," Pattern Recognition Letters, vol. 10, no. 4, pp. 211-215, 1989.
[8] M.C. Fairhurst and H.M.S. Abdel Wahab, "An Interactive Two-Level Architecture for a Memory Network Pattern Classifier," Pattern Recognition Letters, vol. 11, no. 8, pp. 537-540, 1990.
[9] J. Franke and E. Mandler, "A Comparison of Two Approaches for Combining the Votes of Cooperating Classifiers," Proc. 11th IAPR Int'l Conf. Pattern Recognition, Conf. B: Pattern Recognition Methodology and Systems, vol. 2, pp. 611-614, 1992.
[10] L.K. Hansen and P. Salamon, “Neural Network Ensembles,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 10, pp. 993-1001, Oct. 1990.
[11] Hashem and B. Schmeiser, "Improving Model Accuracy Using Optimal Linear Combinations of Trained Neural Networks," IEEE Trans. Neural Networks, vol. 6, no. 3, pp. 792-794, 1995.
[12] T.K. Ho, “Random Decision Forests,” Proc. Third Int'l Conf. Document Analysis and Recognition, pp. 278-282, 1995.
[13] T.K. Ho, J.J. Hull, and S.N. Srihari, “Decision Combination in Multiple Classifiers Systems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 1, pp. 66-75, Jan. 1994.
[14] F. Kimura and M. Shridhar, "Handwritten Numerical Recognition Based on Multiple Algorithms," Pattern Recognition, vol. 24, no. 10, pp. 969-983, 1991.
[15] A. Krogh and J. Vedelsby, "Neural Network Ensembles, Cross Validation, and Active Learning," Advances in Neural Information Processing Systems 7, G. Tesauro, D.S. Touretzky, and T.K. Leen, eds. Cambridge, Mass.: MIT Press, 1995.
[16] M.W. Kurzynski, "On the Identity of Optimal Strategies for Multistage Classifiers," Pattern Recognition Letters, vol. 10, no. 1, pp. 39-46, 1989.
[17] P. Pudil, J. Novovicova, S. Blaha, and J. Kittler, Multistage Pattern Recognition with Reject Option Proc. Int'l Conf. Pattern Recognition Methodology and Systems, pp. 92-95, 1992.
[18] G. Rogova, "Combining the Results of Several Neural Network Classifiers," Neural Networks, vol. 7, no. 5, pp. 777-781, 1994.
[19] M. Skurichina and R.P.W. Duin, "Stabilizing Classifiers for Very Small Sample Sizes," Proc. 11th IAPR Int'l Conf. Pattern Recognition,Vienna, 1996.
[20] V. Tresp and M. Taniguchi, "Combining Estimators Using Non-Constant Weighting Functions," Advances in Neural Iinformation Processing Systems 7, G. Tesauro, D.S. Touretzky, and T.K. Leen, eds. Cambridge, Mass.: MIT Press, 1995.
[21] C.H. Tung, H.J. Lee, and J.Y. Tsai, "Multi-Stage Pre-Candidate Selection in Handwritten Chinese Character Recognition Systems," Pattern Recognition, vol. 27, no. 8, pp. 1,093-1,102, 1994.
[22] D.H. Wolpert, "Stacked Generalization," Neural Networks, vol. 5, no. 2, pp. 241-260, 1992.
[23] L. Xu, A. Krzyzak, and C.Y. Suen, “Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition,” IEEE Trans. Systems, Man, and Cybernetics, vol. 22, no. 3, pp. 418-435, 1992.
[24] J.Y. Zhou and T. Pavlidis, "Discrimination of Characters by a Multi-Stage Recognition Process," Pattern Recognition, vol. 27, no. 11, pp. 1,539-1,549, 1994.
[25] J. Kittler, M. Hatef, and R.P.W. Duin, “Combining Classifiers,” Proc. 13th Int'l Conf. Pattern Recognition, pp. 897-901, Vienna, Aug. 1996.
[26] J. Kittler, J. Matas, K. Jonsson, and M.U. Ramos Sánchez, "Combining Evidence in Personal Identity Verification Systems," Pattern Recognition Letters, pp. 845-852, 1997.
[27] L. Breiman, "Bagging Predictors," Technical Report 421, Dept. of Statistics, Univ. of California at Berkeley, 1994.
[28] Y. Freund and R.E. Shapire, "Experiments With a New Boosting Algorithm," Proc. 13th Int'l Conf. Machine Learning, 1996.
[29] R.E. Shapire, Y. Freund, P. Bartlett, and W.S. Lee, "Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods," Proc. 14th Int'l Conf. Machine Learning, 1997.
[30] K. Tumer and J. Ghosh, "Analysis of Decision Boundaries in Linearly Combined Neural Classifiers," Pattern Recognition, vol. 29, pp. 341-348, 1996.
[31] K. Tumer and J. Ghosh, "Classifier Combining: Analytical Results and Implications," Proc. Nat'l Conf. Artificial Intelligence,Portland, Ore., 1996.
[32] K.S. Woods, K. Bowyer, and W.P. Kergelmeyer, "Combination of Multiple Classifiers Using Local Accuracy Estimates," Proc. CVPR '96, pp. 391-396, 1996.
[33] J. Kittler, A. Hojjatoleslami, and T. Windeatt, "Weighting Factors in Multiple Expert Fusion," Proc. British Machine Vision Conf.,Colchester, England, pp. 41-50, 1997.
[34] J. Kittler, A. Hojjatoleslami, and T. Windeatt, "Strategies for Combining Classifiers Employing Shared and Distinct Pattern Representations," Pattern Recognition Letters, to appear.
[35] J. Kittler, "Improving Recognition Rates by Classifier Combination: A Theoretical Framework," Frontiers of Handwriting Recognition 5, A.G. Downton and S. Impedovo, eds. World Scientific, pp. 231-247, 1997.
[36] Y.S. Huang and C.Y. Suen, A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, pp. 90-93, 1998.
[37] P.A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach.Englewood Cliffs, N.J.: Prentice Hall, 1982.
[38] M. Hatef and J. Kittler, "Constraining Probabilistic Relaxation With Symbolic Attributes," Proc. Sixth Int'l Conf. Computer Analysis of Images and Patterns, V. Hlavac and R. Sara, eds., pp. 862-867,Prague, 1995.
[39] M. Hatef and J. Kittler, "Combining Symbolic With Numeric Attributes in Multiclass Object Recognition Problems," Proc. Second Int'l Conf. Image Processing, vol. 3, pp. 364-367,Washington, D.C., 1995.
[40] A.J. Elms, "A Connected Character Recogniser Using Level Building of HMMs," Proc. 12th IAPR Int'l Conf. Neural Networks, Conf. B: Pattern Recognition Methodology and Systems, vol. 2, pp. 439-441, 1994.
[41] J. Matas, K. Jonsson, and J. Kittler, "Fast Face Localisation and Verification," A. Clark, ed., British Machine Vision Conf., pp. 152-161, BMVA Press, 1997.
[42] S. Pigeon and L. Vandendrope, "The M2VTS Multimodal Face Database (Release 1.00)," J. Bigun, G. Chollet, and G. Borgefors, eds., Audio- and Video-Based Biometric Person Authentication, pp. 403-409. Springer, 1997.
[43] D. Genoud, G. Gravier, F. Bimbot, and G. Chollet, "Combining Methods to Improve the Phone Based Speaker Verification Decision," Proc. Int'l Conf. Speech and Language Processing, vol 3, pp. 1,756-1,760,Philadelphia, 1996.
[44] S. Pigeon and L. Vandendrope, "Profile Authentication Using a Chamfer Matching Algorithm," J. Bigun, G. Chollet, and G. Borgefors, eds., Audio- and Video-Based Biometric Person Authentication, pp. 185-192. Springer, 1997.

Index Terms:
Classification, classifier combination, error sensitivity.
Citation:
Josef Kittler, Mohamad Hatef, Robert P.W. Duin, Jiri Matas, "On Combining Classifiers," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, March 1998, doi:10.1109/34.667881
Usage of this product signifies your acceptance of the Terms of Use.