This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems
June 2005 (vol. 27 no. 6)
pp. 942-956
In this paper, a theoretical and experimental analysis of linear combiners for multiple classifier systems is presented. Although linear combiners are the most frequently used combining rules, many important issues related to their operation for pattern classification tasks lack a theoretical basis. After a critical review of the framework developed in works by Tumer and Ghosh [30], [31] on which our analysis is based, we focus on the simplest and most widely used implementation of linear combiners, which consists of assigning a nonnegative weight to each individual classifier. Moreover, we consider the ideal performance of this combining rule, i.e., that achievable when the optimal values of the weights are used. We do not consider the problem of weights estimation, which has been addressed in the literature. Our theoretical analysis shows how the performance of linear combiners, in terms of misclassification probability, depends on the performance of individual classifiers, and on the correlation between their outputs. In particular, we evaluate the ideal performance improvement that can be achieved using the weighted average over the simple average combining rule and investigate in what way it depends on the individual classifiers. Experimental results on real data sets show that the behavior of linear combiners agrees with the predictions of our analytical model. Finally, we discuss the contribution to the state of the art and the practical relevance of our theoretical and experimental analysis of linear combiners for multiple classifier systems.

[1] L.A. Alexandre, A.C. Campilho, and M. Kamel, “Combining Independent and Unbiased Classifiers Using Weighted Average,” Proc. Int'l Conf. Pattern Recognition, pp. 495-498, 2000.
[2] J.A. Benediktsson, J.R. Sveinsson, O.K. Ersoy, and P.H. Swain, “Parallel Consensual Neural Networks,” IEEE Trans. Neural Networks, vol. 8, pp. 54-64, 1997.
[3] C.M. Bishop, Neural Networks for Pattern Recognition. Oxford: Clarendon Press, 1995.
[4] L. Breiman, “Stacked Regressions,” Machine Learning, vol. 24, pp. 49-64, 1996.
[5] L. Breiman, “Bagging Predictors,” Machine Learning, vol. 24, pp. 123-140, 1996.
[6] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, second ed. Wiley & Sons, 2000.
[7] Y. Freund and R.E. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting,” J. Computer and System Sciences, vol. 55, pp. 119-139, 1999.
[8] G. Fumera and F. Roli, “Performance Analysis and Comparison of Linear Combiners for Classifier Fusion,” Proc. Int'l Workshop Statistical Pattern Recognition, pp. 424-432, 2002.
[9] G. Fumera and F. Roli, “Linear Combiners for Classifier Fusion: Some Theoretical and Experimental Results,” Proc. Int'l Workshop Multiple Classifier Systems, pp. 74-83, 2003.
[10] S. Geman, E. Bienenstock, and R. Doursat, “Neural Networks and the Bias/Variance Dilemma,” Neural Computation, vol. 4, pp. 1-58, 1992.
[11] S. Hashem, “Optimal Linear Combination of Neural Networks,” PhD dissertation, Purdue Univ., 1993.
[12] S. Hashem and B. Schmeiser, “Improving Model Accuracy Using Optimal Linear Combinations of Trained Neural Networks,” IEEE Trans. Neural Networks, vol. 6, pp. 792-794, 1995.
[13] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer, 2001.
[14] T.K. Ho, J.J. Hull, and S.N. Srihari, “Decision Combination in Multiple Classifier Systems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, pp. 66-75, 1994.
[15] T.K. Ho, “The Random Subspace Method for Constructing Decision Forests,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 832-844, 1998.
[16] J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas, “On Combining Classifiers,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, pp. 226-239, 1998.
[17] M. Le Blanc and R. Tibshirani, “Combining Estimates in Regression and Classification,” Technical Report 9318, Dept. of Statistics, Univ. of Toronto, 1993.
[18] “Multiple Classifier Systems,” Lecture Notes in Computer Science, J. Kittler, and F. Roli, eds., vols. 1857 and 2096, 2000 and 2001.
[19] J. Kittler and F.M. Alkoot, “Sum versus Vote Fusion in Multiple Classifier Systems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, pp. 110-115, 2003.
[20] L.I. Kuncheva, “A Theoretical Study on Six Classifier Fusion Strategies,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, pp. 281-286, 2002.
[21] L.I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms. Hoboken, N.J.: Wiley, 2004.
[22] M.P. Perrone and L.N. Cooper, “When Networks Disagree: Ensemble Methods for Hybrid Neural Networks,” Neural Networks for Speech and Vision, R.J. Mammone, ed. pp. 126-142, New York: Chapman-Hall, 1993.
[23] F. Roli, “Multisensor Image Recognition by Neural Networks with Understandable Behaviour,” Int'l J. Pattern Recognition Artificial Intelligence, vol. 10, pp. 887-917, 1996.
[24] F. Roli and G. Fumera, “Analysis of Linear and Order Statistics Combiners for Fusion of Imbalanced Classifiers,” Proc. Int'l Workshop Multiple Classifier Systems, pp. 252-261, 2002.
[25] F. Roli, G. Fumera, and J. Kittler, “Fixed and Trained Combiners for Fusion of Unbalanced Pattern Classifiers,” Proc. Int'l Conf. Information Fusion, pp. 278-284, 2002.
[26] “Multiple Classifier Systems,” Lecture Notes in Computer Science, F. Roli, and J. Kittler, eds., vol. 2364, 2002.
[27] “Multiple Classifier Systems,” Lecture Notes in Computer Science, F. Roli, J. Kittler, and T. Windeatt, eds., vol. 3077, 2004.
[28] D.M.J. Tax, M. van Breukelen, R.P.W. Duin, and J. Kittler, “Combining Multiple Classifiers by Averaging or by Multiplying,” Pattern Recognition, vol. 33, pp. 1475-1485, 2000.
[29] K. Tumer, “Linear and Order Statistics Combiners for Reliable Pattern Classification,” PhD dissertation, The Univ. of Texas, Austin, 1996.
[30] K. Tumer and J. Ghosh, “Analysis of Decision Boundaries in Linearly Combined Neural Classifiers,” Pattern Recognition, vol. 29, pp. 341-348, 1996.
[31] K. Tumer and J. Ghosh, “Linear and Order Statistics Combiners for Pattern Classification,” Combining Artificial Neural Nets, A.J.C. Sharkey, ed. pp. 127-155, London: Springer, 1999.
[32] N. Ueda, “Optimal Linear Combination of Neural Networks for Improving Classification Performance,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, pp. 207-215, 2000.
[33] A. Verikas, A. Lipnickas, K. Malmqvist, M. Bacauskiene, and A. Gelzinis, “Soft Combination of Neural Classifiers: A Comparative Study,” Pattern Recognition Letters, vol. 20, pp. 429-444, 1999.
[34] “Multiple Classifier Systems,” Lecture Notes in Computer Science, T. Windeatt and F. Roli, eds., vol. 2709, 2003.

Index Terms:
Multiple classifier systems, linear combiners, classifier fusion, pattern classification.
Citation:
Giorgio Fumera, Fabio Roli, "A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 942-956, June 2005, doi:10.1109/TPAMI.2005.109
Usage of this product signifies your acceptance of the Terms of Use.