The Community for Technology Leaders
RSS Icon
Issue No.09 - September (2009 vol.31)
pp: 1630-1644
Ernest Valveny , Universitat Automous of Barcelona, Bellaterra
Oriol Ramos Terrades , Universitat Automous of Barcelona, Bellaterra
The combination of the output of classifiers has been one of the strategies used to improve classification rates in general purpose classification systems. Some of the most common approaches can be explained using the Bayes' formula. In this paper, we tackle the problem of the combination of classifiers using a non-Bayesian probabilistic framework. This approach permits us to derive two linear combination rules that minimize misclassification rates under some constraints on the distribution of classifiers. In order to show the validity of this approach we have compared it with other popular combination rules from a theoretical viewpoint using a synthetic data set, and experimentally using two standard databases: the MNIST handwritten digit database and the GREC symbol database. Results on the synthetic data set show the validity of the theoretical approach. Indeed, results on real data show that the proposed methods outperform other common combination schemes.
Classifier fusion, linear combination rules, random variable.
Ernest Valveny, Oriol Ramos Terrades, "Optimal Classifier Fusion in a Non-Bayesian Probabilistic Framework", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.31, no. 9, pp. 1630-1644, September 2009, doi:10.1109/TPAMI.2008.224
[1] M.L. Kherfi, D. Ziou, and A. Bernardi, “Content-Based Image Retrieval Using Positive and Negative Examples,” J. Visual Comm. and Image Representation, vol. 14, no. 4, pp. 428-457, Dec. 2003.
[2] A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, “Content-Based Image Retrieval at the End of the Early Years,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1349-1380, Dec. 2000.
[3] S. Boutemedjet, D. Ziou, and N. Bouguila, “Unsupervised Feature Selection for Accurate Recommendation of High-Dimensional Image Data,” Proc. Ann. Conf. Advances in Neural Information Processing Systems, 2007.
[4] H. Peng, F. Long, and C. Ding, “Feature Selection Based on Mutual Information: Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1226-1238, Aug. 2005.
[5] R.P. Lippmann, “An Introduction to Computing with Neural Nets,” SIGARCH Computer Architecture News, vol. 16, no. 1, pp. 7-25, Mar. 1988.
[6] K. Tumer and J. Ghosh, “Analysis of Decision Boundaries in Linearly Combined Neural Classifiers,” Pattern Recognition, vol. 29, no. 2, pp. 314-348, Feb. 1996.
[7] K. Tumer and J. Ghosh, “Error Correlation and Error Reduction in Ensemble Classifiers,” Connection Science, vol. 8, nos. 3/4, pp. 385-403, 1996.
[8] Y. Freund and R.E. Schapire, “Experiments with a New Boosting Algorithm,” Proc. 13th Int'l Conf. Machine Learning, pp. 148-156, 1996.
[9] R.E. Schapire and Y. Singer, “Improved Boosting Algorithms Using Confidence-Rated Predictions,” Machine Learning, vol. 37, no. 3, pp. 297-336, Mar. 1999.
[10] M. Skurichina and R.P.W. Duin, “Bagging, Boosting and the Random Subspace Method for Linear Classifiers,” Population Assoc. of Am., vol. 5, no. 2, pp. 121-135, 2002.
[11] A. Torralba, K.P. Murphy, and W.T. Freeman, “Sharing Visual Featuers for Multiclass and Multiview Object Detection,” Proc. Conf. Computer Vision and Pattern Recognition, 2004.
[12] P. Viola and M. Jones, “Rapid Object Detection Using Boosted Cascade of Simple Features,” Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 511-518, 2001.
[13] L. Breiman, “Bagging Predictors,” Technical Report 421, Dept. of Statistics, Univ. of California, Sept. 1994.
[14] C.J.C. Burges, “A Tutorial on Support Vector Machines for Pattern Recognition,” Data Mining and Knowledge Discovery, vol. 2, no. 2, pp. 1-43, 1998.
[15] P.H. Chen and C.-J. Lin Schölkopf, “A Tutorial on V-Support Vector Machines,” Applied Stochastics Models in Business and Industry, vol. 21, no. 2, pp. 111-136, 2005.
[16] T.G. Dietterich and G. Bakiri, “Solving Multiclass Learning Problems via Error-Correcting Output Codes,” J. Artificial Intelligence Research, vol. 2, pp. 263-286, 1995.
[17] S. Escalera, O. Pujol, and P. Radeva, “Forest Extension of Error Correcting Output Codes and Boosted Landmarks,” Proc. Int'l Conf. Pattern Recognition, vol. 4, pp. 104-107, 2006.
[18] O. Pujol, P. Radeva, and J. Vitria, “Discriminant Ecoc: A Heuristic Method for Application Dependent Design of Error Correcting Output Codes,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 6, pp. 1007-1012, June 2006.
[19] J. Lladós, E. Valveny, G. Sánchez, and E. Martí, “Symbol Recognition: Current Advances and Perspectives,” Proc. Graphics Recognition. Algorithms, and Applications, pp. 105-127, 2002.
[20] S. Loncaric, “A Survey of Shape Analysis Techniques,” Pattern Recognition, vol. 31, no. 8, pp. 983-1001, 1998.
[21] T. Pavlidis, “Survey: A Review of Algorithms for Shape Analysis,” Computer Graphics and Image Processing, vol. 7, no. 7, pp. 243-258, 1978.
[22] D. Zhang and G. Lu, “Review of Shape Representation and Description Techniques,” Pattern Recognition, vol. 37, no. 1, pp. 1-19, 2004.
[23] J. Kittler, “A Framework for Classifier Fusion: Is It Still Needed?” Proc. Joint IAPR Int'l Workshops Advances in Pattern Recognition, pp.45-56, 2000.
[24] R.E. Schapire, “The Strength of Weak Learnability,” Machine Learning, vol. 5, no. 2, pp. 197-227, 1990.
[25] Z. Stejic, Y. Takama, and K. Hirota, “Mathematical Aggregation Operators in Image Retrieval: Effect on Retrieval Performance and Role in Relevance Feedback,” Signal Processing, vol. 85, no. 2, pp.297-324, 2005.
[26] J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas, “On Combining Classifiers,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 3, pp. 226-239, Mar. 1998.
[27] L. Xu, A. Krzyzak, and C.Y. Suen, “Methods of Combining Multiple Classifiers and Their Applications to Handwriting Recognition,” IEEE Trans. Systems, Man and Cybernetics, vol. 22, no. 3, pp. 418-435, May/June 1992.
[28] E. Valveny and P. Dosch, “Symbol Recognition Contest: A Synthesis,“ Proc. Int'l Workshop Graphics Recognition: Recent Advances and Perspectives, pp. 368-386, 2004.
[29] W.-Y. Kim, Y.-S. Kim, and Y.-S. Kim, “A New Region-Based Shape Descriptor,” technical report, Hanyang Univ. and Konan Tech nology, Dec. 1999.
[30] T.K. Ho, “A Theory of Multiple Classifier Systems and Its Application to Visual Word Recognition,” PhD dissertation, State Univ. of New York at Buffalo, html, 1992.
[31] O. Melnik, Y. Vardi, and C.-H. Zhang, “Mixed Group Ranks: Preference and Confidence in Classifier Combination,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 8, pp.973-981, Aug. 2004.
[32] F.M. Alkoot and J. Kittler, “Experimental Evaluation of Expert Fusion Strategies,” Pattern Recognition Letters, vol. 20, nos. 11-13, pp. 1361-1369, Nov. 1999.
[33] L.I. Kuncheva, C.J. Whitaker, C.A. Shipp, and R.P.W. Duin, “Is Independence Good for Combining Classifiers?” Proc. 15th Int'l Conf. Pattern Recognition, vol. 2, pp. 168-171, 2000.
[34] J. Platt, “Probabilistic Outputs for Support Vector Machines and Comparison to Regularized Likelihood Methods,” Advances in Large Margin Classifiers, pp. 61-74, MIT Press, , 2000.
[35] J. Friedman, T. Hastie, and R. Tibshirani, “Additive Logistic Regression: A Statistical View of Boosting,” technical report, Dept. of Statistics, Standford Univ., 1998.
[36] L.I. Kuncheva, “A Theoretical Study on Six Classifier Fusion Strategies,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 2, pp. 281-286, Feb. 2002.
[37] D.M.J. Tax, M. van Breukelen, R.P.W. Duin, and J. Kittler, “Combining Multiple Classifiers by Averaging or by Multiplying,” Pattern Recognition, vol. 33, no. 9, pp. 1475-1485, 2000.
[38] M.V. Breukelen, R.P.W. Duin, D.M.J. Tax, and J.E.D. Hartog, “Handwritten Digit Recognition by Combined Classifiers,” Kybernetika, vol. 34, no. 4, pp. 381-386, 1998.
[39] A.K. Jain, R.P. Duin, and J. Mao, “Statistical Pattern Recognition: A Review,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[40] O.R. Terrades and E. Valveny, “Local Norm Features Based on Ridgelets Transform,” Proc. Eighth Int'l Conf. Document Analysis and Recognition, pp. 700-704, 2005.
[41] S. Tabbone, L. Wendling, and J.P. Salmon, “A New Shape Descriptor Defined on the Radon Transform,” Computer Vision and Image Understanding, vol. 102, no. 1, pp. 42-51, Apr. 2006.
27 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool