This Article 
 Bibliographic References 
 Add to: 
Linear Dimensionality Reduction via a Heteroscedastic Extension of LDA: The Chernoff Criterion
June 2004 (vol. 26 no. 6)
pp. 732-739

Abstract—We propose an eigenvector-based heteroscedastic linear dimension reduction (LDR) technique for multiclass data. The technique is based on a heteroscedastic two-class technique which utilizes the so-called Chernoff criterion, and successfully extends the well-known linear discriminant analysis (LDA). The latter, which is based on the Fisher criterion, is incapable of dealing with heteroscedastic data in a proper way. For the two-class case, the between-class scatter is generalized so to capture differences in (co)variances. It is shown that the classical notion of between-class scatter can be associated with Euclidean distances between class means. From this viewpoint, the between-class scatter is generalized by employing the Chernoff distance measure, leading to our proposed heteroscedastic measure. Finally, using the results from the two-class case, a multiclass extension of the Chernoff criterion is proposed. This criterion combines separation information present in the class mean as well as the class covariance matrices. Extensive experiments and a comparison with similar dimension reduction techniques are presented.

[1] S. Aeberhard, O. de Vel, and D. Coomans, Comparative Analysis of Statistical Pattern Recognition Methods in High Dimensional Settings Pattern Recognition, vol. 27, pp. 1065-1077, 1994.
[2] H. Brunzell and J. Eriksson, Feature Reduction for Classification of Multidimensional Data Pattern Recognition, vol. 33 pp. 1741-1748, 2000.
[3] L.J. Buturovic, Toward Bayes-Optimal Linear Dimension Reduction IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, pp. 420-424, 1994.
[4] C.H. Chen, On Information and Distance Measures, Error Bounds, and Feature Selection The Information Scientist, vol. 10, pp. 159-173, 1979.
[5] J.K. Chung, P.L. Kannappan, C.T. Ng, and P.K. Sahoo, Measures of Distance between Probability distributions J. Math. Analysis and Applications, vol. 138, pp. 280-292, 1989.
[6] T.M. Cover and J.A. Thomas, Elements of Information Theory. New York: Wiley Interscience 1991.
[7] H.P. Decell and S.M. Mayekar, Feature Combinations and the Divergence Criterion Computers and Math. with Applications, vol. 3, pp. 71-76, 1977.
[8] P.A. Devijver and J. Kittler, Pattern Recognition: A Statistical Approach. London: Prentice-Hall, 1982.
[9] R.A. Fisher, The Use of Multiple Measurements in Taxonomic Problems Annals of Eugenics, vol. 7, pp. 179-188, 1936.
[10] K. Fukunaga, Introduction to Statistical Pattern Recognition. New York: Academic Press, 1990.
[11] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York: Springer-Verlag, 2001.
[12] A.K. Jain, R.P.W. Duin, and J. Mao, Statistical Pattern Recognition: A Review IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[13] N. Kumar and A.G. Andreou, Generalization of Linear Discriminant Analysis in a Maximum Likelihood Framework Proc. Joint Meeting of the Am. Statistical Assoc., 1996.
[14] X. Liu, A. Srivastava, and K. Gallivan, Optimal Linear Representations of Images for Object Recognition Proc. 2003 Conf. Computer Vision and Pattern Recognition, pp. 229-234, June 2003.
[15] M. Loog, Approximate Pairwise Accuracy Criteria for Multiclass Linear Dimension Reduction: Generalisations of the Fisher Criterion, Number 44 in WBBM Report Series. Delft, The Netherlands: Delft Univ. Press, 1999.
[16] M. Loog and R.P.W. Duin, Non-Iterative Heteroscedastic Linear Dimension Reduction for Two-Class Data. From Fisher to Chernoff Proc. Fourth Int'l Workshop S+SSPR 2002, pp. 508-517, 2002.
[17] M. Loog, R.P.W. Duin, and R. Haeb-Umbach, Multiclass Linear Dimension Reduction by Weighted Pairwise Fisher Criteria IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, pp. 762-766, 2001.
[18] G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recognition. New York: John Wiley&Sons, 1992.
[19] P.M. Murphy and D.W. Aha UCI Repository of Machine Learning Databases,, 2004.
[20] T. Okada, and S. Tomita, An Extended Fisher Criterion for Feature Extraction Malina's Method and Its Problems Electronics and Comm. Japan, vol. 67, pp. 10-17, 1984.
[21] C.R. Rao, The Utilization of Multiple Measurements in Problems of Biological Classification J. Royal Statistical Soc., Series B, vol. 10, pp. 159-203, 1948.
[22] J.A. Rice, Mathematical Statistics and Data Analysis, second ed. Belmont: Duxbury Press, 1995.
[23] M. Röhl and C. Weihs, Optimal vs. Classical Linear Dimension Reduction Proc. 22nd Ann. GfKl Conf., pp. 252-259, 1998.
[24] J.D. Tubbs, W.A. Coberly, and D.M. Young, Linear Dimension Reduction and Bayes Classification Pattern Recognition, vol. 15, pp. 167-172, 1982.

Index Terms:
Linear dimension reduction, linear discriminant analysis, Fisher criterion, Chernoff distance, Chernoff criterion.
Marco Loog, Robert P.W. Duin, "Linear Dimensionality Reduction via a Heteroscedastic Extension of LDA: The Chernoff Criterion," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 732-739, June 2004, doi:10.1109/TPAMI.2004.13
Usage of this product signifies your acceptance of the Terms of Use.