This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Where Are Linear Feature Extraction Methods Applicable?
December 2005 (vol. 27 no. 12)
pp. 1934-1944
A fundamental problem in computer vision and pattern recognition is to determine where and, most importantly, why a given technique is applicable. This is not only necessary because it helps us decide which techniques to apply at each given time. Knowing why current algorithms cannot be applied facilitates the design of new algorithms robust to such problems. In this paper, we report on a theoretical study that demonstrates where and why generalized eigen-based linear equations do not work. In particular, we show that when the smallest angle between the i{\rm{th}} eigenvector given by the metric to be maximized and the first i eigenvectors given by the metric to be minimized is close to zero, our results are not guaranteed to be correct. Several properties of such models are also presented. For illustration, we concentrate on the classical applications of classification and feature extraction. We also show how we can use our findings to design more robust algorithms. We conclude with a discussion on the broader impacts of our results.

[1] M.S. Bartlett, “Face Image Analysis by Unsupervised Learning,” Kluwer Int'l Series on Eng. and Computer Science, vol. 612, 2001.
[2] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[3] R. Beveridge, K. She, B. Draper, and G. Givens, “A Nonparametric Statistical Comparison of Principal Component and Linear Discriminant Subspaces for Face Recognition,” Proc. Computer Vision and Pattern Recognition, pp. I:535-542, 2001.
[4] O. Bousquet and A. Elisseeff, “Stability and Generalization,” J. Machine Learning Research, vol. 2, pp. 499-526, 2002.
[5] R.D. Cook, Regression Graphics. Wiley, 1998.
[6] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, second ed. Wiley, 2001.
[7] K. Etemad and R. Chellapa, “Discriminant Analysis for Recognition of Human Face Images,” J. Optics of Am. A, vol. 14, no. 8, pp. 1724-1733, 1997.
[8] M.A.T. Figueiredo and A. Jain, “Unsupervised Learning of Finite Mixture Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 3, pp. 381-396, Mar. 2002.
[9] K. Fukunaga and J.M. Mantock, “Nonparametric Discriminant Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 5, pp. 671-678, 1983.
[10] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1990.
[11] G.H. Golub and C.F. van Loan, Matrix Computations, third ed. John Hopkins Univ. Press, 1996.
[12] H. Gupta, A.K. Agrawal, T. Pruthi, C. Shekhar, and R. Chellappa, An Experimental Evaluation of Linear and Kernel-Based Methods for Face Recognition. John Hopkins Univ. Press, 1996.
[13] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer-Verlag, 2001.
[14] X. He, S. Yan, Y. Hu, P. Niyogi, and H.J. Zhang, “Face Recognition Using Laplacian Faces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 3, pp. 328-340, Mar. 2005.
[15] A.K. Jain and B. Chandrasekaran, “Dimensionality and Sample Size Considerations in Pattern Recognition Practice,” Handbook of Statistics, P.R. Krishnaiah and L.N. Kanal, eds., vol. 2, pp. 835-855, 1982.
[16] A.K. Jain, R.P.W. Duin, and J. Mao, “Statistical Pattern Recognition: A Review,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[17] I.T. Jolliffe, Principal Component Analysis, second ed. Springer-Verlag, 2002.
[18] T.K. Kim and J. Kittler, “Locally Linear Discriminant Analysis for Multimodally Distyributed Classes for Face Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 3, pp. 318-327, Mar. 2005.
[19] B. Leibe and B. Schiele, “Analyzing Appearance and Contour Based Methods for Object Categorization,” Proc. IEEE Computer Vision and Pattern Recognition, June 2003.
[20] J. Li, “Sliced Inverse Regression for Dimensionality Reduction,” J. Am. Statistical Soc., vol. 86, no. 414, pp. 316-327, 1991.
[21] M. Loog and R.P.W. Duin, “Linear Dimensionality Reduction via a Heteroscedastic Extension of LDA: The Chernoff Criterion,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 732-739, June 2004.
[22] K.V. Mardia, J.T. Kent, and J.M. Bibby, Multivariate Analysis. Academic Press, 1979.
[23] A.M. Martinez and A.C. Kak, “PCA versus LDA,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 228-233, Feb. 2001.
[24] G.J. McLachlan, Discriminant Analysis and Statistical Pattern Recognition. Wiley-Interscience, 1992.
[25] K.E. Muller, “Understanding Canonical Correlation through the General Liner Model and Principal Components,” Am. Statistician, vol. 36, pp. 342-354, 1982.
[26] T. Poggio, R. Rifkin, S. Mukherjee, and P. Niyogi, “General Conditions for Predictivity in Learning Theory,” Nature, vol. 428, pp. 419-422, 2004.
[27] D.F. Ransohoff, “Opinion— Rules of Evidence for Cancer Molecular-Marker Discovery and Validation,” Nature Rev. Cancer, vol. 4, pp. 309-314, 2004.
[28] C.R. Rao, Linear Statistical Inference and Its Applications, second ed. Wiley Interscience, 2002.
[29] I. Robledo and S. Sarkar, “Statistical Motion Model Based on the Change of Feature Relationships: Human Gait-Based Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1323-1328, Oct. 2003.
[30] D.L. Swets and J.J. Weng, “Using Discriminant Eigenfeatures for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, 1996.
[31] W.C. Thacker, “Metric-Based Principal Components: Data Uncertanties,” Tellus, vol. 48, no. A, pp. 584-592, 1996.
[32] V.N. Vapnik, The Nature of Statistical Learning Theory, second ed. Springer-Verlag, 2000.
[33] N. Vaswani and R. Chellappa, “Classification Probability Analysis of Principal Component Null Space Analysis,” Proc. Int'l Conf. Pattern Recognition, 2004.
[34] M.-S. Yang, “Kernel Eigenfaces vs. Kernel Fisherfaces: Face Recognition Using Kernel Methods,” Proc. Fifth Int'l Conf. Automatic Face and Gesture Recognition, pp. 215-220, 2002.
[35] H. Yu and J. Yang, “A Direct LDA Algorithm for High-Dimensional Data— with Applications to Face Recognition,” Pattern Recognition, vol. 34, pp. 2067-2070, 2001.
[36] M. Zhu and A.M. Martínez, “Optimal Subclass Discovery for Discriminant Analysis,” Proc. IEEE Workshop Learning in Computer Vision and Pattern Recognition, 2004.

Index Terms:
Index Terms- Feature extraction, generalized eigenvalue decomposition, performance evaluation, classifiers, pattern recognition.
Citation:
Aleix M. Mart?nez, Manli Zhu, "Where Are Linear Feature Extraction Methods Applicable?," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 12, pp. 1934-1944, Dec. 2005, doi:10.1109/TPAMI.2005.250
Usage of this product signifies your acceptance of the Terms of Use.