This Article 
 Bibliographic References 
 Add to: 
KPCA Plus LDA: A Complete Kernel Fisher Discriminant Framework for Feature Extraction and Recognition
February 2005 (vol. 27 no. 2)
pp. 230-244
This paper examines the theory of kernel Fisher discriminant analysis (KFD) in a Hilbert space and develops a two-phase KFD framework, i.e., kernel principal component analysis (KPCA) plus Fisher linear discriminant analysis (LDA). This framework provides novel insights into the nature of KFD. Based on this framework, the authors propose a complete kernel Fisher discriminant analysis (CKFD) algorithm. CKFD can be used to carry out discriminant analysis in "double discriminant subspaces.” The fact that, it can make full use of two kinds of discriminant information, regular and irregular, makes CKFD a more powerful discriminator. The proposed algorithm was tested and evaluated using the FERET face database and the CENPARMI handwritten numeral database. The experimental results show that CKFD outperforms other KFD algorithms.

[1] V. Vapnik, The Nature of Statistical Learning Theory. New York: Springer, 1995.
[2] K.-R. Müller, S. Mika, G. Rätsch, K. Tsuda, and B. Schölkopf, “An Introduction to Kernel-Based Learning Algorithms,” IEEE Trans. Neural Networks, vol. 12, no. 2, pp. 181-201, 2001.
[3] B. Schölkopf, A. Smola, and K.R. Müller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, vol. 10, no. 5, pp. 1299-1319, 1998.
[4] S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K.-R. Müller, “Fisher Discriminant Analysis with Kernels,” Proc. IEEE Int'l Workshop Neural Networks for Signal Processing IX, pp. 41-48, Aug. 1999.
[5] S. Mika, G. Rätsch, B. Schölkopf, A. Smola, J. Weston, and K.-R. Müller, “Invariant Feature Extraction and Classification in Kernel Spaces,” Advances in Neural Information Processing Systems 12, Cambridge, Mass.: MIT Press, 1999.
[6] G. Baudat and F. Anouar, “Generalized Discriminant Analysis Using a Kernel Approach,” Neural Computation, vol. 12, no. 10, pp. 2385-2404, 2000.
[7] V. Roth and V. Steinhage, “Nonlinear Discriminant Analysis Using Kernel Functions,” Advances in Neural Information Processing Systems, S.A. Solla, T.K. Leen, and K.-R. Mueller, eds., vol. 12, pp. 568-574, MIT Press, 2000.
[8] S. Mika, G. Ratsch, and K.-R. Müller, “A Mathematical Programming Approach to the Kernel Fisher Algorithm,” Advances in Neural Information Processing Systems 13, T.K. Leen, T.G. Dietterich, and V. Tresp, eds., pp. 591-597, MIT Press, 2001.
[9] S. Mika, A.J. Smola, and B. Schölkopf, “An Improved Training Algorithm for Kernel Fisher Discriminants,” Proc. Eighth Int'l Workshop Artificial Intelligence and Statistics, T. Jaakkola and T. Richardson, eds., pp. 98-104, 2001.
[10] S. Mika, G. Rätsch, J Weston, B. Schölkopf, A. Smola, and K.-R. Müller, “Constructing Descriptive and Discriminative Nonlinear Features: Rayleigh Coefficients in Kernel Feature Spaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 5, pp. 623-628, May 2003.
[11] M.H. Yang, “Kernel Eigenfaces vs. Kernel Fisherfaces: Face Recognition Using Kernel Methods,” Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 215-220, May 2002.
[12] J. Lu, K.N. Plataniotis, and A.N. Venetsanopoulos, “Face Recognition Using Kernel Direct Discriminant Analysis Algorithms,” IEEE Trans. Neural Networks, vol. 14, no. 1, pp. 117-126, 2003.
[13] J. Xu, X. Zhang, and Y. Li, “Kernel MSE Algorithm: A Unified Framework for KFD, LS-SVM, and KRR,” Proc. Int'l Joint Conf. Neural Networks, pp. 1486-1491, July 2001.
[14] S.A. Billings and K.L Lee, “Nonlinear Fisher Discriminant Analysis Using a Minimum Squared Error Cost Function and the Orthogonal Least Squares Algorithm,” Neural Networks, vol. 15, no. 2, pp. 263-270, 2002.
[15] T.V. Gestel, J.A.K. Suykens, G. Lanckriet, A. Lambrechts, B. De Moor, and J. Vanderwalle, “Bayesian Framework for Least Squares Support Vector Machine Classifiers, Gaussian Processes and Kernel Fisher Discriminant Analysis,” Neural Computation, vol. 15, no. 5, pp. 1115-1148, May 2002.
[16] G.C. Cawley and N.L.C. Talbot, “Efficient Leave-One-Out Cross-Validation of Kernel Fisher Discriminant Classifiers,” Pattern Recognition, vol. 36, no. 11, pp. 2585-2592, 2003.
[17] N.D. Lawrence and B. Schölkopf, “Estimating a Kernel Fisher Discriminant in the Presence of Label Noise,” Proc. 18th Int'l Conf. Machine Learning, pp. 306-313, 2001.
[18] A.N. Tikhonov and V.Y. Arsenin, Solution of Ill-Posed Problems. New York: Wiley, 1997.
[19] D.L. Swets and J. Weng, “Using Discriminant Eigenfeatures for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, Aug. 1996.
[20] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriengman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[21] K. Liu, Y.-Q. Cheng, J.-Y. Yang, and X. Liu, “An Efficient Algorithm for Foley-Sammon Optimal Set of Discriminant Vectors by Algebraic Method,” Int'l J. Pattern Recognition and Artificial Intelligence, vol. 6, no. 5, pp. 817-829, 1992.
[22] L.F. Chen, H.Y.M. Liao, J.C. Lin, M.D. Kao, and G.J. Yu, “A New LDA-Based Face Recognition System which Can Solve the Small Sample Size Problem,” Pattern Recognition, vol. 33, no. 10, pp. 1713-1726, 2000.
[23] H. Yu and J. Yang, “A Direct LDA Algorithm for High-Dimensional Data— With Application to Face Recognition,” Pattern Recognition, vol. 34, no. 10, pp. 2067-2070, 2001.
[24] J. Yang and J.Y. Yang, “Why Can LDA Be Performed in PCA Transformed Space?” Pattern Recognition, vol. 36, no. 2, pp. 563-566, 2003.
[25] J. Yang and J.Y. Yang, “Optimal FLD Algorithm for Facial Feature Extraction,” Proc. SPIE Intelligent Robots and Computer Vision XX: Algorithms, Techniques, and Active Vision, pp. 438-444, Oct. 2001.
[26] C.J. Liu and H. Wechsler, “A Shape- and Texture-Based Enhanced Fisher Classifier for Face Recognition,” IEEE Trans. Image Processing, vol. 10, no. 4, pp. 598-608, 2001.
[27] C.J. Liu and H. Wechsler, “Robust Coding Schemes for Indexing and Retrieval from Large Face Databases,” IEEE Trans. Image Processing, vol. 9, no. 1, pp. 132-137, 2000.
[28] W. Zhao, R. Chellappa, and J. Phillips, “Subspace Linear Discriminant Analysis for Face Recognition,” Technical Report CS-TR4009, Univ. of Maryland, 1999.
[29] W. Zhao, A. Krishnaswamy, R. Chellappa, D. Swets, and J. Weng, “Discriminant Analysis of Principal Components for Face Recognition,” Face Recognition: From Theory to Applications, H. Wechsler, P. J. Phillips, V. Bruce, F. F. Soulie and T. S. Huang, eds., pp. 73-85, Springer-Verlag, 1998.
[30] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.
[31] M. Kirby and L. Sirovich, “Application of the KL Procedure for the Characterization of Human Faces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 1, pp. 103-108, Jan. 1990.
[32] J. Yang, A.F. Frangi, and J.-Y. Yang, “A New Kernel Fisher Discriminant Algorithm with Application to Face Recognition,” Neurocomputing, vol. 56, pp. 415-421, 2004.
[33] G.H. Golub and C.F. Van Loan, Matrix Computations, third ed. Baltimore and London: The Johns Hopkins Univ. Press, 1996.
[34] P. Lancaster and M. Tismenetsky, The Theory of Matrices, second ed. Orlando, Fla.: Academic Press, 1985.
[35] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Boston: Academic Press, 1990.
[36] P.J. Phillips, H. Moon, S.A. Rizvi, and P.J. Rauss, “The FERET Evaluation Methodology for Face-Recognition Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1090-1104, Oct. 2000.
[37] P.J. Phillips, “The Facial Recognition Technology (FERET) Database,” feret_ master.html, 2004.
[38] J. Yang, D. Zhang, and J.-y. Yang, “A Generalized K-L Expansion Method which Can Deal with Small Sample Size and High-Dimensional Problems,” Pattern Analysis and Application, vol. 6, no. 1, pp. 47-54, 2003.
[39] W. Yambor, B. Draper, and R. Beveridge, “Analyzing PCA-Based Face Recognition Algorithms: Eigenvector Selection and Distance Measures,” Empirical Evaluation Methods in Computer Vision, H. Christensen and J. Phillips, eds., Singapore: World Scientific Press, 2002.
[40] J. Devore and R. Peck, Statistics: The Exploration and Analysis of Data, third ed. Brooks Cole, 1997.
[41] B.A. Draper, K. Baek, M.S. Bartlett, and J.R. Beveridge, “Recognizing Faces with PCA and ICA,” Computer Vision and Image Understanding, vol. 91, no. 1-2, pp. 115-137, 2003.
[42] Z. Lou, K. Liu, J.Y. Yang, and C.Y. Suen, “Rejection Criteria and Pairwise Discrimination of Handwritten Numerals Based on Structural Features,” Pattern Analysis and Applications, vol. 2, no. 3, pp. 228-238, 1992.
[43] Y. Hamamoto, S. Uchimura, M. Watanabe, T. Yasuda, and S. Tomita, “Recognition of Handwritten Numerals Using Gabor Features,” Proc. 13th Int'l Conf. Pattern Recognition, pp. 250-253, Aug. 1996.
[44] J. Yang, J.-y. Yang, D. Zhang, and J.F. Lu, “Feature Fusion: Parallel Strategy vs. Serial Strategy,” Pattern Recognition, vol. 36, no. 6, pp. 1369-1381, 2003.
[45] C.J.C. Burges and B. Schölkopf, “Improving the Accuracy and Speed of Support Vector Learning Machines,” Advances in Neural Information Processing Systems 9, M. Mozer, M. Jordan, and T. Petsche, eds., pp. 375-381, Cambridge, Mass.: MIT Press, 1997.
[46] B. Schölkopf and A. Smola, Learning with Kernels. Cambridge, Mass.: MIT Press, 2002.
[47] E. Kreyszig, Introductory Functional Analysis with Applications. John Wiley & Sons, 1978.
[48] W. Rudin, Functional Analysis. McGraw-Hill, 1973.
[49] V. Hutson and J.S. Pym, Applications of Functional Analysis and Operator Theory. London: Academic Press, 1980.
[50] J. Weidmann, Linear Operators in Hilbert Spaces. New York: Springer-Verlag, 1980.
[51] J. Yang, J.-y. Yang, and A.F. Frangi, “Combined Fisherfaces Framework,” Image and Vision Computing, vol. 21, no. 12, pp. 1037-1044, 2003.
[52] J. Yang, J.-y. Yang, and H. Ye, “Theory of Fisher Linear Discriminant Analysis and Its Application,” Acta Automatica Sinica, vol. 29, no. 4, pp. 481-494, 2003 (in Chinese).

Index Terms:
Kernel-based methods, subspace methods, principal component analysis (PCA), Fisher linear discriminant analysis (LDA or FLD), feature extraction, machine learning, face recognition, handwritten digit recognition.
Jian Yang, Alejandro F. Frangi, Jing-yu Yang, David Zhang, Zhong Jin, "KPCA Plus LDA: A Complete Kernel Fisher Discriminant Framework for Feature Extraction and Recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 2, pp. 230-244, Feb. 2005, doi:10.1109/TPAMI.2005.33
Usage of this product signifies your acceptance of the Terms of Use.