The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - October (2008 vol.20)
pp: 1311-1321
Shuiwang Ji , Arizona State University, Tempe
Jieping Ye , Arizona State University, Tempe
ABSTRACT
Linear and kernel discriminant analysis are popular approaches for supervised dimensionality reduction. Uncorrelated and regularized discriminant analysis have been proposed to overcome the singularity problem encountered by classical discriminant analysis. In this paper, we study the properties of kernel uncorrelated and regularized discriminant analysis, called KUDA and KRDA, respectively. In particular, we show that under a mild condition, both linear and kernel uncorrelated discriminant analysis project samples in the same class to a common vector in the dimensionality-reduced space. This implies that uncorrelated discriminant analysis may suffer from the overfitting problem if there are a large number of samples in each class. We show that as the regularization parameter in KRDA tends to zero, KRDA approaches KUDA. This shows that KUDA is a special case of KRDA, and that regularization can be applied to overcome the overfitting problem in uncorrelated discriminant analysis. As the performance of KRDA depends on the value of the regularization parameter, we show that the matrix computations involved in KRDA can be simplified, so that a large number of candidate values can be crossvalidated efficiently. Finally, we conduct experiments to evaluate the proposed theories and algorithms.
INDEX TERMS
Feature extraction or construction, Parameter learning, Singular value decomposition, Eigenvalues and eigenvectors
CITATION
Shuiwang Ji, Jieping Ye, "Kernel Uncorrelated and Regularized Discriminant Analysis: A Theoretical and Computational Study", IEEE Transactions on Knowledge & Data Engineering, vol.20, no. 10, pp. 1311-1321, October 2008, doi:10.1109/TKDE.2008.57
REFERENCES
[1] G. Baudat and F. Anouar, “Generalized Discriminant Analysis Using a Kernel Approach,” Neural Computation, vol. 12, no. 10, pp.2385-2404, 2000.
[2] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, “Eigenfaces versus Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[3] R.E. Bellman, Adaptive Control Processes: A Guided Tour. Princeton Univ. Press, 1961.
[4] L.F. Chen, H.Y.M. Liao, J.C. Lin, M.D. Kao, and G.J. Yu, “A New LDA-Based Face Recognition System Which Can Solve the Small Sample Size Problem,” Pattern Recognition, vol. 33, no. 10, 2000.
[5] S. Dudoit, J. Fridlyand, and T.P. Speed, “Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data,” J. Am. Statistical Assoc., vol. 97, no. 457, pp. 77-87, 2002.
[6] T. Evgeniou, M. Pontil, and T. Poggio, “Regularization Networks and Support Vector Machines,” Advances in Computational Math., vol. 13, no. 1, pp. 1-50, 2000.
[7] J.H. Friedman, “Regularized Discriminant Analysis,” J. Am. Statistical Assoc., vol. 84, no. 405, pp. 165-175, 1989.
[8] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press Professional, 1990.
[9] J.L. Gardy, C. Spencer, K. Wang, M. Ester, G.E. Tusnady, I. Simon, S. Hua, K. deFays, C. Lambert, K. Nakai, and F.S.L. Brinkman, “PSORT-B: Improving Protein Subcellular Localization Prediction for Gram-Negative Bacteria,” Nucleic Acids Research, vol. 31, no. 13, pp. 3613-3617, 2003.
[10] G.H. Golub and C.F. Van Loan, Matrix Computations, third ed. Johns Hopkins Univ. Press, 1996.
[11] Y. Guo, T. Hastie, and R. Tibshirani, “Regularized Linear Discriminant Analysis and Its Application in Microarrays,” Biostatistics, vol. 8, no. 1, pp. 86-100, 2007.
[12] T. Hastie, R. Tibshirani, and J.H. Friedman, The Elements of Statistical Learning : Data Mining, Inference, and Prediction. Springer, 2001.
[13] P. Howland, M. Jeon, and H. Park, “Structure Preserving Dimension Reduction for Clustered Text Data Based on the Generalized Singular Value Decomposition,” SIAM J. Matrix Analysis and Applications, vol. 25, no. 1, pp. 165-179, 2003.
[14] C. Hsu, C. Chang, and C. Lin, “A Practical Guide to Support Vector Classification,” technical report, Dept. of Computer Science, Nat'l Taiwan Univ., 2003.
[15] J.J. Hull, “A Database for Handwritten Text Recognition Research,” IEEE Trans. Pattern Analysis Machine Intelligence, vol. 16, no. 5, pp. 550-554, May 1994.
[16] Z. Jin, J. Yang, Z. Hu, and Z. Lou, “Face Recognition Based on the Uncorrelated Discriminant Transformation,” Pattern Recognition, vol. 34, pp. 1405-1416, 2001.
[17] I.T. Jolliffe, Principal Component Analysis. Springer, 1986.
[18] A. Kornai and J.M. Richards, “Linear Discriminant Text Classification in High Dimension,” Hybrid Information Systems, A.Abraham and M. Koeppen, eds., pp. 527-538, Physica Verlag, 2002.
[19] J. Lu, K.N. Plataniotis, and A.N. Venetsanopoulos, “Face Recognition Using Kernel Direct Discriminant Analysis Algorithms,” IEEE Trans. Neural Networks, vol. 14, no. 1, pp. 117-126, 2003.
[20] A.M. Martinez and A.C. Kak, “PCA versus LDA,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 228-233, Feb. 2001.
[21] A.M. Martinez and M. Zhu, “Where Are Linear Feature Extraction Methods Applicable?” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 12, pp. 1934-1944, Dec. 2005.
[22] S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K.-R. Müller, “Fisher Discriminant Analysis with Kernels,” Neural Networks for Signal Processing IX, Y.H. Hu, J. Larsen, E. Wilson, and S. Douglas, eds., pp. 41-48, IEEE, 1999.
[23] M. Neamtu, H. Cevikalp, M. Wilkes, and A. Barkana, “Discriminative Common Vectors for Face Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 1, pp. 4-13, Jan. 2005.
[24] D.J. Newman, S. Hettich, C.L. Blake, and C.J. Merz, “UCI Repository of Machine Learning Databases,” http://archive. ics.uci.eduml/, 1998.
[25] S.L. Pomeroy, P. Tamayo, M. Gaasenbeek, L.M. Sturla, M. Angelo, M.E. McLaughlin, J.Y.H. Kim, L.C. Goumnerova, P.M. Black, J.C. Allen, D. Zagzag, J.M. Olson, T. Curran, C. Wetmore, J.A. Biegel, T. Poggio, S. Mukherjee, R. Rifkin, A. Califano, G. Stolovitzky, D.N. Louis, J.P. Mesirov, E.S. Lander, and T.R. Golub, “Prediction of Central Nervous System Embryonal Tumour Outcome Based on Gene Expression,” Nature, vol. 415, no. 6870, pp. 436-442, 2002.
[26] S. Ramaswamy, P. Tamayo, R. Rifkin, S. Mukherjee, C.-H. Yeang, M. Angelo, C. Ladd, M. Reich, E. Latulippe, J.P. Mesirov, T. Poggio, W. Gerald, M. Loda, E.S. Lander, and T.R. Golub, “Multiclass Cancer Diagnosis Using Tumor Gene Expression Signatures,” Proc. Nat'l Academy of Sciences (PNAS '01), vol. 98, no. 26, pp. 15149-15154, 2001.
[27] B. Schölkopf, A.J. Smola, and K-R. Müller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, vol. 10, no. 5, pp. 1299-1319, 1998.
[28] S. Schölkopf and A. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond. MIT Press, 2002.
[29] A.I. Su, J.B. Welsh, L.M. Sapinoso, S.G. Kern, P. Dimitrov, H. Lapp, P.G. Schultz, S.M. Powell, C.A. Moskaluk, H.F. Frierson Jr., and G.M. Hampton, “Molecular Classification of Human Carcinomas by Use of Gene Expression Signatures,” Cancer Research, vol. 61, no. 20, pp. 7388-7393, 2001.
[30] D.L. Swets and J. Weng, “Using Discriminant Eigenfeatures for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, Aug. 1996.
[31] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J.Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.
[32] T. Xiong, J. Ye, and V. Cherkassky, “Kernel Uncorrelated and Orthogonal Discriminant Analysis: A Unified Approach,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR '06), pp. 125-131, 2006.
[33] J. Yang, A.F. Frangi, J. Yang, D. Zhang, and Z. Jin, “KPCA Plus LDA: A Complete Kernel Fisher Discriminant Framework for Feature Extraction and Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 2, pp. 230-244, Feb. 2005.
[34] J. Ye, “Characterization of a Family of Algorithms for Generalized Discriminant Analysis on Undersampled Problems,” J. Machine Learning Research, vol. 6, pp. 483-502, 2005.
[35] J. Ye, J. Chen, Q. Li, and S. Kumar, “Classification of Drosophila Embryonic Developmental Stage Range Based on Gene Expression Pattern Images,” Proc. Computational Systems Bioinformatics Conf. (CSB '06), pp. 293-298, 2006.
[36] J. Ye, R. Janardan, Q. Li, and H. Park, “Feature Extraction via Generalized Uncorrelated Linear Discriminant Analysis,” Proc. 21st Int'l Conf. Machine Learning (ICML '04), p. 113, 2004.
[37] J. Ye, T. Li, T. Xiong, and R. Janardan, “Using Uncorrelated Discriminant Analysis for Tissue Classification with Gene Expression Data,” IEEE/ACM Trans. Computational Biology and Bioinformatics, vol. 1, no. 4, pp. 181-190, Oct.-Dec. 2004.
[38] J. Ye and T. Xiong, “Computational and Theoretical Analysis of Null Space and Orthogonal Linear Discriminant Analysis,” J.Machine Learning Research, vol. 7, pp. 1183-1204, July 2006.
[39] H. Yu and J. Yang, “A Direct LDA Algorithm for High-Dimensional Data with Applications to Face Recognition,” Pattern Recognition, vol. 34, pp. 2067-2070, 2001.
[40] W. Zhao, R. Chellappa, and P. Phillips, “Subspace Linear Discriminant Analysis for Face Recognition,” Technical Report CAR-TR-914, Center for Automation Research, Univ. of Maryland, College Park, 1999.
46 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool