This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Feature Extraction and Uncorrelated Discriminant Analysis for High-Dimensional Data
May 2008 (vol. 20 no. 5)
pp. 601-614
High-dimensional data and small sample size problem occur in many modern pattern classification applications, such as face recognition and gene expression data analysis. To deal with such data, an important step is the dimensionality reduction. Principal component analysis(PCA) and between-group analysis(BGA) are two commonly used methods and various extensions exist. The principle of these two approaches comes from their best approximation. From a pattern recognition perspective we show that PCA based on total-scatter matrix preserves linear separability and BGA based on between-scatter matrix retains only the distances between class centroid. Moreover we propose a novel uncorrelated discriminant analysis (UDA) algorithm. It combines rank preserving dimensionality reduction and constraint discriminant analysis, and serves as a simple and complete solution for small sample size problem. We conduct a series of comparative study on face images and gene expression data sets to evaluate UDA in terms of classification accuracy and robustness.

[1] A.A. Alizadeh et al., “Distinct Types of Diffuse Large B-Cell Lymphoma Identified by Gene Expression Profiling,” Nature, vol. 403, pp. 503-511, 2000.
[2] U. Alon et al., “Broad Patterns of Gene Expression Revealed by Clustering Analysis of Tumor and Normal Colon Tissues Probed by Oligonucleotide Arrays,” Proc. Nat'l Academy of Sciences of the USA, vol. 96, pp. 6745-6750, 1999.
[3] E. Alpaydin, “Combined 5x2cv F Test for Comparing Supervised Classification Learning Algorithms,” Neural Computation, vol. 11, no. 8, pp. 1885-1892, 1999.
[4] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, pp. 711-720, 1997.
[5] L. Chen, H. Liao, M. Ko, J. Lin, and G. Yu, “A New LDA-Based Face Recognition System, Which Can Solve the Small Sample Size Problem,” Pattern Recognition, vol. 33, no. 10, pp. 1713-1726, 2000.
[6] W.S. Chen, P.C. Yuen, J. Huang, and D.Q. Dai, “Kernel Machine-Based One-Parameter Regularized Fisher Discriminant Method for Face Recognition,” IEEE Trans. Systems, Man, and Cybernetics— Part B: Cybernetics, vol. 35, no. 4, pp. 657-669, 2005.
[7] A.C. Culhane, G. Perriere, E.C. Considine, T.G. Cotter, and D.G. Higgins, “Between-Group Analysis of Microarray Data,” Bioinformatics, vol. 18, no. 12, pp. 1600-1608, 2002.
[8] D.Q. Dai and P.C. Yuen, “Regularized Discriminant Analysis and Its Application to Face Recognition,” Pattern Recognition, vol. 36, pp. 845-847, 2003.
[9] T.G. Dietterich, “Approximate Statistical Test for Comparing Supervised Classification Learning Algorithms,” Neural Computation, vol. 10, no. 7, pp. 1895-1923, 1998.
[10] S. Dudoit, J. Fridlyand, and T. Speed, “Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data,” J. Am. Statistical Assoc., vol. 97, pp. 77-87, 2002.
[11] J.H. Friedman, “Regularized Discriminant Analysis,” J. Am. Statistical Assoc., vol. 84, no. 405, pp. 165-175, 1989.
[12] K. Fukumizu, F.R. Bach, and M.I. Jordan, “Dimensionality Reduction for Supervised Learning with Reproducing Kernel Hilbert Spaces,” J. Machine Learning Research, vol. 5, pp. 73-99, 2004.
[13] K. Fukunaga, Introduction to Statistical Pattern Classification. Academic Press, 1990.
[14] T.R. Golub et al., “Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring,” Science, vol. 536, pp. 531-537, 1999.
[15] W.M. Hartmann, “Dimension Reduction versus Variable Selection,” Lecture Notes in Computer Science, vol. 3732, pp. 931-938, 2006.
[16] T. Hastie, R. Tibshirani, and J.H. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2001.
[17] P. Howland, J. Wang, and H. Park, “Solving the Small Sample Size Problem in Face Recognition Using Generalized Discriminant Analysis,” Pattern Recognition, vol. 39, pp. 277-287, 2006.
[18] R. Huang, Q. Liu, H. Lu, and S. Ma, “Solving the Small Sample Size Problem of LDA,” Proc. 16th Int'l Conf. Pattern Recognition (ICPR '02), vol. 3, pp. 29-32, 2002.
[19] Z. Jin, J. Yang, Z. Hu, and Z. Lou, “Face Recognition Based on the Uncorrelated Discrimination Transformation,” Pattern Recognition, vol. 34, no. 7, pp. 1405-1416, 2001.
[20] X.Y. Jing, D. Zhang, and Y.Y. Tang, “An Improved LDA Approach,” IEEE Trans. Systems, Man, and Cybernetics—Part B: Cybernetics, vol. 34, no. 5, pp. 1942-1951, 2004.
[21] I.T. Jolliffe, Principal Component Analysis. Springer, 1986.
[22] W.J. Krzanowski, P. Jonathan, W.V. McCarthy, and M.R. Thomas, “Discriminant Analysis with Singular Covariance Matrices: Methods and Applications to Spectroscopic Data,” Applied Statistics, vol. 44, pp. 101-115, 1995.
[23] M. Kyperountas, A. Tefas, and I. Pitas, “Weighted Piecewise LDA for Solving the Small Sample Size Problem in Face Verification,” IEEE Trans. Neural Networks, vol. 18, no. 2, pp. 506-519, 2007.
[24] J. Khan et al., “Classification and Diagnostic Prediction of Cancers Using Expression Profiling and Artificial Neural Networks,” Nature Medicine, vol. 7, pp. 673-679, 2001.
[25] J.W. Lee, J.B. Lee, M. Park, and S.H. Song, “An Extensive Comparison of Recent Classification Tools Applied to Microarray Data,” Computational Statistics & Data Analysis, vol. 48, pp. 869-885, 2005.
[26] H. Li, T. Jiang, and K. Zhang, “Efficient and Robust Feature Extraction by Maximum Margin Criterion,” IEEE Trans. Neural Networks, vol. 17, no. 1, pp. 157-165, 2006.
[27] H. Li, K. Zhang, and T. Jiang, “Robust and Accurate Cancer Classification with Gene Expression Profiling,” Proc. Fourth IEEE Computational Systems Bioinformatics Conf. (CSB '05), pp. 310-321, 2005.
[28] Y. Liang, C. Li, W. Gong, and Y. Pan, “Uncorrelated Linear Discriminant Analysis Based on Weighted Pairwise Fisher Criterion,” Pattern Recognition, vol. 40, no. 12, pp. 3606-3615, 2007.
[29] A.M. Martinez and M. Zhu, “Where Are Linear Feature Extraction Methods Applicable,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 12, pp. 1934-1944, Dec. 2005.
[30] H. Park, M. Jeon, and J. Rosen, “Lower Dimensional Representation of Text Data Based on Centroids and Least Squares,” BIT Numerical Math., vol. 43, pp. 427-448, 2003.
[31] P.J. Phillips, H. Moo, S.A. Rizvi, and P.J. Rauss, “The Feret Evaluation Methodology for Face Recognition Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp.1090-1104, Oct. 2000.
[32] S. Pomeroy et al., “Prediction of Central Nervous System Embryonal Tumor Outcome Based on Gene Expression,” Nature, vol. 415, pp. 436-442, 2002.
[33] A.K. Qin, P.N. Suganthan, and M. Loog, “Uncorrelated Heteroscedastic LDA Based on the Weighted Pairwise Chernoff Criterion,” Pattern Recognition, vol. 38, no. 4, pp. 613-616, 2005.
[34] A.K. Qin, P.N. Suganthan, and M. Loog, “Generalized Null Space Uncorrelated Fisher Discriminant Analysis for Linear Dimensionality Reduction,” Pattern Recognition, vol. 39, no. 9, pp. 1805-1808, 2006.
[35] S.J. Raudys and A.K. Jain, “Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 252-264, 1991.
[36] J.C. Roden, B.W. King, D. Trout, A. Mortazavi, B.J. Wold, and C.E. Hart, “Mining Gene Expression Data by Interpreting Principal Components,” BMC Bioinformatics, vol. 7, no. 194, Apr. 2006.
[37] L. Shen and E.C. Tan, “Dimension Reduction-Based Penalized Logistic Regression for Cancer Classification Using Microarray Data,” IEEE/ACM Trans. Computational Biology and Bioinformatics, vol. 2, no. 2, pp. 166-175, Apr.-June 2005.
[38] T. Sim, S. Baker, and M. Bsat, “The CMU Pose, Illumination, and Expression (PIE) Database of Human Faces,” Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition (FG '02), pp. 46-51, 2002.
[39] R. Tibshirani, T. Hastie, B. Narasimhan, and G. Chu, “Diagnosis of Multiple Cancer Types by Shrunken Centroids of Gene Expression,” Proc. Nat'l Academy of Sciences of the USA, vol. 99, pp. 6567-6572, 2002.
[40] V.N. Vapnik, Statistical Learning Theory. John Wiley & Sons, 1998.
[41] J. Yan, B.Y. Zhang, N. Liu, S.C. Yan, Q.S. Cheng, W.G. Fan, Q. Yang, W.S. Xi, and Z. Chen, “Effective and Efficient Dimensionality Reduction for Large-Scale and Streaming Data Preprocessing,” IEEE Trans. Knowledge and Data Eng., vol. 18, no. 3, pp. 320-333, Mar. 2006.
[42] J. Yang, A.F. Frangi, J.Y. Yang, D. Zhang, and Z. Jin, “KPCA plus LDA: A Complete Kernel Fisher Discriminant Framework for Feature Extraction and Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 2, pp. 230-244, Feb. 2005.
[43] J. Yang, J. Yang, and D. Zhang, “What's Wrong with Fisher Criterion,” Pattern Recognition, vol. 35, no. 11, pp. 2665-2668, 2002.
[44] W.H. Yang, D.Q. Dai, and H. Yan, “Generalized Discriminant Analysis for Tumor Classification with Gene Expression Data,” Proc. Fifth Int'l Conf. Machine Learning and Cybernetics (ICMLC '06), pp. 4322-4327, Aug. 2006.
[45] J.P. Ye, R. Janardan, Q. Li, and H. Park, “Feature Reduction via Generalized Uncorrelated Linear Discriminant Analysis,” IEEE Trans. Knowledge and Data Eng., vol. 18, no. 10, pp. 1312-1322, Oct. 2006.
[46] J.P. Ye, T. Li, T. Xiong, and R. Janardan, “Using Uncorrelated Discriminant Analysis for Tissue Classification with Gene Expression Data,” IEEE/ACM Trans. Computational Biology and Bioinformatics, vol. 1, no. 4, pp. 181-190, Oct.-Dec. 2004.
[47] J.P. Ye and T. Wang, “Regularized Discriminant Analysis for High Dimensional, Low Sample Size Data,” Proc. 12th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '06), pp. 454-463, Aug. 2006.
[48] E.J. Yeoh et al., “Classification, Subtype Discovery, and Prediction of Outcome in Pediatric Lymphoblastic Leukemia by Gene Expression Profiling,” Cancer Cell, vol. 1, pp. 133-143, 2002.
[49] K.Y. Yeung and W.L. Ruzzo, “Principal Component Analysis for Clustering Gene Expression Data,” Bioinformatics, vol. 17, no. 9, pp. 763-774, Sept. 2001.
[50] H. Yu and J. Yang, “A Direct LDA Algorithm for High-Dimensional Data with Application to Face Recognition,” Pattern Recognition, vol. 34, no. 10, pp. 2067-2070, 2001.
[51] W.S. Zheng, J.H. Lai, and P.C. Yuen, “Ga-Fisher: A New LDA-Based Face Recognition Algorithm with Selection of Principal Components,” IEEE Trans. Systems, Man, and Cybernetics—Part B: Cybernetics, vol. 35, no. 5, pp. 1065-1078, 2005.
[52] X.S. Zhuang and D.Q. Dai, “Inverse Fisher Discriminate Criteria for Small Sample Size Problem and Its Application to Face Recognition,” Pattern Recognition, vol. 38, no. 11, pp. 2192-2194, 2005.

Index Terms:
Feature extraction or construction, Pattern Recognition
Citation:
Wen-Hui Yang, Dao-Qing Dai, Hong Yan, "Feature Extraction and Uncorrelated Discriminant Analysis for High-Dimensional Data," IEEE Transactions on Knowledge and Data Engineering, vol. 20, no. 5, pp. 601-614, May 2008, doi:10.1109/TKDE.2007.190720
Usage of this product signifies your acceptance of the Terms of Use.