This Article 
 Bibliographic References 
 Add to: 
Feature Reduction via Generalized Uncorrelated Linear Discriminant Analysis
October 2006 (vol. 18 no. 10)
pp. 1312-1322
High-dimensional data appear in many applications of data mining, machine learning, and bioinformatics. Feature reduction is commonly applied as a preprocessing step to overcome the curse of dimensionality. Uncorrelated Linear Discriminant Analysis (ULDA) was recently proposed for feature reduction. The extracted features via ULDA were shown to be statistically uncorrelated, which is desirable for many applications. In this paper, an algorithm called ULDA/QR is proposed to simplify the previous implementation of ULDA. Then, the ULDA/GSVD algorithm is proposed, based on a novel optimization criterion, to address the singularity problem which occurs in undersampled problems, where the data dimension is larger than the sample size. The criterion used is the regularized version of the one in ULDA/QR. Surprisingly, our theoretical result shows that the solution to ULDA/GSVD is independent of the value of the regularization parameter. Experimental results on various types of data sets are reported to show the effectiveness of the proposed algorithm and to compare it with other commonly used feature reduction algorithms.

[1] P. Belhumeour, J. Hespanha, and D. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, 1997.
[2] M. Berry, S. Dumais, and G. O'Brie, “Using Linear Algebra for Intelligent Information Retrieval,” SIAM Rev., vol. 37, pp. 573-595, 1995.
[3] L. Chen, H. Liao, M. Ko, J. Lin, and G. Yu, “A New LDA-Based Face Recognition System Which Can Solve the Small Sample Size Problem,” Pattern Recognition, vol. 33, pp. 1713-1726, 2000.
[4] S. Deerwester, S. Dumais, G. Furnas, T. Landauer, and R. Harshman, “Indexing by Latent Semantic Analysis,” J. Soc. for Information Science, vol. 41, pp. 391-407, 1990.
[5] L. Duchene and S. Leclerq, “An Optimal Transformation for Discriminant and Principal Component Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 10, no. 6, pp. 978-983, 1988.
[6] R. Duda, P. Hart, and D. Stork, Pattern Classification. Wiley, 2000.
[7] S. Dudoit, J. Fridlyand, and T.P. Speed, “Comparison of Discrimination Methods for the Classification of Tumors Using Gene Expression Data,” J. Am. Statistical Assoc., vol. 97, no. 457, pp. 77-87, 2002.
[8] D. Foley and J. Sammon, “An Optimal Set of Discriminant Vectors,” IEEE Trans. Computers, vol. 24, no. 3, pp. 281-289, 1975.
[9] J. Friedman, “Regularized Discriminant Analysis,” J. Am. Statistical Assoc., vol. 84, no. 405, pp. 165-175, 1989.
[10] K. Fukunaga, Introduction to Statistical Pattern Classification. Academic Press, 1990.
[11] G.H. Golub and C.F. Van Loan, Matrix Computations, third ed. The Johns Hopkins Univ. Press, 1996.
[12] T. Hastie, A. Buja, and R. Tibshirani, “Penalized Discriminant Analysis,” Annals of Statistics, vol. 23, pp. 73-102, 1995.
[13] T. Hastie and R. Tibshirani, “Discriminant Analysis by Gaussian Mixtures,” J. Royal Statistical Soc. series B, vol. 58, pp. 158-176, 1996.
[14] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2001.
[15] P. Howland, M. Jeon, and H. Park, “Structure Preserving Dimension Reduction for Clustered Text Data Based on the Generalized Singular Value Decomposition,” SIAM J. Matrix Analysis and Applications, vol. 25, no. 1, pp. 165-179, 2003.
[16] P. Howland and H. Park, “Generalizing Discriminant Analysis Using the Generalized Singular Value Decomposition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 8, pp. 995-1006, Aug. 2004.
[17] Z. Jin, J.Y. Yang, Z.-S. Hu, and Z. Lou, “Face Recognition Based on the Uncorrelated Discriminant Transformation,” Pattern Recognition, vol. 34, pp. 1405-1416, 2001.
[18] Z. Jin, J.-Y. Yang, Z.-M. Tang, and Z.-S. Hu, “A Theorem on the Uncorrelated Optimal Discriminant Vectors,” Pattern Recognition, vol. 34, no. 10, pp. 2041-2047, 2001.
[19] I.T. Jolliffe, Principal Component Analysis. Springer-Verlag, 1986.
[20] W. Krzanowski, P. Jonathan, W. McCarthy, and M. Thomas, “Discriminant Analysis with Singular Covariance Matrices: Methods and Applications to Spectroscopic Data,” Applied Statistics, vol. 44, pp. 101-115, 1995.
[21] C. Paige and M. Saunders, “Towards a Generalized Singular Value Decomposition,” SIAM J. Numerical Analysis, vol. 18, pp. 398-405, 1981.
[22] H. Park, M. Jeon, and J. Rosen, “Lower Dimensional Representation of Text Data Based on Centroids and Least Squares,” BIT Numerical Math., vol. 43, no. 2, pp. 1-22, 2003.
[23] S. Ramaswamy et al., “Multiclass Cancer Diagnosis Using Tumor Gene Expression Signatures,” Proc. Nat'l Academy of Science, vol. 98, no. 26, pp. 15149-15154, 2001.
[24] B. Schökopf and A. Smola, Learning with Kernels: Support Vector Machines, Regularization, Optimization and Beyond. MIT Press, 2002.
[25] D.L. Swets and J. Weng, “Using Discriminant Eigenfeatures for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, Aug. 1996.
[26] M. Turk and A. Pentland, “Face Recognition Using Eigenfaces,” Proc. Computer Vision and Pattern Recognition Conf., pp. 586-591, 1991.
[27] V. Vapnik, Statistical Learning Theory. Wiley, 1998.
[28] J. Ye, “Characterization of a Family of Algorithms for Generalized Discriminant Analysis on Undersampled Problems,” J. Machine Learning Research, vol. 6, pp. 483-502, 2005.
[29] J. Ye, R. Janardan, C. Park, and H. Park, “An Optimization Criterion for Generalized Discriminant Analysis on Undersampled Problems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 8, pp. 982-994, Aug. 2004.
[30] C.H. Yeang et al., “Molecular Classification of Multiple Tumor Types,” Bioinformatics, vol. 17, no. 1, pp. 1-7, 2001.
[31] E.J. Yeoh et al., “Classification, Subtype Discovery, and Prediction of Outcome in Pediatric Lymphoblastic Leukemia by Gene Expression Profiling,” Cancer Cell, vol. 1, no. 2, pp. 133-143, 2002.

Index Terms:
Feature reduction, uncorrelated linear discriminant analysis, QR-decomposition, generalized singular value decomposition.
Jieping Ye, Ravi Janardan, Qi Li, Haesun Park, "Feature Reduction via Generalized Uncorrelated Linear Discriminant Analysis," IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 10, pp. 1312-1322, Oct. 2006, doi:10.1109/TKDE.2006.160
Usage of this product signifies your acceptance of the Terms of Use.