This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Adaptive Nonlinear Discriminant Analysis by Regularized Minimum Squared Errors
May 2006 (vol. 18 no. 5)
pp. 603-612
ASCII Text
x 
 
Hyunsoo Kim, Barry L. Drake, Haesun Park, "Adaptive Nonlinear Discriminant Analysis by Regularized Minimum Squared Errors," IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 5, pp. 603-612, May, 2006.
 
 
BibTex
x
 
@article{ 10.1109/TKDE.2006.72,
author = {Hyunsoo Kim and Barry L. Drake and Haesun Park},
title = {Adaptive Nonlinear Discriminant Analysis by Regularized Minimum Squared Errors},
journal ={IEEE Transactions on Knowledge and Data Engineering},
volume = {18},
number = {5},
issn = {1041-4347},
year = {2006},
pages = {603-612},
doi = {http://doi.ieeecomputersociety.org/10.1109/TKDE.2006.72},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Knowledge and Data Engineering
TI - Adaptive Nonlinear Discriminant Analysis by Regularized Minimum Squared Errors
IS - 5
SN - 1041-4347
SP603
EP612
EPD - 603-612
A1 - Hyunsoo Kim,
A1 - Barry L. Drake,
A1 - Haesun Park,
PY - 2006
KW - QR decomposition updating and downdating
KW - adaptive classifier
KW - leave-one-out cross validation
KW - linear discriminant analysis
KW - kernel methods
KW - regularization.
VL - 18
JA - IEEE Transactions on Knowledge and Data Engineering
ER -
 
Kernelized nonlinear extensions of Fisher's discriminant analysis, discriminant analysis based on generalized singular value decomposition (LDA/GSVD), and discriminant analysis based on the minimum squared error formulation (MSE) have recently been widely utilized for handling undersampled high-dimensional problems and nonlinearly separable data sets. As the data sets are modified from incorporating new data points and deleting obsolete data points, there is a need to develop efficient updating and downdating algorithms for these methods to avoid expensive recomputation of the solution from scratch. In this paper, an efficient algorithm for adaptive linear and nonlinear kernel discriminant analysis based on regularized MSE, called adaptive KDA/RMSE, is proposed. In adaptive KDA/RMSE, updating and downdating of the computationally expensive eigenvalue decomposition (EVD) or singular value decomposition (SVD) is approximated by updating and downdating of the QR decomposition achieving an order of magnitude speed up. This fast algorithm for adaptive kernelized discriminant analysis is designed by utilizing regularization techniques and the relationship between linear and nonlinear discriminant analysis and the MSE. In addition, an efficient algorithm to compute leave-one-out cross validation is also introduced by utilizing downdating of KDA/RMSE.

[1] S.A. Billings and K.L. Lee, “Nonlinear Fisher Discriminant Analysis Using a Minimum Squared Error Cost Function and the Orthogonal Least Squares Algorithm,” Neural Networks, vol. 15, pp. 263-270, 2002.
[2] A. Björck, “Numerical Methods for Least Square Problems,” Proc. SIAM Conf., 1996.
[3] A. Björck, H. Park, and L. Eldén, “Accurate Downdating of Least Squares Solutions,” SIAM J. Matrix Analysis Applications, vol. 15, pp. 549-568, 1994.
[4] A.W. Bojanczyk, R.P. Brent, P. van Dooren, and F.R. de Hoog, “A Note on Downdating the Cholesky Factorization,” SIAM J. Science and Statistical Computation, vol. 8, pp. 210-221, 1987.
[5] P. Businger, “Updating a Singular Value Decomposition,” BIT, vol. 10, pp. 376-385, 1970.
[6] L. Chen, H.M. Liao, M. Ko, J. Lin, and G. Yu, “A New LDA-Based Face Recognition System which Can Solve the Small Sample Size Problem,” Pattern Recognition, vol. 33, pp. 1713-1726, 2000.
[7] N. Cristianini and J. Shawe-Taylor, Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge: Univ. Press, 2000.
[8] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification. Wiley-Interscience, 2001.
[9] L. Eldén and H. Park, “Block Downdating of Least Squares Solutions,” SIAM J. Matrix Analysis Applications, vol. 15, pp. 1018-1034, 1994.
[10] L. Eldén and H. Park, “Perturbation Analysis for Block Downdating of a Cholesky Decomposition,” Numerical Math., vol. 68, pp. 457-468, 1994.
[11] W.R. Ferng, G.H. Golub, and R.J. Plemmons, “Adaptive Lanczos Methods for Recursive Condition Estimation,” Numerical Algorithms, vol. 1, pp. 1-20, 1991.
[12] K. Fukunaga, Introduction to Statistical Pattern Recognition, second ed. Academic Press, 1990.
[13] P.E. Gill, G.H. Golub, W. Murray, and M.A. Saunders, “Methods for Modifying Matrix Factorizations,” Math. Computation, vol. 28, pp. 505-535, 1974.
[14] G.H. Golub and C.F. van Loan, Matrix Computations, third ed. Johns Hopkins Univ. Press, 1996.
[15] P. Howland, M. Jeon, and H. Park, “Structure Preserving Dimension Reduction for Clustered Text Data Based on the Generalized Singular Value Decomposition,” SIAM J. Matrix Analysis Application, vol. 25, pp. 165-179, 2003.
[16] H. Kim, “Machine Learning and Bioinformatics,” PhD thesis, Univ. of Minnesota, Twin Cities, 2004.
[17] MATLAB User's Guide, The MathWorks, Inc., 1992.
[18] S. Mika, G. Rätsch, J. Weston, B. Schölkopf, and K.R. Müller, “Fisher Discriminant Analysis with Kernels,” Proc. Conf. Neural Networks for Signal Processing, pp. 41-48, 1999.
[19] M. Moonen, P. van Dooren, and J. Vandewalle, “A Singular Value Decomposition Updating Algorithm,” SIAM J. Matrix Analysis Applications, vol. 13, pp. 1015-1038, 1992.
[20] C.C. Paige and M.A. Saunders, “Towards a Generalized Singular Value Decomposition,” SIAM J. Numerical Analysis, vol. 18, pp. 398-405, 1981.
[21] C.H. Park and H. Park, “Nonlinear Discriminant Analysis Using Kernel Functions and the Generalized Singular Value Decomposition,” SIAM J. Matrix Analysis Application, to appear.
[22] C.H. Park and H. Park, “A Relationship between LDA and the Generalized Minimum Squared Error Solution,” J. Matrix Analysis Application, to appear.
[23] D.J. Pierce and R.J. Plemmons, “Fast Adaptive Condition Estimation,” SIAM J. Matrix Analysis Applications, vol. 13, pp. 274-291, 1992.
[24] G. Shroff and C.H. Bischof, “Adaptive Condition Estimation for Rank-One Updates of QR Factorizations,” SIAM J. Matrix Analysis Application, vol. 13, pp. 1264-1278, 1992.
[25] G.W. Stewart, “The Effects of Rounding Error on an Algorithm for Downdating a Cholesky Factorization,” The Inst. Math. and Its Applications J., vol. 23, pp. 203-213, 1979.
[26] G.W. Stewart, “Updating a Rank-Revealing ULV Decomposition,” SIAM J. Matrix Analysis Application, vol. 14, pp. 494-499, 1993.
[27] G.W. Stewart, “Updating URV Decompositions in Parallel,” Parallel Computing, vol. 20, pp. 151-172, 1994.
[28] M. Stewart and P. van Dooren, “Updating a Generalized URV Decomposition,” SIAM J. Matrix Analysis Application, vol. 22, pp. 479-500, 2000.
[29] N.A. Syed, H. Liu, and K.K. Sung, “Incremental Learning with Support Vector Machines,” Proc. Workshop Support Vector Machines at the Int'l Joint Conf. Articial Intelligence (IJCAI-99), 1999.
[30] V. Vapnik, The Nature of Statistical Learning Theory. Springer-Verlag, 1995.
[31] V. Vapnik, Statistical Learning Theory. John Wiley and Sons, 1998.
[32] J. Yang and J.Y. Yang, “Why Can LDA Be Performed in PCA Transformed Space?” Pattern Recognition, vol. 36, pp. 563-566, 2003.
[33] H. Yu and J. Yang, “A Direct LDA Algorithm for High-Dimensional Data with Application to Face Recognition,” Pattern Recognition, vol. 34, pp. 2067-2070, 2001.

Index Terms:
QR decomposition updating and downdating, adaptive classifier, leave-one-out cross validation, linear discriminant analysis, kernel methods, regularization.
Citation:
Hyunsoo Kim, Barry L. Drake, Haesun Park, "Adaptive Nonlinear Discriminant Analysis by Regularized Minimum Squared Errors," IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 5, pp. 603-612, May 2006, doi:10.1109/TKDE.2006.72
Usage of this product signifies your acceptance of the Terms of Use.