The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.07 - July (2010 vol.22)
pp: 929-942
Si Si , Nanyang Technological University, Singapore
Dacheng Tao , Nanyang Technological University, Singapore
Bo Geng , Nanyang Technological University, Singapore
ABSTRACT
The regularization principals [CHECK END OF SENTENCE] lead approximation schemes to deal with various learning problems, e.g., the regularization of the norm in a reproducing kernel Hilbert space for the ill-posed problem. In this paper, we present a family of subspace learning algorithms based on a new form of regularization, which transfers the knowledge gained in training samples to testing samples. In particular, the new regularization minimizes the Bregman divergence between the distribution of training samples and that of testing samples in the selected subspace, so it boosts the performance when training and testing samples are not independent and identically distributed. To test the effectiveness of the proposed regularization, we introduce it to popular subspace learning algorithms, e.g., principal components analysis (PCA) for cross-domain face modeling; and Fisher's linear discriminant analysis (FLDA), locality preserving projections (LPP), marginal Fisher's analysis (MFA), and discriminative locality alignment (DLA) for cross-domain face recognition and text categorization. Finally, we present experimental evidence on both face image data sets and text data sets, suggesting that the proposed Bregman divergence-based regularization is effective to deal with cross-domain learning problems.
INDEX TERMS
Dimensionality reduction, regularization, and Bregman divergence.
CITATION
Si Si, Dacheng Tao, Bo Geng, "Bregman Divergence-Based Regularization for Transfer Subspace Learning", IEEE Transactions on Knowledge & Data Engineering, vol.22, no. 7, pp. 929-942, July 2010, doi:10.1109/TKDE.2009.126
REFERENCES
[1] W. Dai, Q. Yang, and G.-R. Xue, "Boosting for Transfer Learning," Proc. 24th Int'l Conf. Machine Learning, pp. 193-200, 2007.
[2] D. Tao, X. Li, X. Wu, and S.J. Maybank, "Geometric Mean for Subspace Selection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 260-274, Feb. 2009.
[3] G. Obozinski, B. Taskar, and M.I. Jordan, "Multi-Task Feature Selection," technical report, Dept. of Statistics, Univ. of California, Berkeley, 2006.
[4] M. Belkin, P. Niyogi, and V. Sindhwani, "Manifold Regularization: A Geometric Framework for Learning from Labeled and Unlabeled Examples," J. Machine Learning Research, vol. 7, pp. 2399-2434, 2006.
[5] K. Lang, "Newsweeder: Learning to Filter Netnews," Proc. 12th Int'l Conf. Machine Learning, pp. 331-339, 1995.
[6] R. Caruana, "Multitask Learning," Machine Learning, vol. 29, no. 1, pp. 41-75, July 1997.
[7] P.N. Belhumeur, J.P. Hespanha, and D.J. Kriegman, "Eigenfaces versus Fisherfaces: Recognition Using Class Specific Linear Projection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[8] D. Cai, X. He, and J. Han, "SRDA: An Efficient Algorithm for Large Scale Discriminant Analysis," IEEE Trans. Knowledge and Data Eng., vol. 20, no. 1, pp. 1-12, Jan. 2008.
[9] L. Torrey, J. Shavlik, S. Natarajan, P. Kuppili, and T. Walker, "Transfer in Reinforcement Learning via Markov Logic Networks," Proc. AAAI Workshop Transfer Learning for Complex Tasks, July 2008.
[10] J. Zhang, Z. Ghahramani, and Y. Yang, "Learning Multiple Related Tasks Using Latent Independent Component Analysis," Advances in Neural Information Processing Systems, vol. 18, pp. 1585-1592, MIT Press, 2006.
[11] X. Li, S. Lin, S. Yan, and D. Xu, "Discriminant Locally Linear Embedding with High-Order Tensor Data," IEEE Trans. Systems, Man, and Cybernetics, Part B, vol. 38, no. 2, pp. 342-352, Apr. 2008.
[12] D. Cai, X. He, J. Han, and H.-J. Zhang, "Orthogonal Laplacianfaces for Face Recognition," IEEE Trans. Image Processing, vol. 15, no. 11, pp. 3608-3614, Nov. 2006.
[13] C.B. Do and A.Y. Ng, "Transfer Learning for Text Classification," Advances in Neural Information Processing System, pp. 299-306, MIT Press, 2006.
[14] S.J. Pan, J.T. Kwok, and Q. Yang, "Transfer Learning via Dimensionality Reduction," Proc. 23rd Nat'l Conf. Artificial Intelligence, pp. 677-682, 2008.
[15] X. He, S. Yan, Y. Hu, P. Niyogi, and H.-J. Zhang, "Face Recognition Using Laplacianfaces," IEEE Trans. Patten Analysis and Machine Intelligence, vol. 27, no. 3, pp. 328-340, Mar. 2005.
[16] K. Torkkola, I. Guyon, and A. Elisseeff, "Feature Extraction by Non Parametric Mutual Information Maximization," J. Machine Learning Research, vol. 3, pp. 1415-1438, 2003.
[17] S. Bickel, M. Brückner, and T. Scheffer, "Discriminative Learning for Differing Training and Test Distributions," Proc. 24th Int'l Conf. Machine Learning, pp. 81-88, June 2007.
[18] D. Cai, X. He, and J. Han, "Semi-Supervised Discriminant Analysis," Proc. IEEE Int'l Conf. Computer Vision, pp. 1-7, Oct. 2007.
[19] M.T. Rosenstein, Z. Marx, and L.P. Kaelbling, "To Transfer or Not to Transfer," Proc. Neural Information Processing Systems 2005 Workshop Inductive Transfer: 10 Years Later, 2005.
[20] D. Tao, X. Li, X. Wu, and S.J. Maybank, "General Tensor Discriminant Analysis and Gabor Features for Gait Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1700-1715, Oct. 2007.
[21] X. He, D. Cai, and J. Han, "Learning a Maximum Margin Subspace for Image Retrieval," IEEE Trans. Knowledge and Data Eng., vol. 20, no. 2, pp. 189-201, Feb. 2008.
[22] H. Hotelling, "Analysis of a Complex of Statistical Variables into Principal Components," J. Educational Psychology, vol. 24, pp. 417-441, 1933.
[23] A. Farhadi, D. Forsyth, and R. White, "Transfer Learning in Sign Language," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1-8, June 2007.
[24] J. Huang, A. Smola, A. Gretton, K.M. Borgwardt, and B. Scholkopf, "Correcting Sample Selection Bias by Unlabeled Data," Advances in Neural Information Processing Systems, pp. 601-608, MIT Press, 2006.
[25] B. Zadrozny, "Learning and Evaluating Classifiers under Sample Selection Bias," Proc. 21st Int'l Conf. Machine Learning, pp. 114-121, 2004.
[26] T.-K. Kim, J. Kittler, and R. Cipolla, "Discriminative Learning and Recognition of Image Set Classes Using Canonical Correlations," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 1005-1018, June 2007.
[27] R.A. Fisher, "The Use of Multiple Measurements in Taxonomic Problems," Annals of Eugenics, vol. 7, pp. 179-188, 1936.
[28] X. He and P. Niyogi, "Locality Preserving Projections," Advances in Neural Information Processing Systems, vol. 16, MIT Press, 2003.
[29] Y. Shuicheng, D. Xu, B. Zhang, and H.-J. Zhang, "Graph-Embedding: A General Framework for Dimensionality Reduction," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 830-837, 2005.
[30] J.P. Phillips, H. Moon, S.A. Rizvi, and P.J. Rauss, "The FERET Evaluation Methodology for Face-Recognition Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1090-1104, Oct. 2000.
[31] F. Girosi, M. Jones, and T. Poggio, "Regularisation Theory and Neural Networks Architectures," Neural Computation, vol. 7, pp. 219-269, 1995.
[32] V.N. Vapnik, Statistical Learning Theory. Wiley, 1998.
[33] D.B. Graham and N.M. Allinson, "Characterizing Virtual Eigensignatures for General Purpose Face Recognition," Face Recognition: From Theory to Applications, H. Wechsler, P.J. Pillips, V. Bruce, F. Fogelman-Soulie, and T.S. Huang, eds., pp. 446-456, Springer, 1998.
[34] T. Zhang, D. Tao, and J. Yang, "Discriminative Locality Alignment," Proc. 10th European Conf. Computer Vision, pp. 725-738, 2008.
[35] A.N. Tikhonov, "Regularization of Incorrectly Posed Problems," Soviet Math Dokl, vol. 4, pp. 1624-1627, 1963.
[36] R.K. Ando and T. Zhang, "Learning on Graph with Laplacian Regularization," Advances in Neural Information Processing Systems, vol. 19, pp. 25-33, MIT Press, 2006.
[37] B. Geng, D. Tao, C. Xu, L. Yang, and X.-S. Hua, "Ensemble Manifold Regularization," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2009.
[38] T. Zhang, D. Tao, X. Li, and J. Yang, "Patch Alignment for Dimensionality Reduction," IEEE Trans. Knowledge and Data Eng., vol. 21, no. 9, pp. 1299-1313, 2009.
[39] T. Zhang, D. Tao, X. Li, and J. Yang, "A Unifying Framework for Spectral Analysis Based Dimensionality Reduction," Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 1671-1678, 2008.
[40] W. Bian and D. Tao, "Harmonic Mean for Subspace Selection," Proc. Int'l Conf. Pattern Recognition, pp. 1-4, 2008.
21 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool