This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Out-of-Sample Extrapolation of Learned Manifolds
September 2008 (vol. 30 no. 9)
pp. 1547-1556
We investigate the problem of extrapolating the embedding of a manifold learned from finite samples to novel out-of-sample data. We concentrate on the manifold learning method called Maximum Variance Unfolding (MVU) for which the extrapolation problem is still largely unsolved. Taking the perspective of MVU learning being equivalent to Kernel PCA, our problem reduces to extending a kernel matrix generated from an unknown kernel function to novel points. Leveraging on previous developments, we propose a novel solution which involves approximating the kernel eigenfunction using Gaussian basis functions. We also show how the width of the Gaussian can be tuned to achieve extrapolation. Experimental results which demonstrate the effectiveness of the proposed approach are also included.

[1] C. Burges, “Geometric Methods for Feature Extraction and Dimensional Reduction,” Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers, L.Rokach and O. Maimon, eds. Kluwer Academic Publishers, 2005.
[2] L. Saul, K. Weinberger, F. Sha, J. Ham, and D. Lee, “Spectral Methods for Dimensionality Reduction,” Semisupervised Learning, O. Chapelle, B. Schölkopf, and A. Zien, eds. MIT Press, 2006.
[3] K. Weinberger and L. Saul, “Unsupervised Learning of Image Manifolds by Semidefinite Programming,” Int'l J. Computer Vision, vol. 70, no. 1, pp. 77-90, 2006.
[4] Y. Bengio, J.-F. Paiement, P. Vincent, O. Delalleau, N. Le Roux, and M. Ouimet, “Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering,” Proc. Advances in Neural Information Processing Systems, 2003.
[5] Y. Bengio, O. Delalleau, N. Le Roux, J.-F. Paiement, P. Vincent, and M. Ouimet, “Learning Eigenfunctions Links Spectral Embedding and Kernel PCA,” Neural Computation, vol. 16, no. 10, pp.2197-2219, 2004.
[6] B. Schölkopf, A. Smola, and K.-R. Müller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, vol. 10, pp. 1299-1319, 1998.
[7] V. de Silva and J. Tenenbaum, “Global versus Local Methods in Nonlinear Dimensionality Reduction,” Proc. Advances in Neural Information Processing Systems, pp. 705-712, 2003.
[8] K. Weinberger, B. Packer, and L. Saul, “Nonlinear Dimensionality Reduction by Semidefinite Programming and Kernel Matrix Factorization,” Proc. 10th Int'l Workshop Artificial Intelligence and Statistics (AISTATS '05), 2005.
[9] S. Vishwanathan, K. Borgwardt, O. Guttman, and A. Smola, “Kernel Extrapolation,” Neurocomputing, no. 69, pp. 721-729, 2006.
[10] H. Gong, C. Pan, Q. Yang, H. Lu, and S. Ma, “Neural Network Modeling of Spectral Embedding,” Proc. 17th British Machine Vision Conf. (BMVC '06), 2006.
[11] A. Elgammal and C.-S. Lee, “Inferring 3D Body Pose from Silhouettes Using Activity Manifold Learning,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR '04), vol. 2, pp. 681-688, 2004.
[12] L.K. Saul and S.T. Roweis, “Think Globally, Fit Locally: Unsupervised Learning of Low-Dimensional Manifolds,” J.Machine Learning Research, vol. 4, pp. 119-155, 2003.
[13] C. Baker, The Numerical Treatment of Integral Equations. Clarendon Press, 1977.
[14] C. Williams and M. Seeger, “Using the Nyström Method to Speed Up Kernel Machines,” Proc. Advances in Neural Information Processing Systems, pp. 682-688, 2001.
[15] A. Schwaighofer, V. Tresp, and K. Yu, “Learning Gaussian Process Kernels via Hierarchical Bayes,” Proc. Advances in Neural Information Processing Systems, 2005.
[16] I.T. Nabney, NETLAB: Algorithms for Pattern Recognition. Springer, 2004.
[17] G. Peters, “Efficient Pose Estimation Using View-Based Object Representations,” Machine Vision and Applications, vol. 16, no. 1, pp. 59-63, 2004.

Index Terms:
manifold learning, out-of-sample extrapolation, Maximum Variance Unfolding
Citation:
Tat-Jun Chin, David Suter, "Out-of-Sample Extrapolation of Learned Manifolds," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 9, pp. 1547-1556, Sept. 2008, doi:10.1109/TPAMI.2007.70813
Usage of this product signifies your acceptance of the Terms of Use.