This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Learning Nonlinear Image Manifolds by Global Alignment of Local Linear Models
August 2006 (vol. 28 no. 8)
pp. 1236-1250
Appearance-based methods, based on statistical models of the pixel values in an image (region) rather than geometrical object models, are increasingly popular in computer vision. In many applications, the number of degrees of freedom (DOF) in the image generating process is much lower than the number of pixels in the image. If there is a smooth function that maps the DOF to the pixel values, then the images are confined to a low-dimensional manifold embedded in the image space. We propose a method based on probabilistic mixtures of factor analyzers to 1) model the density of images sampled from such manifolds and 2) recover global parameterizations of the manifold. A globally nonlinear probabilistic two-way mapping between coordinates on the manifold and images is obtained by combining several, locally valid, linear mappings. We propose a parameter estimation scheme that improves upon an existing scheme and experimentally compare the presented approach to self-organizing maps, generative topographic mapping, and mixtures of factor analyzers. In addition, we show that the approach also applies to finding mappings between different embeddings of the same manifold.

[1] A. Leonardis and H. Bischof, “Kernel and Subspace Methods for Computer Vision,” Pattern Recognition, vol. 36, no. 9, pp. 1925-1927, 2003.
[2] S.T. Roweis, L.K. Saul, and G.E. Hinton, “Global Coordination of Loca Linear Models,” Advances in Neural Information Processing Systems, vol. 14, pp. 889-896, 2002.
[3] L. Sirovich and M. Kirby, “Low-Dimensional Procedure for the Characterization of Human Faces,” J. Optical Soc. Am. A, vol. 4, no. 3, pp. 519-524, 1987.
[4] I.T. Jolliffe, Principal Component Analysis. New York: Spinger-Verlag, 1986.
[5] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.
[6] H. Murase and S.K. Nayar, “Visual Learning and Recognition of 3-D Objects from Appearance,” Int'l J. Computer Vision, vol. 14, pp. 5-24, 1995.
[7] E. Oja, “Data Compression, Feature Extraction, and Autoassociation in Feedforward Neural Networks,” Proc. Int'l Conf. Artificial Neural Networks, pp. 737-745, 1991.
[8] T. Hastie and W. Stuetzle, “Principal Curves,” J. Am. Statistical Assoc., vol. 84, no. 406, pp. 502-516, 1989.
[9] B. Kégl, A. Krzyzak, T. Linder, and K. Zeger, “Learning and Design of Principal Curves,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 3, pp. 281-297, Mar. 2000.
[10] K. Chang and J. Ghosh, “A Unified Model for Probabilistic Principal Surfaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 1, pp. 22-41, Jan. 2001.
[11] T. Kohonen, Self-Organizing Maps. New York: Spinger-Verlag, 1995.
[12] J.J. Verbeek, N. Vlassis, and B.J.A. Kröse, “Self-Organizing Mixture Models,” Neurocomputing, vol. 63, pp. 99-123, 2005.
[13] C.M. Bishop, M. Svensén, and C.K.I. Williams, “GTM: The Generative Topographic Mapping,” Neural Computation, vol. 10, pp. 215-234, 1998.
[14] J.J. Verbeek, “Mixture Models for Clustering and Dimension Reduction,” PhD dissertation, Univ. of Amsterdam, 2004.
[15] I.K. Fodor, “A Survey of Dimension Reduction Techniques,” Technical Report UCRL-ID-148494, Lawrence Livermore Nat'l Laboratory, Center for Applied Scientific Computing, 2002.
[16] M.Á. Carreira-Perpiñán, “A Review of Dimension Reduction Techniques,” Technical Report CS-96-09, Dept. of Computer Science, Univ. of Sheffield, 1997.
[17] J.B. Tenenbaum, V. de Silva, and J.C. Langford, “A Global Geometric Framework for Nonlinear-Dimensionality Reduction,” Science, vol. 290, no. 5500, pp. 2319-2323, 2000.
[18] S.T. Roweis and L.K. Saul, “Nonlinear-Dimensionality Reduction by Locally Linear Embedding,” Science, vol. 290, no. 5500, pp. 2323-2326, 2000.
[19] B. Schölkopf, A.J. Smola, and K.-R. Müller, “Nonlinear Component Analysis as a Kernel Eigenvalue Problem,” Neural Computation, vol. 10, pp. 1299-1319, 1998.
[20] M. Brand, “Charting a Manifold,” Advances in Neural Information Processing Systems, vol. 15, pp. 961-968, 2003.
[21] X. He and P. Niyogi, “Locality Preserving Projections,” Advances in Neural Information Processing Systems, vol. 16, pp. 153-160, 2004.
[22] X. He, S. Yan, Y. Hu, P. Niyogi, and H.-J. Zhang, “Face Recognition Using Laplacianfaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 3, pp. 328-340, Mar. 2005.
[23] M. Belkin and P. Niyogi, “Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering,” Advances in Neural Information Processing Systems, vol. 14, pp. 585-591, 2002.
[24] K.Q. Weinberger and L.K. Saul, “Unsupervised Learning of Image Manifolds by Semidefinite Programming,” Proc. Int'l Conf. Computer Vision and Pattern Recognition, pp. 988-995, 2004.
[25] Y. Bengio, J.-F. Paiement, P. Vincent, O. Delalleau, N. Le Roux, and M. Ouimet, “Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering,” Advances in Neural Information Processing Systems, vol. 16, pp. 177-184, 2004.
[26] N. Kambhatla and T.K. Leen, “Fast Nonlinear Dimension Reduction,” Advances in Neural Information Processing Systems, vol. 6, pp. 152-159, 1994.
[27] C. Bregler and S.M. Omohundro, “Nonlinear Image Interpolation Using Manifold Learning,” Advances in Neural Information Processing Systems, vol. 7, pp. 973-980, 1995.
[28] M.E. Tipping and C.M. Bishop, “Mixtures of Probabilistic Principal Component Analysers,” Neural Computation, vol. 11, no. 2, pp. 443-482, 1999.
[29] G.E. Hinton, P. Dayan, and M. Revow, “Modeling the Manifolds of Images of Handwritten Digits,” IEEE Trans. Neural Networks, vol. 8, no. 1, pp. 65-74, 1997.
[30] R. Bellman, Adaptive Control Processes: A Guided Tour. Princeton Univ. Press, 1961.
[31] J.H. Ham, D.D. Lee, and L.K. Saul, “Learning High-Dimensional Correspondences from Low-Dimensional Manifolds,” Proc. Workshop the Continuum from Labeled to Unlabeled Data in Machine Learning and Data Mining, 2003.
[32] X. Meng and D. van Dyk, “The EM Algorithm— An Old Folk Song Sung to a Fast New Tune,” J. Royal Statistical Soc., Series B (Methodological), vol. 59, no. 1, pp. 511-567, 1997.
[33] D. de Ridder and V. Franc, “Robust Subspace Mixture Models Using t-Distributions,” Proc. British Machine Vision Conf., pp. 319-328, 2003.
[34] A.P. Dempster, N.M. Laird, and D.B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. Royal Statistical Soc., Series B (Methodological), vol. 39, no. 1, pp. 1-38, 1977.
[35] Z. Ghahramani and G.E. Hinton, “The EM Algorithm for Mixtures of Factor Analyzers,” Technical Report CRG-TR-96-1, Univ. of Toronto, 1996.
[36] Y.W. Teh and S.T. Roweis, “Automatic Alignment of Local Representations,” Advances in Neural Information Processing Systems, vol. 15, pp. 841-848, 2003.
[37] J.J. Verbeek, S.T. Roweis, and N. Vlassis, “Nonlinear CCA and PCA by Alignment of Local Models,” Advances in Neural Information Processing Systems, vol. 16, pp. 297-304, 2004.
[38] J. Wieghardt, “Learning the Topology of Views: From Images to Objects,” PhD dissertation, Ruhr-Univ.-Bochum, Bochum, Germany, 2001.
[39] L.K. Saul and S.T. Roweis, “Think Globally, Fit Locally: Unsupervised Learning of Low-Dimensional Manifolds,” J. Machine Learning Research, vol. 4, pp. 119-155, 2003.
[40] S. Chretien and A.O. Hero, “Kullback Proximal Algorithms for Maximum Likelihood Estimation,” IEEE Trans. Information Theory, vol. 46, no. 5, pp. 1800-1810, 2000.
[41] R. Jacobs, M.I. Jordan, S.J. Nowlan, and G.E. Hinton, “Adaptive Mixtures of Local Experts,” Neural Computation, vol. 3, pp. 79-87, 1991.
[42] J.J. Verbeek, N. Vlassis, and B.J.A. Kröse, “Coordinating Principal Component Analyzers,” Proc. Int'l Conf. Artificial Neural Networks, vol. 12, pp. 914-919, 2002.
[43] B. Kégl, “Intrinsic Dimension Estimation Using Packing Numbers,” Advances in Neural Information Processing Systems, vol. 15, pp. 681-688, 2003.
[44] J.A. Costa and A.O. Hero, “Geodesic Entropic Graphs for Dimension and Entropy Estimation in Manifold Learning,” IEEE Trans. Signal Processing, vol. 52, no. 8, pp. 2210-2221, 2004.
[45] E. Levina and P.J. Bickel, “Maximum Likelihood Estimation of Intrinsic Dimension,” Advances in Neural Information Processing Systems, vol. 17, pp. 777-784, 2005.
[46] J.J. Oliver, R.A. Baxter, and C.S. Wallace, “Unsupervised Learning Using MML,” Proc. Int'l Conf. Machine Learning, vol. 13, pp. 364-374, 1996.
[47] M.A.T. Figueiredo and A.K. Jain, “Unsupervised Learning of Finite Mixture Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 3, pp. 381-396, Mar. 2002.
[48] Z. Ghahramani and M.J. Beal, “Variational Inference for Bayesian Mixtures of Factor Analysers,” Advances in Neural Information Processing Systems, vol. 12, pp. 449-455, 2000.
[49] J.H. Ham, D.D. Lee, and L.K. Saul, “Semisupervised Alignment of Manifolds,” Proc. Ann. Conf. Uncertainty in Artificial Intelligence, vol. 10, pp. 120-127, 2005.
[50] H. Ritter, “Parametrized Self-Organizing Maps,” Proc. Int'l Conf. Artificial Neural Networks, vol. 3, pp. 568-577, 1993.
[51] Z. Ghahramani and M.I. Jordan, “Supervised Learning from Incomplete Data via an EM Approach,” Advances in Neural Information Processing Systems, vol. 6, pp. 120-127, 1994.
[52] H. Karger, “Riemann Center of Mass and Mollifier Smoothing,” Comm. Pure and Applied Math., vol. 3, pp. 509-541, 1977.
[53] A.R. Webb, Statistical Pattern Recognition. New York: Wiley, 2002.
[54] G. Peters, B. Zitova, and C. von der Malsburg, “How to Measure the Pose Robustness of Object Views,” Image and Vision Computing, vol. 20, no. 4, pp. 249-256, 2002.

Index Terms:
Feature extraction or construction, machine learning, statistical image representation.
Citation:
Jakob Verbeek, "Learning Nonlinear Image Manifolds by Global Alignment of Local Linear Models," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 8, pp. 1236-1250, Aug. 2006, doi:10.1109/TPAMI.2006.166
Usage of this product signifies your acceptance of the Terms of Use.