Subscribe
Issue No.11 - Nov. (2013 vol.35)
pp: 2765-2781
E. Elhamifar , Dept. of Electr. Eng. & Comput. Sci., Univ. of California, Berkeley, Berkeley, CA, USA
R. Vidal , Dept. of Biomed. Eng., Johns Hopkins Univ., Baltimore, MD, USA
ABSTRACT
Many real-world problems deal with collections of high-dimensional data, such as images, videos, text, and web documents, DNA microarray data, and more. Often, such high-dimensional data lie close to low-dimensional structures corresponding to several classes or categories to which the data belong. In this paper, we propose and study an algorithm, called sparse subspace clustering, to cluster data points that lie in a union of low-dimensional subspaces. The key idea is that, among the infinitely many possible representations of a data point in terms of other points, a sparse representation corresponds to selecting a few points from the same subspace. This motivates solving a sparse optimization program whose solution is used in a spectral clustering framework to infer the clustering of the data into subspaces. Since solving the sparse optimization program is in general NP-hard, we consider a convex relaxation and show that, under appropriate conditions on the arrangement of the subspaces and the distribution of the data, the proposed minimization program succeeds in recovering the desired sparse representations. The proposed algorithm is efficient and can handle data points near the intersections of subspaces. Another key advantage of the proposed algorithm with respect to the state of the art is that it can deal directly with data nuisances, such as noise, sparse outlying entries, and missing entries, by incorporating the model of the data into the sparse optimization program. We demonstrate the effectiveness of the proposed algorithm through experiments on synthetic data as well as the two real-world problems of motion segmentation and face clustering.
INDEX TERMS
Clustering algorithms, Noise, Optimization, Sparse matrices, Vectors, Computer vision, Face,face clustering, High-dimensional data, intrinsic low-dimensionality, subspaces, clustering, sparse representation, $(\ell_1)$-minimization, convex programming, spectral clustering, principal angles, motion segmentation
CITATION
E. Elhamifar, R. Vidal, "Sparse Subspace Clustering: Algorithm, Theory, and Applications", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 11, pp. 2765-2781, Nov. 2013, doi:10.1109/TPAMI.2013.57
REFERENCES
 [1] R.E. Bellman, Dynamic Programming. Princeton Univ. Press, 1957. [2] C. Tomasi and T. Kanade, "Shape and Motion from Image Streams under Orthography," Int'l J. Computer Vision, vol. 9, no. 2, pp. 137-154, 1992. [3] R. Basri and D. Jacobs, "Lambertian Reflection and Linear Subspaces," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 3, pp. 218-233, Feb. 2003. [4] T. Hastie and P. Simard, "Metrics and Models for Handwritten Character Recognition," Statistical Science, vol. 13, no. 1, pp. 54-65, 1998. [5] R. Vidal, "Subspace Clustering," Signal Processing Magazine, vol. 28, no. 2, pp. 52-68, 2011. [6] W. Hong, J. Wright, K. Huang, and Y. Ma, "Multi-Scale Hybrid Linear Models for Lossy Image Representation," IEEE Trans. Image Processing, vol. 15, no. 12, pp. 3655-3671, Dec. 2006. [7] A. Yang, J. Wright, Y. Ma, and S. Sastry, "Unsupervised Segmentation of Natural Images via Lossy Data Compression," Computer Vision and Image Understanding, vol. 110, no. 2, pp. 212-225, 2008. [8] J. Costeira and T. Kanade, "A Multibody Factorization Method for Independently Moving Objects," Int'l J. Computer Vision, vol. 29, no. 3, pp. 159-179, 1998. [9] K. Kanatani, "Motion Segmentation by Subspace Separation and Model Selection," Proc. IEEE Int'l Conf. Computer Vision, vol. 2, pp. 586-591, 2001. [10] R. Vidal, Y. Ma, and S. Sastry, "Generalized Principal Component Analysis (GPCA)," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 12, pp. 1-15, Dec. 2005. [11] R. Duda, P. Hart, and D. Stork, Pattern Classification. Wiley-Interscience, Oct. 2004. [12] P. Tseng, "Nearest $q$ -Flat to $m$ Points," J. Optimization Theory and Applications, vol. 105, no. 1, pp. 249-252, 2000. [13] J. Ho, M.H. Yang, J. Lim, K. Lee, and D. Kriegman, "Clustering Appearances of Objects under Varying Illumination Conditions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2003. [14] T. Zhang, A. Szlam, and G. Lerman, "Median k-Flats for Hybrid Linear Modeling with Many Outliers," Proc. IEEE Int'l Workshop Subspace Methods, 2009. [15] C.W. Gear, "Multibody Grouping from Motion Images," Int'l J. Computer Vision, vol. 29, no. 2, pp. 133-150, 1998. [16] Y. Ma, A. Yang, H. Derksen, and R. Fossum, "Estimation of Subspace Arrangements with Applications in Modeling and Segmenting Mixed Data," SIAM Rev., vol. 50, pp. 413-458, 2008. [17] M. Tipping and C. Bishop, "Mixtures of Probabilistic Principal Component Analyzers," Neural Computation, vol. 11, no. 2, pp. 443-482, 1999. [18] Y. Sugaya and K. Kanatani, "Geometric Structure of Degeneracy for Multi-Body Motion Segmentation," Proc. Workshop Statistical Methods in Video Processing, 2004. [19] A. Gruber and Y. Weiss, "Multibody Factorization with Uncertainty and Missing Data Using the EM Algorithm," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 707-714, 2004. [20] M.A. Fischler and R.C. Bolles, "RANSAC Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," Comm. ACM, vol. 26, pp. 381-395, 1981. [21] S. Rao, R. Tron, R. Vidal, and Y. Ma, "Motion Segmentation in the Presence of Outlying, Incomplete, or Corrupted Trajectories," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 10, pp. 1832-1845, Oct. 2010. [22] J. Yan and M. Pollefeys, "A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-Rigid, Degenerate and Non-Degenerate," Proc. European Conf. Computer Vision, pp. 94-106, 2006. [23] A. Goh and R. Vidal, "Segmenting Motions of Different Types by Unsupervised Manifold Clustering," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007. [24] T. Zhang, A. Szlam, Y. Wang, and G. Lerman, "Hybrid Linear Modeling via Local Best-Fit Flats," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1927-1934, 2010. [25] L. Zelnik-Manor and M. Irani, "Degeneracies, Dependencies and Their Implications in Multi-Body and Multi-Sequence Factorization," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 287-293, 2003. [26] A. Ng, Y. Weiss, and M. Jordan, "On Spectral Clustering: Analysis and an Algorithm," Proc. Neural Information Processing Systems Conf., pp. 849-856, 2001. [27] U. von Luxburg, "A Tutorial on Spectral Clustering," Statistics and Computing, vol. 17, pp. 395-416, 2007. [28] G. Chen and G. Lerman, "Spectral Curvature Clustering (SCC)," Int'l J. Computer Vision, vol. 81, no. 3, pp. 317-330, 2009. [29] D.L. Donoho, "For Most Large Underdetermined Systems of Linear Equations the Minimal $\ell^1$ -Norm Solution Is Also the Sparsest Solution," Comm. Pure and Applied Math., vol. 59, no. 6, pp. 797-829, 2006. [30] E. Candès and T. Tao, "Decoding by Linear Programming," IEEE Trans. Information Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005. [31] R. Tibshirani, "Regression Shrinkage and Selection via the LASSO," J. Royal Statistical Soc. B, vol. 58, no. 1, pp. 267-288, 1996. [32] E. Candès, X. Li, Y. Ma, and J. Wright, "Robust Principal Component Analysis," J. ACM, vol. 58, 2011. [33] E. Candès and B. Recht, "Exact Matrix Completion via Convex Optimization," Foundations of Computational Math., vol. 9, pp. 717-772, 2008. [34] B. Recht, M. Fazel, and P. Parrilo, "Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization," SIAM Rev., vol. 52, no. 3, pp. 471-501, 2010. [35] E. Elhamifar and R. Vidal, "Sparse Subspace Clustering," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009. [36] E. Elhamifar and R. Vidal, "Clustering Disjoint Subspaces via Sparse Representation," Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, 2010. [37] M. Soltanolkotabi and E.J. Candes, "A Geometric Analysis of Subspace Clustering with Outliers," Annals of Statistics, vol. 40, pp. 2195-2238, 2012. [38] G. Liu, Z. Lin, and Y. Yu, "Robust Subspace Segmentation by Low-Rank Representation," Proc. Int'l Conf. Machine Learning, 2010. [39] G. Liu and S. Yan, "Latent Low-Rank Representation for Subspace Segmentation and Feature Extraction," Proc. Int'l Conf. Computer Vision, 2011. [40] G. Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma, "Robust Recovery of Subspace Structures by Low-Rank Representation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 171-184, Jan. 2013. [41] P. Favaro, R. Vidal, and A. Ravichandran, "A Closed Form Solution to Robust Subspace Estimation and Clustering," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011. [42] E. Elhamifar and R. Vidal, "Robust Classification Using Structured Sparse Representation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011. [43] E. Elhamifar and R. Vidal, "Block-Sparse Recovery via Convex Optimization," IEEE Trans. Signal Processing, 2012. [44] F. Parvaresh, H. Vikalo, S. Misra, and B. Hassibi, "Recovering Sparse Signals Using Sparse Measurement Matrices in Compressed DNA Microarrays," IEEE J. Selected Topics in Signal Processing, vol. 2, no. 3, pp. 275-285, June 2008. [45] M. Stojnic, F. Parvaresh, and B. Hassibi, "On the Reconstruction of Block-Sparse Signals with and Optimal Number of Measurements," IEEE Trans. Signal Processing, vol. 57, no. 8, pp. 3075-3085, Aug. 2009. [46] Y.C. Eldar and M. Mishali, "Robust Recovery of Signals from a Structured Union of Subspaces," IEEE Trans. Information Theory, vol. 55, no. 11, pp. 5302-5316, Nov. 2009. [47] Y.C. Eldar, P. Kuppinger, and H. Bolcskei, "Compressed Sensing of Block-Sparse Signals: Uncertainty Relations and Efficient Recovery," IEEE Trans. Signal Processing, vol. 58, no. 6, pp. 3042-3054, June 2010. [48] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge Univ. Press, 2004. [49] S.J. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinevsky, "An Interior-Point Method for Large-Scale l1-Regularized Least Squares," IEEE J. Selected Topics in Signal Processing, vol. 1, no. 4, pp. 606-617, Dec. 2007. [50] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein, "Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers," Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1-122, 2010. [51] E. Amaldi and V. Kann, "On the Approximability of Minimizing Nonzero Variables or Unsatisfied Relations in Linear Systems," Theoretical Computer Science, vol. 209, pp. 237-260, 1998. [52] R. Zass and A. Shashua, "Doubly Stochastic Normalization for Spectral Clustering," Neural Information Processing Systems, vol. 19, pp. 1569-1576, 2006. [53] T. Brox and J. Malik, "Object Segmentation by Long Term Analysis of Point Trajectories," Proc. European Conf. Computer Vision, 2010. [54] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, "Robust Face Recognition via Sparse Representation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 210-227, Feb. 2009. [55] R. Vidal and R. Hartley, "Motion Segmentation with Missing Data by Power Factorization and Generalized PCA," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 310-316, 2004. [56] D.L. Donoho and M. Elad, "Optimally Sparse Representation in General (Nonorthogonal) Dictionaries via $\ell_1$ Minimization," Proc. Nat'l Academy of Sciences USA, vol. 100, no. 5, pp. 2197-2202, 2003. [57] R. Gribonval and M. Nielsen, "Sparse Representations in Unions of Bases," IEEE Trans. Information Theory, vol. 49, no. 12, pp. 3320-3325, Dec. 2003. [58] E. van den Berg and M. Friedlander, "Theoretical and Empirical Results for Recovery from Multiple Measurements," IEEE Trans. Information Theory, vol. 56, no. 5, pp. 2516-2527, May 2010. [59] D.L. Donoho, "Neighborly Polytopes and Sparse Solution of Underdetermined Linear Equations," technical report, Stanford Univ., 2005. [60] B. Nasihatkon and R. Hartley, "Graph Connectivity in Sparse Subspace Clustering," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2011. [61] E. Elhamifar, G. Sapiro, and R. Vidal, "See All by Looking at a Few: Sparse Modeling for Finding Representative Objects," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2012. [62] R. Jenatton, J.Y. Audibert, and F. Bach, "Structured Variable Selection with Sparsity-Inducing Norms," J. Machine Learning Research, vol. 12, pp. 2777-2824, 2011. [63] J.A. Tropp, "Algorithms for Simultaneous Sparse Approximation: Part II: Convex Relaxation," Signal Processing, vol. 86, pp. 589-602, 2006. [64] D. Gabay and B. Mercier, "A Dual Algorithm for the Solution of Nonlinear Variational Problems via Finite-Element Approximations," Computer Math. Applications, vol. 2, pp. 17-40, 1976. [65] F. Lauer and C. Schnörr, "Spectral Clustering of Linear Subspaces for Motion Segmentation," Proc. IEEE Int'l Conf. Computer Vision, 2009. [66] R. Tron and R. Vidal, "A Benchmark for the Comparison of 3-D Motion Segmentation Algorithms," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007. [67] T. Boult and L. Brown, "Factorization-Based Segmentation of Motions," Proc. IEEE Workshop Motion Understanding, pp. 179-186, 1991. [68] K.-C. Lee, J. Ho, and D. Kriegman, "Acquiring Linear Subspaces for Face Recognition under Variable Lighting," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 684-698, May 2005.