The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.09 - Sept. (2013 vol.35)
pp: 2117-2130
Yao Hu , State Key Lab. of CAD&CG, Zhejiang Univ., Hangzhou, China
Debing Zhang , State Key Lab. of CAD&CG, Zhejiang Univ., Hangzhou, China
Jieping Ye , Comput. Sci. & Eng. Dept., Arizona State Univ., Tempe, AZ, USA
Xuelong Li , State Key Lab. of Transicent Opt. & Photonics, Xi'an Inst. of Opt. & Precision Mech., Xi'an, China
Xiaofei He , State Key Lab. of CAD&CG, Zhejiang Univ., Hangzhou, China
ABSTRACT
Recovering a large matrix from a small subset of its entries is a challenging problem arising in many real applications, such as image inpainting and recommender systems. Many existing approaches formulate this problem as a general low-rank matrix approximation problem. Since the rank operator is nonconvex and discontinuous, most of the recent theoretical studies use the nuclear norm as a convex relaxation. One major limitation of the existing approaches based on nuclear norm minimization is that all the singular values are simultaneously minimized, and thus the rank may not be well approximated in practice. In this paper, we propose to achieve a better approximation to the rank of matrix by truncated nuclear norm, which is given by the nuclear norm subtracted by the sum of the largest few singular values. In addition, we develop a novel matrix completion algorithm by minimizing the Truncated Nuclear Norm. We further develop three efficient iterative procedures, TNNR-ADMM, TNNR-APGL, and TNNR-ADMMAP, to solve the optimization problem. TNNR-ADMM utilizes the alternating direction method of multipliers (ADMM), while TNNR-AGPL applies the accelerated proximal gradient line search method (APGL) for the final optimization. For TNNR-ADMMAP, we make use of an adaptive penalty according to a novel update rule for ADMM to achieve a faster convergence rate. Our empirical study shows encouraging results of the proposed algorithms in comparison to the state-of-the-art matrix completion algorithms on both synthetic and real visual datasets.
INDEX TERMS
Optimization, Approximation methods, Minimization, Convergence, Acceleration, Matrix decomposition, Computer vision,accelerated proximal gradient method, Matrix completion, nuclear norm minimization, alternating direction method of multipliers
CITATION
Yao Hu, Debing Zhang, Jieping Ye, Xuelong Li, Xiaofei He, "Fast and Accurate Matrix Completion via Truncated Nuclear Norm Regularization", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 9, pp. 2117-2130, Sept. 2013, doi:10.1109/TPAMI.2012.271
REFERENCES
[1] E.J. Candès and B. Recht, "Exact Matrix Completion via Convex Optimization," Foundations on Computational Math., vol. 9, pp. 717-772, 2009.
[2] E.J. Candès and T. Tao, "The Power of Convex Relaxation: Near-Optimal Matrix Completion," IEEE Trans. Information Theory, vol. 56, no. 5, pp. 2053-2080, May 2009.
[3] J. Liu, P. Musialski, P. Wonka, and J. Ye, "Tensor Completion for Estimating Missing Values in Visual Data," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 35, no. 1, pp. 208-220, Jan. 2013.
[4] A. Eriksson and A. van den Hengel, "Efficient Computation of Robust Low-Rank Matrix Approximations in the Presence of Missing Data Using the l1 Norm," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[5] T. Okatani, T. Yoshida, and K. Deguchi, "Efficient Algorithm for Low-Rank Matrix Factorization with Missing Components and Performance Comparison of Latest Algorithms," Proc. IEEE Int'l Conf. Computer Vision, 2011.
[6] H.-Y. Shum, K. Ikeuchi, and R. Reddy, "Principal Component Analysis with Missing Data and Its Application to Polyhedral Object Modeling," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 9, pp. 854-867, Sept. 1995.
[7] S.J. Ting Kei Tong, P. Tseng, and J. Ye, "Trace Norm Regularization: Reformulation, Algorithms, and Multi-Task Learning," SIAM J. Optimization, vol. 20, no. 6, pp. 3465-3489, 2010.
[8] N. Komodakis and G. Tziritas, "Image Completion Using Global Optimization," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[9] C. Rasmussen and T. Korah, "Spatiotemporal Inpainting for Recovering Texture Maps of Partially Occluded Building Facades," Proc. IEEE Int'l Conf. Image Processing, 2005.
[10] H. Ji, C. Liu, Z. Shen, and Y. Xu, "Robust Video Denoising Using Low Rank Matrix Completion," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[11] Y. Koren, "Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model," Proc. 14th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2008.
[12] H. Steck, "Training and Testing of Recommender Systems on Data Missing Not at Random," Proc. 16th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2010.
[13] B. Recht, M. Fazel, and P.A. Parrilo, "Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization," SIAM Rev., vol. 52, no. 3, pp. 471-501, 2010.
[14] S. Ma, D. Goldfarb, and L. Chen, "Fixed Point and Bregman Iterative Methods for Matrix Rank Minimization," Math. Programming, vol. 128, nos. 1-2, pp. 321-353, 2011.
[15] B. Recht, "A Simpler Approach to Matrix Completion," J. Machine Learning Research, vol. 12, pp. 413-3430, 2011.
[16] E.J. Candès and Y. Plan, "Matrix Completion with Noise," Proc. IEEE, vol. 98, no. 6, pp. 925-936, 2009.
[17] J.F. Cai, E.J. Candès, and Z. Shen, "A Singular Value Thresholding Algorithm for Matrix Completion," SIAM J. Optimization, vol. 20, pp. 1956-1982, 2010.
[18] K.-C. Toh and S. Yun, "An Accelerated Proximal Gradient Algorithm for Nuclear Norm Regularized Least Squares Problems," Pacific J. Optimization, pp. 615-640, 2010.
[19] J. Wright, A. Ganesh, S. Rao, Y. Peng, and Y. Ma, "Robust Principal Component Analysis: Exact Recovery of Corrupted Low-Rank Matrices via Convex Optimization," Proc. Advances in Neural Information Processing Systems, 2009.
[20] D. Zhang, Y. Hu, J. Ye, X. Li, and X. He, "Matrix Completion by Truncated Nuclear Norm Regularization," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2012.
[21] R.S. Cabral, F. De la Torre, J.P. Costeira, and A. Bernardino, "Matrix Completion for Multi-Label Image Classification," Proc. Advances in Neural Information Processing Systems, 2011.
[22] R. Foygel, R.R. Salakhutdinov, O. Shamir, and N. Srebro, "Learning with the Weighted Trace-Norm under Arbitrary Sampling Distributions," Proc. Advances in Neural Information Processing Systems, 2011.
[23] A. Goldberg, X.J. Zhu, B. Recht, J. Xu, and R. Nowak, "Transduction with Matrix Completion: Three Birds with One Stone," Proc. Advances in Neural Information Processing Systems, 2010.
[24] Y. Peng, A. Ganesh, J. Wright, W. Xu, and Y. Ma, "RASL: Robust Alignment by Sparse and Low-Rank Decomposition for Linearly Correlated Images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2233-2246, Nov. 2012.
[25] R. Salakhutdinov and N. Srebro, "Collaborative Filtering in a Non-Uniform World: Learning with the Weighted Trace Norm," Proc. Advances in Neural Information Processing Systems, 2010.
[26] M. Fazel, H. Hindi, and S.P. Boyd, "A Rank Minimization Heuristic with Application to Minimum Order System Approximation," Proc. Am. Control Conf., pp. 4734-4739. 2001,
[27] W. Dai, O. Milenkovic, and E. Kerman, "Subspace Evolution and Transfer (Set) for Low-Rank Matrix Completion," IEEE Trans. Signal Processing, vol. 59, no. 7, pp. 3120-3132, July 2011.
[28] M. Fazel, "Matrix Rank Minimization with Applications," PhD thesis, Stanford Univ., 2002.
[29] R.H. Tutuncu, K.C. Toh, and M.J. Todd, Sdpt3—a Matlab Software Package for Semidefinite Quadratic Linear Programming, Version 3.0, 2001.
[30] J.F. Sturm, Using Sedumi 1.02, a Matlab Toolbox for Optimization over Symmetric Cones, 1998.
[31] S. Ji and J. Ye, "An Accelerated Gradient Method for Trace Norm Minimization," Proc. 26th Ann. Int'l Conf. Machine Learning, 2009.
[32] N. Srebro, J.D.M. Rennie, and T.S. Jaakkola, "Maximum-Margin Matrix Factorization," Proc. Advances in Neural Information Processing Systems, 2005.
[33] J.D.M. Rennie and N. Srebro, "Fast Maximum Margin Matrix Factorization for Collaborative Prediction," Proc. 22nd Int'l Conf. Machine Learning, 2005.
[34] G. Takács, I. Pilászy, B. Németh, and D. Tikk, "Scalable Collaborative Filtering Approaches for Large Recommender Systems," J. Machine Learning Research, 2009.
[35] S. Burer and R.D.C. Monteiro, "Local Minima and Convergence in Low-Rank Semidefinite Programming," Math. Programming, vol. 103, no. 3, pp. 427-444, 2005.
[36] R.H. Keshavan, A. Montanari, and S. Oh, "Matrix Completion from Noisy Entries," J. Machine Learning Research, vol. 11, pp. 2057-2078, 2010.
[37] R. Meka, P. Jain, and I.S. Dhillon, "Guaranteed Rank Minimization via Singular Value Projection," Proc. Advances in Neural Information Processing Systems, 2010.
[38] J. Yang and X. Yuan, "Linearized Augmented Lagrangian and Alternating Direction Methods for Nuclear Norm Minimization," submitted, 2011.
[39] Z. Lin, R. Liu, and Z. Su, "Linearized Alternating Direction Method with Adaptive Penalty for Low-Rank Representation," Proc. Advances in Neural Information Processing Systems, 2011.
[40] M. Li, J.T. Kwok, and B.-L. Lu, "Making Large-Scale Nyström Approximation Possible," Proc. 27th Ann. Int'l Conf. Machine Learning, 2010.
[41] S. Boyd, N. Parikh, E. Chu, and J. Eckstein, "Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers," Information Systems J., vol. 3, no. 1, pp. 1-122, 2010.
[42] Y. Nesterov, "Gradient Methods for Minimizing Composite Objective Function," technical report, 2007.
[43] A. Beck and M. Teboulle, "A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems," SIAM J. Imaging Sciences, vol. 2, no. 1, pp. 183-202, 2009.
[44] Y. Nesterov, "A Method of Solving a Convex Programming Problem with Convergence Rate o(1/sqr(k))," Soviet Math. Doklady, vol. 27, pp. 372-376, 1983.
[45] B. He, M. Tao, and X. Yuan, "Alternating Direction Method with Gaussian Back Substitution for Separable Convex Programming," SIAM J. Optimization, vol. 22, no. 2, pp. 313-340, 2012.
[46] M. Tao and X. Yuan, "Recovering Low-Rank and Sparse Components of Matrices from Incomplete and Noisy Observations," SIAM J. Optimizaiton, vol. 21, no. 1, pp. 57-81, 2011.
[47] B.S. He, H. Yang, and S.L. Wang, "Alternating Direction Method with Self-Adaptive Penalty Parameters for Monotone Variational Inequalities," J. Optimization Theory and Applications, vol. 106, no. 2, pp. 337-356, 2000.
[48] R.H. Keshavan, A. Montanari, and S. Oh, "Matrix Completion from a Few Entries," IEEE Trans. Information Theory, vol. 56, pp. 2980-2998, 2010.
82 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool