The Community for Technology Leaders
Subscribe
Issue No.02 - Feb. (2014 vol.36)
pp: 261-275
Ran He , Nat. Lab. of Pattern Recognition (NLPR), Inst. of Autom., Beijing, China
Wei-Shi Zheng , Sch. of Inf. Sci. & Technol., Sun Yat-sen Univ., Guangzhou, China
Tieniu Tan , Nat. Lab. of Pattern Recognition (NLPR), Inst. of Autom., Beijing, China
Zhenan Sun , Nat. Lab. of Pattern Recognition (NLPR), Inst. of Autom., Beijing, China
ABSTRACT
Robust sparse representation has shown significant potential in solving challenging problems in computer vision such as biometrics and visual surveillance. Although several robust sparse models have been proposed and promising results have been obtained, they are either for error correction or for error detection, and learning a general framework that systematically unifies these two aspects and explores their relation is still an open problem. In this paper, we develop a half-quadratic (HQ) framework to solve the robust sparse representation problem. By defining different kinds of half-quadratic functions, the proposed HQ framework is applicable to performing both error correction and error detection. More specifically, by using the additive form of HQ, we propose an ℓ1-regularized error correction method by iteratively recovering corrupted data from errors incurred by noises and outliers; by using the multiplicative form of HQ, we propose an ℓ1-regularized error detection method by learning from uncorrupted data iteratively. We also show that the ℓ1-regularization solved by soft-thresholding function has a dual relationship to Huber M-estimator, which theoretically guarantees the performance of robust sparse representation in terms of M-estimation. Experiments on robust face recognition under severe occlusion and corruption validate our framework and findings.
INDEX TERMS
Robustness, Minimization, Additives, Error correction, Optimization, Algorithm design and analysis, Noise,correntropy, $(\ell_1)$-minimization, half-quadratic optimization, sparse representation, M-estimator
CITATION
Ran He, Wei-Shi Zheng, Tieniu Tan, Zhenan Sun, "Half-Quadratic-Based Iterative Minimization for Robust Sparse Representation", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.36, no. 2, pp. 261-275, Feb. 2014, doi:10.1109/TPAMI.2013.102
REFERENCES
 [1] J. Wright, Y. Ma, J. Mairal, G. Sapiro, T.S. Huang, and S. Yan, "Sparse Representation for Computer Vision and Pattern Recognition," Proc. IEEE, vol. 98, no. 6, pp. 1031-1044, June 2010. [2] B. Cheng, J. Yang, S. Yan, Y. Fu, and T.S. Huang, "Learning with $\ell_1$ -Graph for Image Analysis," IEEE Trans. Image Processing, vol. 19, no. 4, pp. 858-866, Apr. 2010. [3] R. He, W.-S. Zheng, and B.-G. Hu, "Maximum Correntropy Criterion for Robust Face Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 33, no. 8, pp. 1561-1576, Aug. 2011. [4] E.J. Candés and T. Tao, "Near Optimal Signal Recovery from Random Projections: Universal Encoding Strategies," IEEE Trans. Information Theory, vol. 52, no. 12, pp. 5406-5425, Dec. 2006. [5] D.L. Donoho, "Compressed Sensing," IEEE Trans. Information Theory, vol. 52, no. 4, pp. 1289-1306, Apr. 2006. [6] M. Aharon, M. Elad, and A.M. Bruckstein, "The K-SVD: An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representations," IEEE Trans. Signal Processing, vol. 54, no. 11, pp. 4311-4322, Nov. 2006. [7] J. Mairal, G. Sapiro, and M. Elad, "Learning Multiscale Sparse Representations for Image and Video Restoration," SIAM Multiscale Modeling and Simulation, vol. 7, no. 1, pp. 214-241, 2008. [8] M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. Kelly, and R. Baraniuk, "An Architecture for Compressive Image," Proc. Int'l Conf. Image Processing, pp. 1273-1276, 2006. [9] D. Takhar, J. Laska, M. Wakin, M. Duarte, D. Baron, S. Sarvotham, K. Kelly, and R. Baraniuk, "A New Compressive Imaging Camera Architecture Using Optical-Domain Compression," Proc. Computational Imaging IV at SPIE Electronic Imaging, pp. 43-52, 2006. [10] W. Bajwa, J. Haupt, A. Sayeed, and R. Nowak, "Compressive Wireless Sensing," Proc. Int'l Conf. Information Processing in Sensor Networks, 2006. [11] E. Elhamifar and R. Vidal, "Sparse Subspace Clustering: Algorithm, Theory, and Applications," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 35, no. 11, pp. 2765-2781, Nov. 2013. [12] J. Wright and Y. Ma, "Dense Error Correction via $\ell_1$ -Minimization," IEEE Trans. Information Theory, vol. 56, no. 7, pp. 3540-3560, July 2010. [13] A.Y. Yang, S.S. Sastry, A. Ganesh, and Y. Ma, "Fast $\ell_1$ -Minimization Algorithms and an Application in Robust Face Recognition: A Review," Proc. Int'l Conf. Image Processing, 2010. [14] J. Wright, A.Y. Yang, A. Ganesh, S.S. Sastry, and Y. Ma, "Robust Face Recognition via Sparse Representation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 2, pp. 210-227, Feb. 2009. [15] R. He, W.-S. Zheng, B.-G. Hu, and X.-W. Kong, "A Regularized Correntropy Framework for Robust Pattern Recognition," Neural Computation, vol. 23, no. 8, pp. 2074-2100, 2011. [16] M. Yang, L. Zhang, J. Yang, and D. Zhang, "Robust Sparse Coding for Face Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 625-632, 2011. [17] N.H. Nguyen and T.D. Tran, "Exact Recoverability from Dense Corrupted Observations via $\ell_1$ -Minimization," IEEE Trans. Information Theory, vol. 59, no. 4, pp. 2017-2035, Apr. 2013. [18] S. Vaiter, G. Peyre, C. Dossal, and J. Fadili, "Robust Sparse Analysis Regularization," IEEE Trans. Information Theory, vol. 59, no. 4, pp. 2001-2016, 2013. [19] S. Chen, D.L. Donoho, and M. Saunders, "Atomic Decomposition by Basis Pursuit," SIAM Rev., vol. 43, no. 1, pp. 129-159, 2001. [20] E.J. Candés, J. Romberg, and T. Tao, "Stable Signal Recovery from Incomplete and Inaccurate Measurements," Comm. Pure and Applied Math, vol. 59, no. 8, pp. 1207-1223, 2006. [21] H. Zou, "The Adaptive Lasso and Its Oracle Properties," J. Am. Statistical Assoc., vol. 101, no. 476, pp. 1418-1429, 2006. [22] E.J. Candés, M. Wakin, and S. Boyd, "Enhancing Sparsity by Reweighted $\ell_1$ Minimization," J. Fourier Analysis and Applications, vol. 14, no. 5, pp. 877-905, 2008. [23] T. Zhang, "Multi-Stage Convex Relaxation for Learning with Sparse Regularization," Proc. Neural Information Processing Systems Conf., pp. 16-21, 2008. [24] W. Yin, S. Osher, D. Goldfarb, and J. Darbon, "Bregman Iterative Algorithms for $\ell_1$ -Minimization with Applications to Compressed Sensing," SIAM J. Imaging Sciences, vol. 1, no. 1, pp. 143-168, 2008. [25] P.L. Combettes and J.-C. Pesquet, "Proximal Thresholding Algorithm for Minimization over Orthonormal Bases," SIAM J. Optimization, vol. 18, no. 4, pp. 1531-1376, 2007. [26] R. Nowak and M. Figueiredo, "Fast Wavelet-Based Image Deconvolution Using the EM Algorithm," Proc. Asilomar Conf. Signals, Systems, and Computers, vol. 1, pp. 371-375, 2001. [27] D.L. Donoho and J. Tanner, "Observed Universality of Phase Transitions in High-Dimensional Geometry, with Implications for Modern Data Analysis and Signal Processing," Philosophical Trans. The Royal Soc. A, vol. 367, no. 1906, pp. 4273-4293, 2009. [28] F.D. la Torre and M. Black, "A Framework for Robust Subspace Learning," Int'l J. Computer Vision, vol. 54, nos. 1-3, pp. 117-142, 2003. [29] S. Fidler, D. Skocaj, and A. Leonardis, "Combining Reconstructive and Discriminative Subspace Methods for Robust Classification and Regression by Subsampling," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 3, pp. 337-350, Mar. 2006. [30] H. Jia and A.M. Martinez, "Support Vector Machines in Face Recognition with Occlusions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 136-141, 2009. [31] W.F. Liu, P.P. Pokharel, and J.C. Principe, "Correntropy: Properties and Applications in Non-Gaussian Signal Processing," IEEE Trans. Signal Processing, vol. 55, no. 11, pp. 5286-5298, Nov. 2007. [32] J. Idier, "Convex Half-Quadratic Criteria and Interacting Auxiliary Variables for Image Restoration," IEEE Trans. Image Processing, vol. 10, no. 7, pp. 1001-1009, July 2001. [33] M. Nikolova and M.K. NG, "Analysis of Half-Quadratic Minimization Methods for Signal and Image Recovery," SIAM J. Scientific Computing, vol. 27, no. 3, pp. 937-966, 2005. [34] X.-X. Li, D.-Q. Dai, X.-F. Zhang, and C.-X. Ren, "Structured Sparse Error Coding for Face Recognition with Occlusion," IEEE Trans. Image Processing, vol. 22, no. 5, pp. 1889-1900, May 2013. [35] N.H. Nguyen and T.D. Tran, "Robust Lasso with Missing and Grossly Corrupted Observations," IEEE Trans. Information Theory, vol. 59, no. 4, pp. 2036-2058, 2013. [36] D. Geman and G. Reynolds, "Constrained Restoration and Recovery of Discontinuities," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 3, pp. 367-383, Mar. 1992. [37] D. Geman and C. Yang, "Nonlinear Image Recovery with Half-Quadratic Regularization," IEEE Trans. Image Processing, vol. 4, no. 7, pp. 932-946, July 1995. [38] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge Univ. Press, 2004. [39] P.L. Combettes and V.R. Wajs, "Signal Recovery by Proximal Forwardbackward Splitting," SIAM J. Multiscale Modeling and Simulation, vol. 4, no. 5, pp. 1168-1200, 2005. [40] J. Bioucas-Dias and M. Figueiredo, "A New Twist: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration," IEEE Trans. Image Processing, vol. 16, no. 12, pp. 2992-3004, Dec. 2007. [41] X.T. Yuan and B.G. Hu, "Robust Feature Extraction via Information Theoretic Learning," Proc. Int'l Conf. Machine Learning, pp. 1193-1200, 2009. [42] M.J. Fadili and J.L. Starck, "Sparse Representation-Based Image Deconvolution by Iterative Thresholding," Astronomical Data Analysis ADA, 2006. [43] M. Figueiredo, R. Nowak, and S.J. Wright, "Gradient Projection for Sparse Reconstruction: Application to Compressed Sensing and other Inverse Problems," IEEE J. Selected Topics in Signal Processing, vol. 1, no. 4, pp. 586-597, Dec. 2007. [44] T. Ragheb, S. Kirolos, J. Laska, A. Gilbert, M. Strauss, R. Baraniuk, and Y. Massoud, "Implementation Models for Analog-to-Information Conversion via Random Sampling," Proc. Midwest Symp. Circuits and Systems, pp. 325-328, 2007. [45] Z. Zhang, "Parameter Estimation Techniques: A Tutorial with Application to Conic Fitting," Image and Vision Computing, vol. 15, no. 1, pp. 59-76, 1997. [46] J.C. Principe, D. Xu, and J.W. Fisher, "Information-Theoretic Learning," S. Haykin, ed., Unsupervised Adaptive Filtering, Vol. 1: Blind-Souurce Separation. Wiley, 2000. [47] A.M. Martinez and R. Benavente, "The AR Face Database," Computer Vision Center, technical report, 1998. [48] A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman, "From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp. 643-660, June 2001. [49] K.-C. Lee, J. Ho, and D. Kriegman, "Acquiring Linear Subspaces for Face Recognition under Variable Lighting," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 684-698, May 2005. [50] H. Lee, A. Battle, R. Raina, and A.Y. Ng, "Efficient Sparse Coding Algorithms," Proc. Neural Information Processing Systems Conf., vol. 19, pp. 801-808, 2006. [51] A. Beck and M. Teboulle, "A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems," SIAM J. Imaging Sciences, vol. 2, no. 1, pp. 183-202, 2009. [52] D.L. Donoho and Y. Tsaig, "Fast Solution of $l_1$ -Norm Minimization Problems When the Solution May Be Sparse," IEEE Trans. Information Theory, vol. 54, no. 11, pp. 4789-4812, Nov. 2008. [53] M. Plumbley, "Recovery of Sparse Representations by Polytope Faces Pursuit," Proc. Int'l Conf. Independent Component Analysis and Blind Source Separation, pp. 206-213, 2006. [54] I. Naseem, R. Togneri, and M. Bennamoun, "Linear Regression for Face Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 11, pp. 2106-2112, Nov. 2010. [55] L. Zhang, M. Yang, and X. Feng, "Sparse Representation or Collaborative Representation: Which Helps Face Recognition?" Proc. IEEE Int'l Conf. Computer Vision, 2011. [56] S. Wright, R. Nowak, and M. Figueiredo, "Sparse Reconstruction by Separable Approximation," Proc. IEEE Conf. Acoustics, Speech and Signal Processing, 2008. [57] D.L. Donoho and V. Stodden, "Breakdown Point of Model Selection When the Number of Variables Exceeds the Number of Observations," Proc. Int'l Joint Conf. Neural Networks, pp. 16-21, 2006. [58] A. Wagner, J. Wright, A. Ganesh, Z. Zhou, H. Mobahi, and Y. Ma, "Toward a Practical Face Recognition System: Robust Alignment and Illumination by Sparse Representation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 34, no. 2, pp. 372-386, Feb. 2012. [59] E.J. Candés, X. Li, Y. Ma, and J. Wright, "Robust Principal Component Analysis?" J. ACM, vol. 8, no. 58, pp. 1-37, 2010. [60] R. He, Z. Sun, T. Tan, and W.-S. Zheng, "Recovery of Corrupted Low-Rank Matrices via Half-Quadratic Based Nonconvex Minimization," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2889-2896, 2011. [61] R. Jenatton, G. Obozinski, and F. Bach, "Structured Sparse Principal Component Analysis," Proc. Artificial Intelligence and Statistics, pp. 366-373, 2010. [62] R. He, T. Tan, L. Wang, and W.-S. Zheng, "$\ell_{2,1}$ Regularized Correntropy for Robust Feature Selection," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2504-2511, 2012. [63] P. Hellier, C. Barillot, E. Memin, and P. Perez, "An Energy-Based Framework for Dense 3D Registration of Volumetric Brain Images," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000. [64] F. Champagnat and J. Idier, "A Connection between Half-Quadratic Criteria and EM Algorithms," IEEE Signal Processing Letters, vol. 11, no. 9, pp. 709-712, Sept. 2004. [65] M. Allain, J. Idier, and Y. Goussard, "On Global and Local Convergence of Half-Quadratic Algorithms," IEEE Trans. Image Processing, vol. 15, no. 5, pp. 1030-1042, May 2006. [66] X.T. Yuan and S. Li, "Half Quadratic Analysis for Mean Shift: With Extension to a Sequential Data Mode-Seeking Method," Proc. IEEE Int'l Conf. Computer Vision, 2007. [67] R. Chartrand, "Exact Reconstruction of Sparse Signals via Nonconvex Minimization," IEEE Signal Processing Letters, vol. 14, no. 10, pp. 707-710, Oct. 2007. [68] I. Daubechies, R. DeVore, M. Fornasier, and C. Gunturk, "Iteratively Re-Weighted Least Squares Minimization for Sparse Recovery," Comm. Pure and Applied Math., vol. 63, no. 1, pp. 1-38, 2010. [69] S. Seth and J.C. Principe, "Compressed Signal Reconstruction Using the Correntropy Induced Metric," Proc. IEEE Conf. Acoustics, Speech, and Signal Processing, pp. 3845-3848, 2008. [70] R. Chartrand and W. Yin, "Iteratively Reweighted Algorithms for Compressive Sensing," Proc. IEEE Conf. Acoustics, Speech, and Signal Processing, pp. 3869-3872, 2008. [71] G. Davis, S. Mallat, and M. Avellaneda, "Adaptive Greedy Approximations," J. Constructive Approximation, vol. 13, pp. 57-98, 1997. [72] I. Daubechies, M. Defrise, and C.D. Mol, "An Iterative Thresholding Algorithm for Linear Inverse Problems with a Sparsity Constraint," Comm. Pure and Applied Math., vol. 57, pp. 1413-1457, 2006. [73] M. Fornasier, "Theoretical Foundations and Numerical Methods for Sparse Recovery," Radon Series on Computational and Applied Math., vol. 9, pp. 1-121, 2010. [74] S.Z. Li, "Face Recognition Based on Nearest Linear Combinations," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 839-844, 1998. [75] Q. Shi, A. Eriksson, A. van den Hengel, and C. Shen, "Face Recognition Really a Compressive Sensing Problem," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 553-560, 2011. [76] X. Sun, Matrix Perturbation Analysis. Chinese Science Press, 2001.