This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Bilinear Modeling via Augmented Lagrange Multipliers (BALM)
Aug. 2012 (vol. 34 no. 8)
pp. 1496-1508
L. Agapito, Sch. of Electron. Eng. & Comput. Sci. (EECS), Queen Mary Univ. of London, London, UK
J. Xavier, Inst. for Syst. & Robot. (ISR), Tech. Univ. of Lisbon, Lisbon, Portugal
A. Del Bue, Dept. of the Ist. Italiano di Tecnol., PAVIS, Genova, Italy
M. Paladini, Sch. of Electron. Eng. & Comput. Sci. (EECS), Queen Mary Univ. of London, London, UK
This paper presents a unified approach to solve different bilinear factorization problems in computer vision in the presence of missing data in the measurements. The problem is formulated as a constrained optimization where one of the factors must lie on a specific manifold. To achieve this, we introduce an equivalent reformulation of the bilinear factorization problem that decouples the core bilinear aspect from the manifold specificity. We then tackle the resulting constrained optimization problem via Augmented Lagrange Multipliers. The strength and the novelty of our approach is that this framework can seamlessly handle different computer vision problems. The algorithm is such that only a projector onto the manifold constraint is needed. We present experiments and results for some popular factorization problems in computer vision such as rigid, non-rigid, and articulated Structure from Motion, photometric stereo, and 2D-3D non-rigid registration.

[1] R. Basri, D. Jacobs, and I. Kemelmacher, "Photometric Stereo with General, Unknown Lighting," Int'l J. Computer Vision, vol. 72, no. 3, pp. 239-257, 2007.
[2] C. Tomasi and T. Kanade, "Shape and Motion from Image Streams under Orthography: A Factorization Approach," Int'l J. Computer Vision, vol. 9, no. 2, pp. 137-154, 1992.
[3] S. Thrun, "Affine Structure from Sound," Advances in Neural Information Processing Systems, vol. 18, p. 1353, 2006.
[4] B. Bascle and A. Blake, "Separability of Pose and Expression in Facial Tracking and Animation," Proc. Sixth IEEE Int'l Conf. Computer Vision, pp. 323-328, 1998.
[5] C. Bregler, A. Hertzmann, and H. Biermann, "Recovering Non-Rigid 3D Shape From Image Streams," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 690-696, June 2000.
[6] J. Tenenbaum and W. Freeman, "Separating Style and Content with Bilinear Models," Neural Computation, vol. 12, no. 6, pp. 1247-1283, 2000.
[7] M. Schlick, "Form and Content: An Introduction to Philosophical Thinking," Philosophical Papers, Springer, 1938.
[8] P. Tresadern and I. Reid, "Articulated Structure from Motion by Factorization," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 1110-1115, June 2005.
[9] R. Ferreira, J. Xavier, and J. Costeira, "Shape from Motion of Non-rigid Objects: The Case of Isometrically Deformable Flat Surfaces," Proc. 20th British Machine Vision Conf., 2009.
[10] M. Chandraker and D. Kriegman, "Globally Optimal Bilinear Programming for Computer Vision Applications," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[11] I. Akhter, Y. Sheikh, S. Khan, and T. Kanade, "Non-Rigid Structure from Motion in Trajectory Space," Proc. Neural Information Processing Systems, 2008.
[12] B. Abboud and F. Davoine, "Bilinear Factorisation for Facial Expression Analysis and Synthesis," IEE Proc. Vision, Image, and Signal Processing, vol. 152, no. 3, pp. 327-333, June 2005.
[13] J. Gonzalez-Mora, F. De la Torre, R. Murthi, N. Guil, and E. Zapata, "Bilinear Active Appearance Models," Proc. 11th IEEE Int'l Conf. Computer Vision, pp. 1-8, 2007.
[14] D. Shin, H. Lee, and D. Kim, "Illumination-Robust Face Recognition Using Ridge Regressive Bilinear Models," Pattern Recognition Letters, vol. 29, no. 1, pp. 49-58, 2008.
[15] E. Chuang and C. Bregler, "Mood Swings: Expressive Speech Animation," ACM Trans. Graphics, vol. 24, no. 2, pp. 331-347, 2005.
[16] F. Cuzzolin, "Using Bilinear Models for View-Invariant Action and Identity Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 1701-1708, 2006.
[17] A. Elgammal and C. Lee, "Separating Style and Content on a Nonlinear Manifold," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, 2004.
[18] F. Cole, "Automatic BRDF Factorization," PhD dissertation, Harvard College, Cambridge, Mass., 2002.
[19] T. Wiberg, "Computation of Principal Components When Data Are Missing," Proc. Second Symp. Computational Statistics, pp. 229-236, 1976.
[20] T. Okatani and K. Deguchi, "On the Wiberg Algorithm for Matrix Factorization in the Presence of Missing Components," Int'l J. Computer Vision, vol. 72, pp. 329-337, May 2007.
[21] A.M. Buchanan and A. Fitzgibbon, "Damped Newton Algorithms for Matrix Factorization with Missing Data," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 316-322, 2005.
[22] D. Jacobs, "Linear Fitting with Missing Data for Structure-from-Motion," Computer Vision and Image Understanding, vol. 82, no. 1, pp. 57-81, 2001.
[23] S. Olsen and A. Bartoli, "Using Priors for Improving Generalization in Non-Rigid Structure-from-Motion," Proc. British Machine Vision Conf., 2007.
[24] P. Chen and D. Suter, "Recovering the Missing Components in a Large Noisy Low-Rank Matrix: Application to SfM," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 8, pp. 1051-1063, Aug. 2004.
[25] R. Guerreiro and P. Aguiar, "Factorization with Missing Data for 3d Structure Recovery," Proc. IEEE Workshop Multimedia Signal Processing, pp. 105-108, Dec. 2002.
[26] R.I. Hartley and F. Schaffalitzky, "Powerfactorization: An Approach to Affine Reconstruction with Missing and Uncertain Data," Proc. Australia-Japan Advanced Workshop Computer Vision, Sept. 2003.
[27] L. Torresani, D. Yang, E. Alexander, and C. Bregler, "Tracking and Modeling Non-Rigid Objects with Rank Constraints," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2001.
[28] P. Chen, "Optimization Algorithms on Subspaces: Revisiting Missing Data Problem in Low-Rank Matrix," Int'l J. Computer Vision, vol. 80, no. 1, pp. 125-142, 2008.
[29] M. Marques and J. Costeira, "Estimating 3D Shape from Degenerate Sequences with Missing Data," Computer Vision and Image Understanding, vol. 113, pp. 261-272, 2009.
[30] M. Paladini, A. Del Bue, M. Stosic, M. Dodig, J. Xavier, and L. Agapito, "Optimal Metric Projections for Deformable and Articulated Structure-From-Motion," Int'l J. Computer Vision, vol. 96, no. 2, pp. 252-276, 2012, 10.1007/s11263-011-0468-5.
[31] A. Shaji, S. Chandran, and D. Suter, "Manifold Optimisation for Motion Factorisation," Proc. 19th Int'l Conf. Pattern Recognition, pp. 1-4, 2008.
[32] F. Mai and Y.S. Hung, "Augmented Lagrangian-Based Algorithm for Projective Reconstruction from Multiple Views with Minimization of 2D Reprojection Error," J. Signal Processing Systems, vol. 61, pp. 181-192, 2009.
[33] Z. Lin, M. Chen, L. Wu, and Y. Ma, "The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices," Technical Report UILU-ENG-09-2215, Univ. of Illinois Urbana-Champaign, 2009.
[34] M. Guignard, "Lagrangean Relaxation," TOP—Sociedad de Estadistica e Investigacion Operativa, vol. 11, no. 2, pp. 151-200, 2003.
[35] M. Hestenes, "Multiplier and Gradient Methods," J. Optimization Theory and Applications, vol. 4, no. 5, pp. 303-320, 1969.
[36] D. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods. Academic Press, 1982.
[37] A. Del Bue, J. Xavier, L. Agapito, and M. Paladini, "Bilinear Factorization via Augmented Lagrange Multipliers," Proc. 11th European Conf. Computer Vision, vol. 6314, pp. 283-296, 2010.
[38] J. Xavier, A. Del Bue, L. Agapito, and M. Paladini, "Convergence Analysis of BALM," technical report, http://users.isr.ist.utl.pt/ jxaviertechnical-report-balm.pdf , 2012.
[39] A. Conn, N. Gould, A. Sartenaer, and P. Toint, "Convergence Properties of an Augmented Lagrangian Algorithm for Optimization with a Combination of General Equality and Linear Constraints," SIAM J. Optimization, vol. 6, no. 3, pp. 674-703, 1996.
[40] J. Yan and M. Pollefeys, "A Factorization-Based Approach for Articulated Non-Rigid Shape, Motion, and Kinematic Chain Recovery from Video," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 5, pp. 865-877, May 2008.
[41] L. Torresani, A. Hertzmann, and C. Bregler, "Non-Rigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 5, pp. 878-892, May 2008.
[42] A. Del Bue, F. Smeraldi, and L. Agapito, "Non-Rigid Structure from Motion Using Ranklet-Based Tracking and Non-Linear Optimization," Image and Vision Computing, vol. 25, no. 3, pp. 297-310, Mar. 2007.
[43] G. Wang, H. Tsui, and Q. Wu, "Rotation Constrained Power Factorization for Structure from Motion of Nonrigid Objects," Pattern Recognition Letters, vol. 29, no. 1, pp. 72-80, 2008.
[44] M.A. Lourakis and A. Argyros, "SBA: A Software Package for Generic Sparse Bundle Adjustment," ACM Trans. Math. Software, vol. 36, no. 1, pp. 1-30, 2009.
[45] S. Agarwal, N. Snavely, S. Seitz, and R. Szeliski, "Bundle Adjustment in the Large," vol. 6314, pp. 29-42, 2010.
[46] A. Criminisi, P. Pérez, and K. Toyama, "Region Filling and Object Removal by Exemplar-Based Image Inpainting," IEEE Trans. Image Processing, vol. 13, no. 9, pp. 1200-1212, Sept. 2004.
[47] H. Aanæs, A. Dahl, and K. Steenstrup Pedersen, "Interesting Interest Points," Int'l J. Computer Vision, vol. 97, pp. 18-35, 2012.
[48] C. Julia, A. Sappa, F. Lumbreras, J. Serrat, and A. Lopez, "Recovery of Surface Normals and Reflectance from Different Lighting Conditions," Proc. Fifth Int'l Conf. Image Analysis and Recognition, pp. 315-325, 2008.

Index Terms:
stereo image processing,computer vision,image registration,optimisation,articulated structure,computer vision,bilinear modeling via augmented Lagrange multipliers,BALM,bilinear factorization problems,missing data,constrained optimization,equivalent reformulation,core bilinear aspect,manifold specificity,manifold constraint,2D-3D nonrigid registration,photometric stereo,rigid structure,nonrigid structure,Manifolds,Optimization,Shape,Three dimensional displays,Computer vision,Nickel,Cameras,image registration.,Bilinear optimization,augmented Lagrangian,SfM,photometric stereo
Citation:
L. Agapito, J. Xavier, A. Del Bue, M. Paladini, "Bilinear Modeling via Augmented Lagrange Multipliers (BALM)," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 8, pp. 1496-1508, Aug. 2012, doi:10.1109/TPAMI.2011.238
Usage of this product signifies your acceptance of the Terms of Use.