Subscribe
Issue No.05 - May (2011 vol.33)
pp: 931-944
Mathieu Salzmann , Toyota Technological Institute at Chicago, Chicago
Pascal Fua , Ecole Polytechnique Fédérale (EPFL), Lausanane
ABSTRACT
Recovering the 3D shape of a nonrigid surface from a single viewpoint is known to be both ambiguous and challenging. Resolving the ambiguities typically requires prior knowledge about the most likely deformations that the surface may undergo. It often takes the form of a global deformation model that can be learned from training data. While effective, this approach suffers from the fact that a new model must be learned for each new surface, which means acquiring new training data, and may be impractical. In this paper, we replace the global models by linear local models for surface patches, which can be assembled to represent arbitrary surface shapes as long as they are made of the same material. Not only do they eliminate the need to retrain the model for different surface shapes, they also let us formulate 3D shape reconstruction from correspondences as either an algebraic problem that can be solved in closed form or a convex optimization problem whose solution can be found using standard numerical packages. We present quantitative results on synthetic data, as well as qualitative results on real images.
INDEX TERMS
Deformable surfaces, monocular shape recovery, deformation models.
CITATION
Mathieu Salzmann, Pascal Fua, "Linear Local Models for Monocular Reconstruction of Deformable Surfaces", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.33, no. 5, pp. 931-944, May 2011, doi:10.1109/TPAMI.2010.158
REFERENCES
 [1] H. Aanaes and F. Kahl, “Estimation of Deformable Structure and Motion,” Proc. Vision and Modelling of Dynamic Scenes Workshop, 2002. [2] A. Bartoli and S.I. Olsen, “A Batch Algorithm for Implicit Non-Rigid Shape and Motion Recovery,” Proc. IEEE Int'l Conf. Computer Vision Workshop Dynamical Vision, Oct. 2005. [3] K.S. Bhat, C.D. Twigg, J.K. Hodgins, P.K. Khosla, Z. Popovic, and S.M. Seitz, “Estimating Cloth Simulation Parameters from Video,” Proc. ACM Symp. Computer Animation, 2003. [4] V. Blanz and T. Vetter, “A Morphable Model for the Synthesis of 3D Faces,” Proc. ACM SIGGRAPH, pp. 187-194, Aug. 1999. [5] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge Univ. Press, 2004. [6] M. Brand, “Morphable 3d Models from Video,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2001. [7] C. Bregler, A. Hertzmann, and H. Biermann, “Recovering Non-Rigid 3D Shape from Image Streams,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2000. [8] L.D. Cohen and I. Cohen, “Finite-Element Methods for Active Contour Models and Balloons for 2D and 3D Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1131-1147, Nov. 1993. [9] T.F. Cootes, G.J. Edwards, and C.J. Taylor, “Active Appearance Models,” Proc. European Conf. Computer Vision, pp. 484-498, June 1998. [10] A. Ecker, A.D. Jepson, and K.N. Kutulakos, “Semidefinite Programming Heuristics for Surface Reconstruction Ambiguities,” Proc. European Conf. Computer Vision, Oct. 2008. [11] N.A. Gumerov, A. Zandifar, R. Duraiswami, and L.S. Davis, “Structure of Applicable Surfaces from Single Views,” Proc. European Conf. Computer Vision, May 2004. [12] G.E. Hinton, “Products of Experts,” Proc. Int'l Conf. Artificial Neural Networks, pp. 1-6, 1999. [13] N.D. Lawrence, “Gaussian Process Models for Visualisation of High Dimensional Data,” Neural Information Processing Systems, MIT Press, 2004. [14] J. Liang, D. DeMenthon, and D. Doermann, “Flattening Curved Documents in Images,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 338-345, 2005. [15] X. Llado, A. Del Bue, and L. Agapito, “Non-Rigid 3D Factorization for Projective Reconstruction,” Proc. British Machine Vision Conf., Sept. 2005. [16] D.G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int'l J. Computer Vision, vol. 20, no. 2, pp. 91-110, 2004. [17] I. Matthews and S. Baker, “Active Appearance Models Revisited,” Int'l J. Computer Vision, vol. 60, pp. 135-164, Nov. 2004. [18] T. McInerney and D. Terzopoulos, “A Finite Element Model for 3D Shape Reconstruction and Nonrigid Motion Tracking,” Proc. IEEE Int'l Conf. Computer Vision, pp. 518-523, 1993. [19] D. Metaxas and D. Terzopoulos, “Constrained Deformable Superquadrics and Nonrigid Motion Tracking,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 580-591, June 1993. [20] F. Moreno-Noguer, V. Lepetit, and P. Fua, “Accurate Non-Iterative $O(n)$ Solution to the P$n$ P Problem,” Proc. 11th IEEE Int'l Conf. Computer Vision, Oct. 2007. [21] C. Nastar and N. Ayache, “Frequency-Based Nonrigid Motion Analysis: Application to Four-Dimensional Medical Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 11, pp. 1067-1079, Nov. 1996. [22] A. Pentland and S. Sclaroff, “Closed-Form Solutions for Physically Based Shape Modeling and Recognition,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 7, pp. 715-729, July 1991. [23] M. Perriollat and A. Bartoli, “A Quasi-Minimal Model for Paper-Like Surfaces,” Proc. Towards Benchmarking Automated Calibration, Orientation, and Surface Reconstruction from Images (BenCOS) Workshop at IEEE Conf. Computer Vision and Pattern Recognition, 2007. [24] M. Perriollat, R. Hartley, and A. Bartoli, “Monocular Template-Based Reconstruction of Inextensible Surfaces,” Proc. British Machine Vision Conf., 2008. [25] M. Salzmann and P. Fua, “Reconstructing Sharply Folding Surfaces: A Convex Formulation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2009. [26] M. Salzmann, V. Lepetit, and P. Fua, “Deformable Surface Tracking Ambiguities,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2007. [27] M. Salzmann, F. Moreno-Noguer, V. Lepetit, and P. Fua, “Closed-Form Solution to Non-Rigid 3D Surface Registration,” Proc. European Conf. Computer Vision, Oct. 2008. [28] M. Salzmann, J. Pilet, S. Ilić, and P. Fua, “Surface Deformation Models for Non-Rigid 3-D Shape Recovery,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 8, pp. 1481-1487, Feb. 2007. [29] M. Salzmann, R. Urtasun, and P. Fua, “Local Deformation Models for Monocular 3D Shape Recovery,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2008. [30] J.F. Sturm, “Using SeDuMi 1.02, a MATLAB Toolbox for Optimization over Symmetric Cones,” 1999. [31] J. Taylor, A.D. Jepson, and K.N. Kutulakos, “Non-Rigid Structure from Locally Rigid Motion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2010. [32] D. Terzopoulos, J. Platt, A. Barr, and K. Fleicher, “Elastically Deformable Models,” Computer Graphics, vol. 21, no. 4, pp. 205-214, 1987. [33] L. Torresani, A. Hertzmann, and C. Bregler, “Learning Non-Rigid 3D Shape from 2D Motion,” Advances in Neural Information Processing Systems, MIT Press, 2003. [34] L. Torresani, A. Hertzmann, and C. Bregler, “Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 5, pp. 878-892, May 2008. [35] L.V. Tsap, D.B. Goldgof, and S. Sarkar, “Nonrigid Motion Analysis Based on Dynamic Refinement of Finite Element Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 5, pp. 526-543, May 2000. [36] R. Urtasun, D. Fleet, A. Hertzman, and P. Fua, “Priors for People Tracking from Small Training Sets,” Proc. 10th IEEE Int'l Conf. Computer Vision, Oct. 2005. [37] A. Varol, M. Salzmann, E. Tola, and P. Fua, “Template-Free Monocular Reconstruction of Deformable Surfaces,” Proc. IEEE Int'l Conf. Computer Vision, Oct. 2009. [38] R. Vidal and R. Hartley, “Perspective Nonrigid Shape and Motion Recovery,” Proc. European Conf. Computer Vision, Oct. 2008. [39] J. Xiao and T. Kanade, “Uncalibrated Perspective Reconstruction of Deformable Structures,” Proc. 10th IEEE Int'l Conf. Computer Vision, 2005.