Subscribe
Issue No.07 - July (2011 vol.33)
pp: 1442-1456
Ijaz Akhter , Lahore University of Management Sciences, Lahore
Yaser Sheikh , Carnegie Mellon University, Pittsburgh
Sohaib Khan , Lahore University of Management Sciences, Lahore
Takeo Kanade , Carnegie Mellon University, Pittsburgh
ABSTRACT
Existing approaches to nonrigid structure from motion assume that the instantaneous 3D shape of a deforming object is a linear combination of basis shapes. These bases are object dependent and therefore have to be estimated anew for each video sequence. In contrast, we propose a dual approach to describe the evolving 3D structure in trajectory space by a linear combination of basis trajectories. We describe the dual relationship between the two approaches, showing that they both have equal power for representing 3D structure. We further show that the temporal smoothness in 3D trajectories alone can be used for recovering nonrigid structure from a moving camera. The principal advantage of expressing deforming 3D structure in trajectory space is that we can define an object independent basis. This results in a significant reduction in unknowns and corresponding stability in estimation. We propose the use of the Discrete Cosine Transform (DCT) as the object independent basis and empirically demonstrate that it approaches Principal Component Analysis (PCA) for natural motions. We report the performance of the proposed method, quantitatively using motion capture data, and qualitatively on several video sequences exhibiting nonrigid motions, including piecewise rigid motion, partially nonrigid motion (such as a facial expressions), and highly nonrigid motion (such as a person walking or dancing).
INDEX TERMS
Nonrigid structure from motion, 3D reconstruction, motion and tracking.
CITATION
Ijaz Akhter, Yaser Sheikh, Sohaib Khan, Takeo Kanade, "Trajectory Space: A Dual Representation for Nonrigid Structure from Motion", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.33, no. 7, pp. 1442-1456, July 2011, doi:10.1109/TPAMI.2010.201
REFERENCES
 [1] G. Johansson, "Visual Perception of Biological Motion and a Model for Its Analysis," Perception and Psychophysics, vol. 14, pp. 201-211, 1973. [2] C. Bregler, A. Hertzmann, and H. Biermann, "Recovering Non-Rigid 3D Shape from Image Streams," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 690-696, 2000. [3] A.K. Jain, Fundamentals of Digital Image Processing. Prentice Hall, 1989. [4] K. Rao and P. Yip, Discrete Cosine Transform: Algorithms, Advantages, Applications. Academic, 1990. [5] J. Xiao, J. Chai, and T. Kanade, "A Closed Form Solution to Non-Rigid Shape and Motion Recovery," Int'l J. Computer Vision, vol. 67, pp. 233-246, 2006. [6] L. Torresani, A. Hertzmann, and C. Bregler, "Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 5, pp. 878-892, May 2008. [7] L. Torresani, A. Hertzmann, and C. Bregler, "Learning Non-Rigid 3D Shape from 2D Motion," Proc. 19th Ann. Conf. Neural Information Processing Systems, 2005. [8] M. Brand, "A Direct Method for 3D Factorization of Nonrigid Motion Observed in 2D," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, 2005. [9] A.D. Bue, F. Smeraldi, and L. Agapito, "Non-Rigid Structure from Motion Using Ranklet-Based Tracking and Non-Linear Optimization," Image and Vision Computing, vol. 25, pp. 297-310, 2007. [10] L. Torresani, D. Yang, E. Alexander, and C. Bregler, "Tracking and Modeling Non-Rigid Objects with Rank Constraints," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 493-500, 2001. [11] M. Brand, "Morphable 3D Models from Video," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, 2001. [12] H. Longuet-Higgins, "A Computer Algorithm for Reconstructing a Scene from Two Projections," Nature, vol. 293, pp. 133-135, 1981. [13] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, second ed. Cambridge Univ. Press, 2004. [14] O. Faugeras and Q.T. Luong, The Geometry of Multiple Images. MIT Press, 2001. [15] Y. Ma, S. Soatto, J. Kosecka, and S.S. Sastry, An Invitation to 3-D Vision: From Images to Geomatric Models. Springer, 2003. [16] C. Tomasi and T. Kanade, "Shape and Motion from Image Streams under Orthography: A Factorization Method," Int'l J. Computer Vision, vol. 9, pp. 137-154, 1992. [17] L. Kontsevich, M. Kontsevich, and A. Shen, "Two Algorithms for Reconstructing Shapes," Optoelectronics, Instrumentation and Data Processing, vol. 5, pp. 76-81, 1987. [18] S. Ullman, "Maximizing Rigidity: The Incremental Recovery of 3-D Structure from Rigid and Rubbery Motion," Perception, vol. 13, pp. 255-274, 1984. [19] S. Chen, "Structure from Motion without the Rigidity Assumption," Proc. IEEE CS Workshop Computer Vision, pp. 105-112, 1985. [20] S. Chen Penna, "Shape and Motion of Non-Rigid Bodies," Computer Vision, Graphics, and Image Processing, vol. 36, pp. 175-207, 1986. [21] D. Shulman and J.Y. Aloimonos, "(Non-)Rigid Motion Interpretation: A Regularized Approach," Proc. Royal Soc. London Series B, vol. 233, pp. 217-234, 1988. [22] J. Costeira and T. Kanade, "A Multibody Factorization Method for Independently Moving Objects," Int'l J. Computer Vision, vol. 49, pp. 159-179, 1998. [23] L. Wolf and A. Shashua, "On Projection Matrices ${\cal P}^k\rightarrow {\cal P}^2$ , $k=3,\ldots,6$ , and Their Application in Computer Vision," Int'l J. Computer Vision, vol. 48, pp. 53-67, 2002. [24] M. Han and T. Kanade, "Reconstruction of a Scene with Multiple Linearly Moving Objects," Int'l J. Computer Vision, vol. 59, pp. 285-300, 2004. [25] A. Gruber and Y. Weiss, "Multibody Factorization with Uncertainty and Missing Data Using the EM Algorithm," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 707-714, 2004. [26] R. Vidal and R. Hartley, "Motion Segementaion with Missing Data Using PowerFactorization and GPCA," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 310-316, 2004. [27] A.D. Bue, X. Llado, and L. Agapito, "Non-Rigid Metric Shape and Motion Recovery from Uncalibrated Images Using Priors," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2006. [28] A.D. Bue, "A Factorization Approach to Structure from Motion with Shape Priors," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2008. [29] A. Bartoli, V. Gay-Bellile, U. Castellani, J. Peyras, S. Olsen, and P. Sayd, "Coarse-to-Fine Low-Rank Structure-from-Motion," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2008. [30] J. Yan and M. Pollefeys, "A Factorization-Based Approach to Articulated Motion Recovery," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 815-821, 2005. [31] J. Yan and M. Pollefeys, "Automatic Kinematic Chain Building from Feature Trajectories of Articulated Objects," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, 2006. [32] P. Tresadern and I. Reid, "Articulated Structure from Motion by Factorization," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2005. [33] M. Paladini, A.D. Bue, M. Stosic, M. Dodig, J. Xavier, and L. Agapito, "Factorization for Non-Rigid and Articulated Structure Using Metric Projections," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2009. [34] V. Rabaud and S. Belongie, "Linear Embeddings in Non-Rigid Structure from Motion," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2009. [35] I. Akhter, Y. Sheikh, and S. Khan, "In Defense of Orthonormality Constraints for Nonrigid Structure from Motion," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2009. [36] J. Xiao and T. Kanade, "Uncalibrated Perspective Reconstruction of Deformable Structures," Proc. 10th IEEE Int'l Conf. Computer Vision, vol. 2, pp. 1075-1082, 2005. [37] R. Vidal and D. Abretske, "Nonrigid Shape and Motion from Multiple Perspective Views," Proc. European Conf. Computer Vision, 2006. [38] R. Hartley and R. Vidal, "Perspective Nonrigid Shape and Motion Recovery," Proc. 10th European Conf. Computer Vision, 2008. [39] V. Rabaud and S. Belongie, "Rethinking Non-Rigid Structure from Motion," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2008. [40] I. Akhter, Y. Sheikh, S. Khan, and T. Kanade, "Nonrigid Structure from Motion in Trajectory Space," Proc. Neural Information Processing Systems, 2008. [41] S. Carlsson and D. Weinshall, "Dual Computation of Projective Shape and Camera Positions from Multiple Images," Int'l J. Computer Vision, vol. 27, no. 3, pp. 227-241, 1998. [42] A. Shashua, "Trilinear Tensor: The Fundamental Construct of Multiple-View Geometry and Its Applications," Proc. Int'l Workshop Algebraic Frames for the Perception-Action Cycle, 1997. [43] L. Zelnik-Manor and M. Irani, "Temporal Factorization vs. Spatial Factorization," Proc. Eighth European Conf. Computer Vision, 2004. [44] S. Olsen and A. Bartoli, "Implicit Non-Rigid Structure-from-Motion with Priors," J. Math. Imaging and Vision, vol. 31, nos. 2/3, pp. 233-244, 2008. [45] G. Golub and W. Kahan, "Calculating the Singular Values and Pseudo-Inverse of a Matrix," J. SIAM: Series B, Numerical Analysis, vol. 2, pp. 205-224, 1965. [46] R. Zelinski and P. Noll, "Adaptive Transfrom Coding of Speech Signals," IEEE Trans. Acoustics, Speech, and Signal Processing, vol. 25, no. 4, pp. 299-309, Aug. 1977. [47] J. Huang and Y. Zhao, "A DCT-Based Fast Signal Subspace Technique for Robust Speech Recognition," IEEE Trans. Speech and Audio Processing, vol. 8, no. 6, pp. 747-751, Nov. 2000. [48] L.A. Park, M. Palaniswami, and K. Ramamohanarao, "A Novel Document Ranking Method Using the Discrete Cosine Transform," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 1, pp. 130-135, Jan. 2005. [49] Z.M. Hafed and N.D. Levine, "Face Recognition Using the Discrete Cosine Transform," Int'l J. Computer Vision, vol. 43, no. 3, pp. 167-188, 2001. [50] H. Yan Li and T. Wang, "Motion Texture: A Two-Level Statistical Model for Character Motion Synthesis," ACM Trans. Graphics, vol. 21, no. 3, pp. 465-472, 2002. [51] Carnegie-Mellon Mocap Database, http:/mocap.cs.cmu.edu/, 2003. [52] J. Xiao and T. Kanade, "Non-Rigid Shape and Motion Recovery: Degenerate Deformations," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 668-675, 2004. [53] R. White, K. Crane, and D. Forsyth, "Capturing and Animating Occluded Cloth," Proc. ACM SIGGRAPH, 2007. [54] A. Datta, Y. Sheikh, and T. Kanade, "Linear Motion Estimation for Systems of Articulated Planes," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2008.
21 ms
(Ver 2.0)

Marketing Automation Platform