The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - October (2011 vol.17)
pp: 1475-1486
Guofeng Zhang , Zhejiang University, Hangzhou
Jin Huang , Zhejiang University, Hangzhou
Jiaya Jia , The Chinese University of Hong Kong, Hong Kong
Tien-Tsin Wong , The Chinese University of Hong Kong, Hong Kong
Kun Zhou , Zhejiang University, Hangzhou
Hujun Bao , Zhejiang University, Hangzhou
ABSTRACT
In this paper, we present a novel method to extract motion of a dynamic object from a video that is captured by a handheld camera, and apply it to a 3D character. Unlike the motion capture techniques, neither special sensors/trackers nor a controllable environment is required. Our system significantly automates motion imitation which is traditionally conducted by professional animators via manual keyframing. Given the input video sequence, we track the dynamic reference object to obtain trajectories of both 2D and 3D tracking points. With them as constraints, we then transfer the motion to the target 3D character by solving an optimization problem to maintain the motion gradients. We also provide a user-friendly editing environment for users to fine tune the motion details. As casual videos can be used, our system, therefore, greatly increases the supply source of motion data. Examples of imitating various types of animal motion are shown.
INDEX TERMS
Motion imitation, motion gradient, mesh deformation, depth recovery, motion tracking.
CITATION
Guofeng Zhang, Jin Huang, Jiaya Jia, Tien-Tsin Wong, Kun Zhou, Hujun Bao, "Motion Imitation with a Handheld Camera", IEEE Transactions on Visualization & Computer Graphics, vol.17, no. 10, pp. 1475-1486, October 2011, doi:10.1109/TVCG.2010.254
REFERENCES
[1] D. Vlasic, R. Adelsberger, G. Vannucci, J. Barnwell, M.H. Gross, W. Matusik, and J. Popovic, “Practical Motion Capture in Everyday Surroundings,” ACM Trans. Graphics, vol. 26, no. 3, p. 35, 2007.
[2] M. Gleicher, “Retargeting Motion to New Characters,” Proc. SIGGRAPH, pp. 33-42, 1998.
[3] C. Hecker, B. Raabe, R.W. Enslow, J. DeWeese, J. Maynard, and K. van Prooijen, “Real-Time Motion Retargeting to Highly Varied User-Created Morphologies,” ACM Trans. Graphics, vol. 27, no. 3, 2008.
[4] R.I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, second ed. Cambridge Univ. Press, 2004.
[5] M. Pollefeys, L.J.V. Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, and R. Koch, “Visual Modeling with a Hand-Held Camera,” Int'l J. Computer Vision, vol. 59, no. 3, pp. 207-232, 2004.
[6] S.M. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, “A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 519-528, 2006.
[7] P. Tan, G. Zeng, J. Wang, S.B. Kang, and L. Quan, “Image-Based Tree Modeling,” ACM Trans. Graphics, vol. 26, no. 3, p. 87, 2007.
[8] A. van den Hengel, A.R. Dick, T. Thormählen, B. Ward, and P.H.S. Torr, “Videotrace: Rapid Interactive Scene Modelling from Video,” ACM Trans. Graphics, vol. 26, no. 3, p. 86, 2007.
[9] C.L. Zitnick, S.B. Kang, M. Uyttendaele, S.A.J. Winder, and R. Szeliski, “High-Quality Video View Interpolation Using a Layered Representation,” ACM Trans. Graphics, vol. 23, no. 3, pp. 600-608, 2004.
[10] Y. Furukawa and J. Ponce, “Dense 3D Motion Capture from Synchronized Video Streams,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008.
[11] D. Bradley, T. Popa, A. Sheffer, W. Heidrich, and T. Boubekeur, “Markerless Garment Capture,” ACM Trans. Graphics, vol. 27, no. 3, 2008.
[12] L. Torresani, A. Hertzmann, and C. Bregler, “Nonrigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 5, pp. 878-892, May 2008.
[13] V. Rabaud and S. Belongie, “Re-Thinking Non-Rigid Structure from Motion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008.
[14] M. Gleicher and N.J. Ferrier, “Evaluating Video-Based Motion Capture,” Proc. Computer Animation (CA), pp. 75-80, 2002.
[15] T.B. Moeslund, A. Hilton, and V. Krüger, “A Survey of Advances in Vision-Based Human Motion Capture and Analysis,” Computer Vision and Image Understanding, vol. 104, nos. 2/3, pp. 90-126, 2006.
[16] R. Poppe, “Vision-Based Human Motion Analysis: An Overview,” Computer Vision and Image Understanding, vol. 108, nos. 1/2, pp. 4-18, 2007.
[17] P. Sand, L. McMillan, and J. Popovic, “Continuous Capture of Skin Deformation,” ACM Trans. Graphics, vol. 22, no. 3, pp. 578-586, 2003.
[18] J.P. Lewis, M. Cordner, and N. Fong, “Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation,” Proc. SIGGRAPH, pp. 165-172, 2000.
[19] T. Ju, S. Schaefer, and J. Warren, “Mean Value Coordinates for Closed Triangular Meshes,” ACM Trans. Graphics, vol. 24, no. 3, pp. 561-566, 2005.
[20] D. Zorin, P. Schröder, and W. Sweldens, “Interactive Multiresolution Mesh Editing,” Proc. SIGGRAPH, pp. 259-268, 1997.
[21] S. Kircher and M. Garland, “Editing Arbitrarily Deforming Surface Animations,” ACM Trans. Graphics, vol. 25, no. 3, pp. 1098-1107, 2006.
[22] Y. Yu, K. Zhou, D. Xu, X. Shi, H. Bao, B. Guo, and H.-Y. Shum, “Mesh Editing with Poisson-Based Gradient Field Manipulation,” ACM Trans. Graphics, vol. 23, no. 3, pp. 644-651, 2004.
[23] Y. Lipman, O. Sorkine, D. Levin, and D. Cohen-Or, “Linear Rotation-Invariant Coordinates for Meshes,” ACM Trans. Graphics, vol. 24, no. 3, pp. 479-487, 2005.
[24] K. Zhou, J. Huang, J. Snyder, X. Liu, H. Bao, B. Guo, and H.-Y. Shum, “Large Mesh Deformation Using the Volumetric Graph Laplacian,” ACM Trans. Graphics, vol. 24, no. 3, pp. 496-503, 2005.
[25] A. Nealen, O. Sorkine, M. Alexa, and D. Cohen-Or, “A Sketch-Based Interface for Detail-Preserving Mesh Editing,” ACM Trans. Graphics, vol. 24, no. 3, pp. 1142-1147, 2005.
[26] J. Huang, X. Shi, X. Liu, K. Zhou, L.-Y. Wei, S.-H. Teng, H. Bao, B. Guo, and H.-Y. Shum, “Subspace Gradient Domain Mesh Deformation,” ACM Trans. Graphics, vol. 25, no. 3, pp. 1126-1134, 2006.
[27] O.K.-C. Au, C.-L. Tai, L. Liu, and H. Fu, “Dual Laplacian Editing for Meshes,” IEEE Trans. Visualization and Computer Graphics, vol. 12, no. 3, pp. 386-395, May/June 2006.
[28] W. Xu, K. Zhou, Y. Yu, Q. Tan, Q. Peng, and B. Guo, “Gradient Domain Editing of Deforming Mesh Sequences,” ACM Trans. Graphics, vol. 26, no. 3, p. 84, 2007.
[29] R.W. Sumner and J. Popovic, “Deformation Transfer for Triangle Meshes,” ACM Trans. Graphics, vol. 23, no. 3, pp. 399-405, 2004.
[30] C. Bregler, L. Loeb, E. Chuang, and H. Deshpande, “Turning to the Masters: Motion Capturing Cartoons,” Proc. SIGGRAPH, pp. 399-407, 2002.
[31] L. Favreau, L. Revéret, C. Depraz, and M.-P. Cani, “Animal Gaits from Video: Comparative Studies,” Graphical Models, vol. 68, no. 2, pp. 212-234, 2006.
[32] G. Zhang, X. Qin, W. Hua, T.-T. Wong, P.-A. Heng, and H. Bao, “Robust Metric Reconstruction from Challenging Video Sequences,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2007.
[33] G. Zhang, J. Jia, T.-T. Wong, and H. Bao, “Consistent Depth Maps Recovery from a Video Sequence,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 6, pp. 974-988, June 2009.
[34] N. Ray and S.T. Acton, “Motion Gradient Vector Flow: An External Force for Tracking Rolling Leukocytes with Shape and Size Constrained Active Contours,” IEEE Trans. Medical Imaging, vol. 23, no. 12, pp. 1466-1478, Dec. 2004.
[35] M. Alexa, “Differential Coordinates for Local Mesh Morphing and Deformation,” The Visual Computer, vol. 19, nos. 2/3, pp. 105-114, 2003.
[36] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hingorani, “Hierarchical Model-Based Motion Estimation,” Proc. European Conf. Computer Vision (ECCV), pp. 237-252, 1992.
[37] A. Buchanan and A.W. Fitzgibbon, “Interactive Feature Tracking Using K-D Trees and Dynamic Programming,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 626-633, 2006.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool