The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - October (2011 vol.33)
pp: 2104-2114
Gabriel J. Brostow , University College London, London
Carlos Hernández , Google Inc.
George Vogiatzis , Aston University, Birmingham
Björn Stenger , Toshiba Research Europe Ltd., Cambridge
Roberto Cipolla , University of Cambridge, Cambridge
ABSTRACT
We present an algorithm and the associated single-view capture methodology to acquire the detailed 3D shape, bends, and wrinkles of deforming surfaces. Moving 3D data has been difficult to obtain by methods that rely on known surface features, structured light, or silhouettes. Multispectral photometric stereo is an attractive alternative because it can recover a dense normal field from an untextured surface. We show how to capture such data, which in turn allows us to demonstrate the strengths and limitations of our simple frame-to-frame registration over time. Experiments were performed on monocular video sequences of untextured cloth and faces with and without white makeup. Subjects were filmed under spatially separated red, green, and blue lights. Our first finding is that the color photometric stereo setup is able to produce smoothly varying per-frame reconstructions with high detail. Second, when these 3D reconstructions are augmented with 2D tracking results, one can register both the surfaces and relax the homogenous-color restriction of the single-hue subject. Quantitative and qualitative experiments explore both the practicality and limitations of this simple multispectral capture system.
INDEX TERMS
Photometric stereo, multispectral, single view, video normals.
CITATION
Gabriel J. Brostow, Carlos Hernández, George Vogiatzis, Björn Stenger, Roberto Cipolla, "Video Normals from Colored Lights", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.33, no. 10, pp. 2104-2114, October 2011, doi:10.1109/TPAMI.2011.37
REFERENCES
[1] J. Pilet, V. Lepetit, and P. Fua, "Real-Time Non-Rigid Surface Detection," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2005.
[2] M. Salzmann, S. Ilic, and P. Fua, "Physically Valid Shape Parameterization for Monocular 3-D Deformable Surface Tracking," Proc. British Machine Vision Conf., 2005.
[3] V. Scholz, T. Stich, M. Keckeisen, M. Wacker, and M. Magnor, "Garment Motion Capture Using Color-Coded Patterns," Computer Graphics Forum, vol. 24, no. 3, pp. 439-448, Aug. 2005.
[4] R. White and D. Forsyth, "Retexturing Single Views Using Texture and Shading," Proc. European Conf. Computer Vision, pp. 70-81, 2006.
[5] R. White, K. Crane, and D. Forsyth, "Capturing and Animating Occluded Cloth," Proc. ACM SIGGRAPH, 2007.
[6] S. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, "A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 519-528, 2006.
[7] A. Hertzmann and S. Seitz, "Shape and Materials by Example: A Photometric Stereo Approach," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. I-533-I-540, 2003.
[8] C. Hernández, G. Vogiatzis, and R. Cipolla, "Multiview Photometric Stereo," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 3, pp. 548-554, Mar. 2008.
[9] M. Levoy, K. Pulli, B. Curless, S. Rusinkiewicz, D. Koller, L. Pereira, M. Ginzton, S. Anderson, J. Davis, J. Ginsberg, J. Shade, and D. Fulk, "The Digital Michelangelo Project: 3D Scanning of Large Statues," Proc. ACM SIGGRAPH, pp. 131-144, 2000.
[10] T. Malzbender, D.G.B. Wilburn, and B. Ambrisco, "Surface Enhancement Using Real-Time Photometric Stereo and Reflectance Transformation," Proc. European Symp. Rendering, 2006.
[11] E. de Aguiar, C. Stoll, C. Theobalt, N. Ahmed, H.-P. Seidel, and S. Thrun, "Performance Capture from Sparse Multi-View Video," ACM Trans. Graphics, vol. 27, no. 3, pp. 1-10, 2008.
[12] D. Vlasic, I. Baran, W. Matusik, and J. Popović, "Articulated Mesh Animation from Multi-View Silhouettes," ACM Trans. Graphics, vol. 27, no. 3, pp. 1-9, 2008.
[13] C. Hernández and G. Vogiatzis, "Self-Calibrating a Real-Time Monocular 3D Facial Capture System," Proc. Int'l Symp. 3DPVT, 2010.
[14] C. Hernández, G. Vogiatzis, G.J. Brostow, B. Stenger, and R. Cipolla, "Non-Rigid Photometric Stereo with Colored Lights," Proc. 11th IEEE Int'l Conf. Computer Vision, 2007.
[15] D. Pritchard and W. Heidrich, "Cloth Motion Capture," Computer Graphics Forum, vol. 22, no. 3, pp. 263-272, 2003.
[16] R. Woodham, "Photometric Method for Determining Surface Orientation from Multiple Images," Optical Eng., vol. 19, no. 1, pp. 139-144, 1980.
[17] J. Lim, J. Ho, M.-H. Yang, and D. Kriegman, "Passive Photometric Stereo from Motion," Proc. 10th IEEE Int'l Conf. Computer Vision, pp. 1635-1642, 2005.
[18] D.B. Goldman, B. Curless, A. Hertzmann, and S.M. Seitz, "Shape and Spatially-Varying BRDFs from Photometric Stereo," ICCV '05: Proc. 10th IEEE Int'l Conf. Computer Vision, vol. 1, pp. 341-348, 2005.
[19] A. Hertzmann and S. Seitz, "Shape Reconstruction with General, Varying BRDFs," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 8, pp. 1254-1264, Aug. 2005.
[20] W.-C. Ma, T. Hawkins, P. Peers, C.-F. Chabert, M. Weiss, and P. Debevec, "Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination," Proc. Eurographics Symp. Rendering, 2007.
[21] D. Vlasic, P. Peers, I. Baran, P. Debevec, J. Popović, S. Rusinkiewicz, and W. Matusik, "Dynamic Shape Capture Using Multi-View Photometric Stereo," ACM Trans. Graphics, vol. 28, no. 5, pp. 1-11, 2009.
[22] W.-C. Ma, A. Jones, J.-Y. Chiang, T. Hawkins, S. Frederiksen, P. Peers, M. Vukovic, M. Ouhyoung, and P. Debevec, "Facial Performance Synthesis Using Deformation-Driven Polynomial Displacement Maps," ACM Trans. Graphics, vol. 27, no. 5, pp. 1-10, 2008.
[23] C.A. Wilson, A. Ghosh, P. Peers, J.-Y. Chiang, J. Busch, and P. Debevec, "Temporal Upsampling of Performance Geometry Using Photometric Alignment," ACM Trans. Graphics, vol. 29, no. 2, pp. 1-11, 2010.
[24] A. Petrov, "Light, Color and Shape," Proc. Cognitive Processes and Their Simulation, pp. 350-358, 1987.
[25] L. Kontsevich, A. Petrov, and I. Vergelskaya, "Reconstruction of Shape from Shading in Color Images," J. Optical Soc. of Am. A, vol. 11, no. 3, pp. 1047-1052, 1994.
[26] M.S. Drew and L.L. Kontsevich, "Closed-Form Attitude Determination Under Spectrally Varying Illumination," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 985-990, 1994.
[27] R.J. Woodham, "Gradient and Curvature from the Photometric-Stereo Method, Including Local Confidence Estimation," J. Optical Soc. of Am. A, vol. 11, no. 11, pp. 3050-3068, 1994.
[28] M.K. Johnson and E.H. Adelson, "Retrographic Sensing for the Measurement of Surface Texture and Shape," Proc. Computer Vision and Pattern Recognition, pp. 1070-1077, 2009.
[29] K.S. Bhat, C.D. Twigg, J.K. Hodgins, P.K. Khosla, Z. Popović, and S.M. Seitz, "Estimating Cloth Simulation Parameters from Video," Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation, pp. 37-51, 2003.
[30] X. Gu, S. Zhang, P. Huang, L. Zhang, S.-T. Yau, and R. Martin, "Holoimages," Proc. ACM Symp. Solid and Physical Modeling, pp. 129-138, 2006.
[31] T. Weise, B. Leibe, and L.V. Gool, "Fast 3D Scanning with Automatic Motion Compensation," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[32] L. Zhang, N. Snavely, B. Curless, and S.M. Seitz, "Spacetime Faces: High-Resolution Capture for Modeling and Animation," Proc. ACM Ann. Conf. Computer Graphics, pp. 548-558, 2004.
[33] P. Sand, L. McMillan, and J. Popović, "Continuous Capture of Skin Deformation," ACM Trans. Graphics, vol. 22, no. 3, pp. 578-586, 2003.
[34] H. Li, B. Adams, L.J. Guibas, and M. Pauly, "Robust Single-View Geometry and Motion Reconstruction," ACM Trans. Graphics, vol. 28, no. 5, p. 175:1-175:10, 2009.
[35] D. Bradley, T. Popa, A. Sheffer, W. Heidrich, and T. Boubekeur, "Markerless Garment Capture," ACM Trans. Graphics, vol. 27, no. 3, pp. 1-9, 2008.
[36] J. Starck and A. Hilton, "Surface Capture for Performance-Based Animation," IEEE Computer Graphics and Applications, vol. 27, no. 3, pp. 21-31, May/June 2007.
[37] N. Ahmed, C. Theobalt, P. Dobre, H.-P. Seidel, and S. Thrun, "Robust Fusion of Dynamic Shape and Normal Capture for High-Quality Reconstruction of Time-Varying Geometry," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[38] Y. Pekelny and C. Gotsman, "Articulated Object Reconstruction and Markerless Motion Capture from Depth Video," Computer Graphics Forum, vol. 27, no. 2, pp. 399-408, Apr. 2008.
[39] W. Chang and M. Zwicker, "Automatic Registration for Articulated Shapes," Computer Graphics Forum, vol. 27, no. 5, pp. 1459-1468, 2008.
[40] A. Johnson and M. Hebert, "Using Spin Images for Efficient Object Recognition in Cluttered 3D Scenes," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 433-449, May 1999.
[41] A. Tagliasacchi, H. Zhang, and D. Cohen-Or, "Curve Skeleton Extraction from Incomplete Point Cloud," ACM Trans. Graphics, vol. 28, no. 3, 2009.
[42] Q. Zheng, A. Sharf, A. Tagliasacchi, B. Chen, H. Zhang, A. Sheffer, and D. Cohen-Or, "Consensus Skeleton for Non-Rigid Space-Time Registration," Computer Graphcis Forum, Special Issue of Eurographics, vol. 29, no. 2, pp. 635-644, 2010.
[43] A. Sharf, D.A. Alcantara, T. Lewiner, C. Greif, A. Sheffer, N. Amenta, and D. Cohen-Or, "Space-Time Surface Reconstruction Using Incompressible Flow," ACM Trans. Graphics, vol. 27, no. 5, pp. 1-10, 2008.
[44] N.J. Mitra, S. Flory, M. Ovsjanikov, N. Gelfand, L. Guibas, and H. Pottmann, "Dynamic Geometry Registration," Proc. Fifth Eurographics Symp. Geometry Processing Computer Graphics Forum, pp. 173-182, 2007.
[45] J. Süßmuth, M. Winter, and G. Greiner, "Reconstructing Animated Meshes from Time-Varying Point Clouds," Computer Graphics Forum, vol. 27, no. 5, pp. 1469-1476, 2008.
[46] M. Wand, B. Adams, M. Ovsjanikov, A. Berner, M. Bokeloh, P. Jenke, L. Guibas, H.-P. Seidel, and A. Schilling, "Efficient Reconstruction of Nonrigid Shape and Motion from Real-Time 3D Scanner Data," ACM Trans. Graphics, vol. 28, no. 2, p. 15, Apr. 2009.
[47] M.S. Drew, "Direct Solution of Orientation-from-Color Problem Using a Modification of Pentland's Light Source Direction Estimator," Computer Vision and Image Understanding, vol. 64, no. 2, pp. 286-299, 1996.
[48] J.A. Paterson, D. Claus, and A.W. Fitzgibbon, "BRDF and Geometry Capture from Extended Inhomogeneous Samples Using Flash Photography," Computer Graphics Forum, special Eurographics issue, vol. 24, no. 3, pp. 383-391, 2005.
[49] T. Simchony, R. Chellappa, and M. Shao, "Direct Analytical Methods for Solving Poisson Equations in Computer Vision Problems," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 5, pp. 435-446, May 1990.
[50] T. Beeler, B. Bickel, P. Beardsley, B. Sumner, and M. Gross, "High-Quality Single-Shot Capture of Facial Geometry," ACM Trans. Graphics, vol. 29, no. 3, 2010.
[51] Y. Furukawa and J. Ponce, "Dense 3D Motion Capture for Human Faces," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2009.
[52] D. Bradley, W. Heidrich, T. Popa, and A. Sheffer, "High Resolution Passive Facial Performance Capture," ACM Trans. Graphics, vol. 29, no. 3, 2010.
[53] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2004.
[54] Boujou, 2d3 Ltd., http:/www.2d3.com, 2009.
[55] S.N. Sinha, M. Pollefeys, and L. McMillan, "Camera Network Calibration from Dynamic Silhouettes," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 195-202, 2004.
[56] C. Hernández, F. Schmitt, and R. Cipolla, "Silhouette Coherence for Camera Calibration under Circular Motion," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 2, pp. 343-349, Feb. 2007.
[57] C. Hernández and F. Schmitt, "Silhouette and Stereo Fusion for 3D Object Modeling," Computer Vision and Image Understanding, special issue on model-based and image-based 3D scene rRepresentation for interactive visualization, vol. 96, no. 3, pp. 367-392, 2004.
[58] G. Vogiatzis, C. Hernández, P.H.S. Torr, and R. Cipolla, "Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 12, pp. 2241-2246, Dec. 2007.
[59] Y. Furukawa and J. Ponce, "Accurate, Dense, and Robust Multi-View Stereopsis," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[60] C. Hernández, G. Vogiatzis, and R. Cipolla, "Shadows in Three-Source Photometric Stereo," Proc. 10th European Conf. Computer Vision, pp. 290-303, 2008.
[61] B. Allen, B. Curless, and Z. Popović, "Articulated Body Deformation from Range Scan Data," Proc. ACM SIGGRAPH, pp. 612-619, 2002.
[62] M. Black and P. Anandan, "The Robust Estimation of Multiple Motions: Parametric and Piecewise Smooth Flow Fields," Computer Vision and Image Understanding, vol. 63, no. 1, pp. 75-104, 1996.
[63] N. Ahmed, C. Theobalt, C. Rossl, S. Thrun, and H. Seidel, "Dense Correspondence Finding for Parametrization-Free Animation Reconstruction from Video," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[64] M. Botsch and O. Sorkine, "On Linear Variational Surface Deformation Methods," IEEE Trans. Visualization and Computer Graphics, vol. 14, no. 1, pp. 213-230, Jan./Feb. 2008.
[65] P. Sinha and E.H. Adelson, "Recovering Reflectance and Illumination in a World of Painted Polyhedra," Proc. Fourth Int'l Conf. Computer Vision, pp. 156-163, 1993.
[66] R. Onn and A. Bruckstein, "Integrability Disambiguates Surface Recovery in Two-Image Photometric Stereo," Int'l J. Computer Vision, vol. 5, no. 1, pp. 105-113, 1990.
[67] R.T. Frankot and R. Chellappa, "A Method for Enforcing Integrability in Shape from Shading Algorithms," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 10, no. 4, pp. 439-451, July 1988.
[68] J.P. Lewis, M. Cordner, and N. Fong, "Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-Driven Deformation," Proc. ACM SIGGRAPH, pp. 165-172, 2000.
[69] X.C. Wang and C. Phillips, "Multi-Weight Enveloping: Least-Squares Approximation Techniques for Skin Animation," SCA '02: Proc. ACM SIGGRAPH/Eurographics Symp. Computer Animation, pp. 129-138, 2002.
[70] "CMU Graphics Lab Motion Capture Database," http:/ mocap.cs.cmu.edu, 2011.
[71] O. Mac Aodha, G.J. Brostow, and M. Pollefeys, "Segmenting Video into Classes of Algorithm-Suitability," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[72] T. Popa, I. South-Dickinson, D. Bradley, A. Sheffer, and W. Heidrich, "Globally Consistent Space-Time Reconstruction," Proc. Eurographics Symp. Geometry Processing, 2010.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool