This Article 
 Bibliographic References 
 Add to: 
Dense Photometric Stereo: A Markov Random Field Approach
November 2006 (vol. 28 no. 11)
pp. 1830-1846
We address the problem of robust normal reconstruction by dense photometric stereo, in the presence of complex geometry, shadows, highlight, transparencies, variable attenuation in light intensities, and inaccurate estimation in light directions. The input is a dense set of noisy photometric images, conveniently captured by using a very simple set-up consisting of a digital video camera, a reflective mirror sphere, and a handheld spotlight. We formulate the dense photometric stereo problem as a Markov network and investigate two important inference algorithms for Markov Random Fields (MRFs)—graph cuts and belief propagation—to optimize for the most likely setting for each node in the network. In the graph cut algorithm, the MRF formulation is translated into one of energy minimization. A discontinuity-preserving metric is introduced as the compatibility function, which allows \alpha-expansion to efficiently perform the maximum a posteriori (MAP) estimation. Using the identical dense input and the same MRF formulation, our tensor belief propagation algorithm recovers faithful normal directions, preserves underlying discontinuities, improves the normal estimation from one of discrete to continuous, and drastically reduces the storage requirement and running time. Both algorithms produce comparable and very faithful normals for complex scenes. Although the discontinuity-preserving metric in graph cuts permits efficient inference of optimal discrete labels with a theoretical guarantee, our estimation algorithm using tensor belief propagation converges to comparable results, but runs faster because very compact messages are passed and combined. We present very encouraging results on normal reconstruction. A simple algorithm is proposed to reconstruct a surface from a normal map recovered by our method. With the reconstructed surface, an inverse process, known as relighting in computer graphics, is proposed to synthesize novel images of the given scene under user-specified light source and direction. The synthesis is made to run in real time by exploiting the state-of-the-art graphics processing unit (GPU). Our method offers many unique advantages over previous relighting methods and can handle a wide range of novel light sources and directions.

[1] E.H. Adelson and J.R. Bergen, “The Plenoptic Function and the Elements of Early Vision,” Computational Models of Visual Processing, chapter 1, pp. 3-20, 1991.
[2] D.H. Ballard and C.M. Brown, Computer Vision. Prentice Hall, 1982.
[3] S. Barsky and M. Petrou, “The 4-Source Photometric Stereo Technique for Three-Dimensional Surfaces in the Presence of Highlights and Shadows,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1239-1252, Oct. 2003.
[4] R. Basri and D.W. Jacobs, “Photometric Stereo with General, Unknown Lighting,” Proc. Conf. Computer Vision and Pattern Recognition, pp. II:374-381, 2001.
[5] P.N. Belhumeur and D.J. Kriegman, “What Is the Set of Images of an Object under All Possible Lighting Conditions?” Proc. Conf. Computer Vision and Pattern Recognition, pp. 270-277, 1996.
[6] Y. Boykov, O. Veksler, and R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[7] E.N. ColemanJr. and R. Jain, “Obtaining 3-Dimensional Shape of Textured and Specular Surfaces Using Four-Source Photometry,” Computer Graphics and Image Processing, vol. 18, no. 4, pp. 309-328, Apr. 1982.
[8] T.H. Cormen, C.E. Lieserson, and R.L. Rivest, Introduction to Algorithms. MIT Press, 1990.
[9] D.A. Forsyth and J. Ponce, Computer Vision: A Modern Approach. Prentice-Hall, 2003.
[10] D.B. Goldman, B. Curless, A. Hertzmann, and S.M. Seitz, “Shape and Spatially-Varying BRDFs from Photometric Stereo,” Proc. Int'l Conf. Computer Vision, Oct. 2005.
[11] P. Haeberli, “Synthetic Lighting for Photography,” , Jan. 1992.
[12] G. Hager and P. Belhumeur, “Real-Time Tracking of Image Regions with Changes in Geometry and Illumination,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 1996.
[13] A. Hertzmann and S.M. Seitz, “Shape and Materials by Example: A Photometric Stereo Approach,” Proc. Conf. Computer Vision and Pattern Recognition, pp. I:533-540, 2003.
[14] P.-M. Ho, T.-T. Wong, and C.-S. Leung, “Compressing the Illumination-Adjustable Images with Principal Component Analysis,” IEEE Trans. Circuits and Systems for Video Technology, vol. 15, no. 3, pp. 355-364, Mar. 2005.
[15] B.K.P. Horn, Robot Vision. McGraw-Hill, 1986.
[16] B.K.P. Horn, R.J. Woodham, and W.M. Silver, “Determining Shape and Reflectance Using Multiple Images,” MIT AI Memo, 1978.
[17] Y. Ju, K. Lee, and S.U. Lee, “Shape from Shading Using Graph Cuts,” Proc. Int'l Conf. Image Processing, pp. I:421-424, 2003.
[18] G. Kay and T. Caelly, “Estimating the Parameters of an Illumination Model Using Photometric Stereo,” Graphical Models and Image Processing, vol. 57, no. 5, pp. 365-388, 1995.
[19] V. Kolmogorov and R. Zabih, “Multi-Camera Scene Reconstruction via Graph Cuts,” Proc. European Conf. Computer Vision, vol. III, p. 82, 2002.
[20] V. Kolmogorov and R. Zabih, “What Energy Functions Can Be Minimized via Graph Cuts?” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 147-159, Feb. 2004.
[21] P. Kovesi, “Shapelets Correlated with Surface Normals Produce Surfaces,” Proc. Int'l Conf. Computer Vision, pp. 994-1001, 2005.
[22] K.M. Lee and C.C.J. Kuo, “Shape Reconstruction from Photometric Stereo,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 479-484, 1992.
[23] C.-S. Leung, T.-T. Wong, P.-M. Lam, and K.-H. Choy, “An RBF-Based Image Compression Method for Image-Based Rendering,” IEEE Trans. Image Processing, vol. 15, no. 4, pp. 1031-1041, Apr. 2006.
[24] Y. Li, H.Y. Shum, C.K. Tang, and R. Szeliski, “Stereo Reconstruction from Multiperspective Panoramas,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 45-62, Jan. 2004.
[25] T. Malzbender, D. Gelb, and H. Wolters, “Polynomial Texture Maps,” Proc. ACM SIGGRAPH Conf., pp. 519-528, 2001.
[26] G. Medioni, M.S. Lee, and C.K. Tang, A Computational Framework for Feature Extraction and Segmentation. Elsevier Science, 2000.
[27] S.K. Nayar and H. Murase, “Dimensionality of Illumination in Appearance Matching,” Proc. IEEE Int'l Conf. Robotics and Automation, pp. 1326-1332, Apr. 1996.
[28] S.K. Nayar, K. Ikeuchi, and T. Kanade, “Determining Shape and Reflectance of Hybrid Surfaces by Photometric Sampling,” IEEE Trans. Robotics and Automation, vol. 6, no. 4, pp. 418-431, 1990.
[29] R. Ng, R. Ramamoorthi, and P. Hanrahan, “All-Frequency Shadows Using Non-Linear Wavelet Lighting Approximation,” ACM Trans. Graphics, vol. 22, no. 3, pp. 376-381, 2003.
[30] J.S. Nimeroff, E. Simoncelli, and J. Dorsey, “Efficient Re-Rendering of Naturally Illuminated Environments,” Proc. Fifth Eurographics Workshop Rendering, pp. 359-373, June 1994.
[31] K. Nishino, Y. Sato, and K. Ikeuchi, “Eigen-Texture Method: Appearance Compression Based on 3D Model,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 618-624, June 1999.
[32] J. Pearl, Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
[33] N. Petrovic, I. Cohen, B.J. Frey, R. Koetter, and T.S. Huang, “Enforcing Integrability for Surface Reconstruction Algorithms Using Belief Propagation in Graphical Models,” Proc. Conf. Computer Vision and Pattern Recognition, vol. I, pp. 743-748, 2001.
[34] Y. Shan, Z. Liu, and Z. Zhang, “Image-Based Surface Detail Transfer,” Proc. Conf. Computer Vision and Pattern Recognition, vol. II, pp. 794-799, 2001.
[35] F. Solomon and K. Ikeuchi, “Extracting the Shape and Roughness of Specular Lobe Objects Using Four Light Photometric Stereo,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 4, pp. 449-454, Apr. 1996.
[36] J. Sun, N.N. Zheng, and H.Y. Shum, “Stereo Matching Using Belief Propagation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, July 2003.
[37] H.D. Tagare and R.J. P. deFigueiredo, “A Theory of Photometric Stereo for a Class of Diffuse Non-Lambertian Surfaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 2, pp.133-152, Feb. 1991.
[38] K.L. Tang, C.K. Tang, and T.T. Wong, “Dense Photometric Stereo Using Tensorial Belief Propagation,” Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 132-139, June 2005.
[39] M.F. Tappen and W.T. Freeman, “Comparison of Graph Cuts with Belief Propagation for Stereo, Using Identical MRF Parameters,” Proc. Int'l Conf. Computer Vision, pp. 900-907, 2003.
[40] M. Alex, O. Vasilescu, and D. Terzopoulos, “Tensortextures: Multilinear Image-Based Rendering,” ACM Trans. Graphics, vol. 23, no. 3, pp. 336-342, 2004.
[41] A. Wenger, A. Gardner, C. Tchou, J. Unger, T. Hawkins, and P.E. Debevec, “Performance Relighting and Reflectance Transformation with Time-Multiplexed Illumination,” ACM Trans. Graphics, vol. 24, no. 3, pp. 756-764, 2005.
[42] T.-T. Wong, C.-W. Fu, P.-A. Heng, and C.-S. Leung, “The Plenoptic Illumination Function,” IEEE Trans. Multimedia, vol. 4, no. 3, Sept. 2002.
[43] T.-T. Wong, P.-A. Heng, S.-H. Or, and W.-Y. Ng, “Image-Based Rendering with Controllable Illumination,” Proc. Eighth Eurographics Workshop Rendering, pp. 13-22, June 1997.
[44] R.J. Woodham, “Photometric Method for Determining Surface Orientation from Multiple Images,” Optical Eng., vol. 19, no. 1, pp.139-144, Jan. 1980.
[45] R.J. Woodham, “Gradient and Curvature from the Photometric-Stereo Method, Including Local Confidence Estimation,” J. Optical Soc. Am., vol. 11, no. 11, pp. 3050-3068, Nov. 1994.
[46] T.P. Wu and C.K. Tang, “Dense Photometric Stereo Using a Mirror Sphere and Graph Cut,” Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 140-147, June 2005.
[47] Z. Zhang, “Modeling Geometric Structure and Illumination Variation of a Scene from Real Images,” Proc. Int'l Conf. Computer Vision, Jan. 1998.

Index Terms:
Photometric stereo, Markov random fields, belief propagation, graph cuts, normal and surface reconstruction, robust inference, real-time relighting.
Tai-Pang Wu, Kam-Lun Tang, Chi-Keung Tang, Tien-Tsin Wong, "Dense Photometric Stereo: A Markov Random Field Approach," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1830-1846, Nov. 2006, doi:10.1109/TPAMI.2006.224
Usage of this product signifies your acceptance of the Terms of Use.