Subscribe
Issue No.11 - November (2009 vol.31)
pp: 1968-1984
Yang Wang , Carnegie Mellon University, Pittsburgh
Lei Zhang , Stony Brook University, Stony Brook
Zicheng Liu , Microsoft Research, Redmond
Gang Hua , Microsoft Live Labs Research, Redmond
Zhen Wen , IBM T.J. Watson Research Center, Hawthorne
Zhengyou Zhang , Microsoft Research, Redmond
Dimitris Samaras , Stony Brook University, Stony Brook
ABSTRACT
In this paper, we present a new method to modify the appearance of a face image by manipulating the illumination condition, when the face geometry and albedo information is unknown. This problem is particularly difficult when there is only a single image of the subject available. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using a spherical harmonic representation. Moreover, morphable models are statistical ensembles of facial properties such as shape and texture. In this paper, we integrate spherical harmonics into the morphable model framework by proposing a 3D spherical harmonic basis morphable model (SHBMM). The proposed method can represent a face under arbitrary unknown lighting and pose simply by three low-dimensional vectors, i.e., shape parameters, spherical harmonic basis parameters, and illumination coefficients, which are called the SHBMM parameters. However, when the image was taken under an extreme lighting condition, the approximation error can be large, thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion-based framework that uses a Markov random field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to extreme lighting conditions, but also insensitive to partial occlusions. The performance of our framework is demonstrated through various experimental results, including the improved rates for face recognition under extreme lighting conditions.
INDEX TERMS
Face synthesis and recognition, Markov random field, 3D spherical harmonic basis morphable model, vision for graphics.
CITATION
Yang Wang, Lei Zhang, Zicheng Liu, Gang Hua, Zhen Wen, Zhengyou Zhang, Dimitris Samaras, "Face Relighting from a Single Image under Arbitrary Unknown Lighting Conditions", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.31, no. 11, pp. 1968-1984, November 2009, doi:10.1109/TPAMI.2008.244
REFERENCES
 [1] Y. Adini, Y. Moses, and S. Ullman, “Face Recognition: The Problem of Compensating for Changes in Illumination Derection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 721-732, July 1997. [2] R. Basri and D. Jacobs, “Lambertian Reflectance and Linear Subspaces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 2, pp. 218-233, 2003. [3] P. Belhumeur, J. Hespanha, and D. Kriegman, “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997. [4] P. Belhumeur and D. Kriegman, “What Is the Set of Images of an Object under All Possible Lighting Conditions,” Int'l J. Computer Vision, vol. 28, no. 3, pp. 245-260, 1998. [5] V. Blanz, S. Romdhani, and T. Vetter, “Face Identification across Different Poses and Illumination with a 3D Morphable Model,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp.202-207, 2002. [6] V. Blanz, K. Scherbaum, T. Vetter, and H. Seidel, “Exchanging Faces in Images,” Proc. EuroGraphics, 2004. [7] V. Blanz and T. Vetter, “A Morphable Model for the Synthesis of 3D Faces,” Proc. SIGGRAPH, pp. 187-194, 1999. [8] V. Blanz and T. Vetter, “Face Recognition Based on Fitting a 3D Morphable Model,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1063-1074, Sept. 2003. [9] B. Cabral, M. Olano, and P. Nemec, “Reflection Space Image Based Rendering,” Proc. SIGGRAPH, pp. 165-170, 1999. [10] R. Chellappa, C. Wilson, and S. Sirohey, “Human and Machine Recognition of Faces: A Survey,” Proc. IEEE, vol. 83, no. 5, pp. 705-740, May 1995. [11] T.F. Cootes, G.J. Edwards, and C.J. Taylor, “Active Appearance Models,” Proc. European Conf. Computer Vision, pp. 484-498, 1998. [12] P.E. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar, “Acquiring the Reflectance Field of a Human Face,” Proc. SIGGRAPH, pp. 145-156, 2000. [13] M. Dimitrijevic, S. Ilic, and P. Fua, “Accurate Face Models from Uncalibrated and Ill-Lit Video Sequences,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 1034-1041 2004. [14] R. Epstein, P. Hallinan, and A. Yullie, “5 $+/-$ 2 Eigenimages Suffice: An Empirical Investigation of Low Dimensional Lighting Models,” Proc. IEEE Workshop Physics Based Vision, pp. 108-116, 1995. [15] J. Foley and A.V. Dam, Fundamentals of Interactive Computer Graphics. Addison-Wesley, 1984. [16] W. Freeman, E. Pasztor, and O. Carmichael, “Learning Low-Level Vision,” Int'l J. Computer Vision, vol. 40, no. 1, pp. 25-47, 2000. [17] A. Georghiades, P. Belhumeur, and D. Kriegman, “From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp. 643-660, June 2001. [18] P. Hallinan, “A Low-Dimensional Representation of Human Faces for Arbitrary Lighting Conditions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 995-999, 1994. [19] E. Land and J. McCann, “Lightness and Retinex Theory,” J. Optical Soc. Am., vol. 61, no. 1, pp. 1-11, 1971. [20] J. Lee, B. Moghaddam, H. Pfister, and R. Machiraju, “A Bilinear Illumination Model for Robust Face Recognition,” Proc. Int'l Conf. Computer Vision, pp. 1177-1184, 2005. [21] K.-C. Lee, J. Ho, and D. Kriegman, “Nine Points of Light: Acquiring Subspaces for Face Recognition under Variable Lighting,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 357-362, 2001. [22] Z. Liu, Y. Shan, and Z. Zhang, “Expressive Expression Mapping with Ratio Images,” Proc. SIGGRAPH, pp. 271-276, 2001. [23] C. Loscos, G. Drettakis, and L. Robert, “Interactive Virtual Relighting of Real Scenes,” IEEE Trans. Visualization and Computer Graphics, vol. 6, no. 4, pp. 289-305, Oct.-Dec. 2000. [24] Q.-T. Luong, P. Fua, and Y.G. Leclerc, “Recovery of Reflectances and Varying Illuminants from Multiple Views,” Proc. European Conf. Computer Vision—Part III, pp. 163-179, 2002. [25] S.R. Marschner, B. Guenter, and S. Raghupathy, “Modeling and Rendering for Realistic Facial Animation,” Rendering Techniques, pp. 231-242, Springer, 2000. [26] S.R. Marschner, S. Westin, E. Lafortune, K. Torance, and D. Greenberg, “Image-Based brdf Measurement Including Human Skin,” Proc. Eurographics Workshop Rendering Techniques, 1999. [27] I. Matthews and S. Baker, “Active Appearance Models Revisited,” Int'l J. Computer Vision, vol. 60, no. 2, pp. 135-164, Nov. 2004. [28] R. Ramamoorthi and P. Hanrahan, “An Efficient Representation for Irradiance Environment Maps,” Proc. SIGGRAPH, pp. 497-500, 2001. [29] R. Ramamoorthi and P. Hanrahan, “A Signal-Processing Framework for Inverse Rendering,” Proc. SIGGRAPH, pp. 117-128, 2001. [30] T. Riklin-Raviv and A. Shashua, “The Quotient Image: Class Based Re-Rendering and Recongnition with Varying Illuminations,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 566-571, 1999. [31] S. Romdhani and T. Vetter, “Efficient, Robust and Accurate Fitting of a 3d Morphable Model,” Proc. Int'l Conf. Computer Vision, pp.59-66, 2003. [32] E. Sali and S. Ullman, “Recognizing Novel 3D Objects Under New Illumination and Viewing Position Using a Small Number of Examples,” Proc. Int'l Conf. Computer Vision, pp. 153-161, 1998. [33] D. Samaras, D. Metaxas, P. Fua, and Y. Leclerc, “Variable Albedo Surface Reconstruction from Stereo and Shape from Shading,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 480-487, 2000. [34] I. Sato, Y. Sato, and K. Ikeuchi, “Acquiring a Radiance Distribution to Superimpose Virtual Objects onto a Real Scene,” IEEE Trans. Visualization and Computer Graphics, vol. 5, no. 1, pp. 1-12, Jan.-Mar. 1999. [35] M. Savvides, B.V. Kumar, and P. Khosla, “‘Corefaces’—Robust Shift Invariant PCA Based Correlation Filter for Illumination Tolerant Face Recognition,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 834-841, 2004. [36] T. Sim, S. Baker, and M. Bsat, “The cmu Pose, Illumination, and Expression Database,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 12, pp. 1615-1618, Dec. 2003. [37] T. Sim and T. Kanade, “Combining Models and Exemplars for Face Recognition: An Illuminating Example,” Proc. Workshop Models versus Exemplars in Computer Vision, 2001. [38] A. Stoschek, “Image-Based Re-Rendering of Faces for Continuous Pose and Illumination Directions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 582-587, 2000. [39] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J.Cognitive Neuroscience, vol. 3, no. 1, pp. 71-96, 1991. [40] B.V. Kumar, M. Savvides, and C. Xie, “Correlation Pattern Recognition for Face Recognition,” Proc. IEEE, vol. 94, no. 11, pp. 1963-1976, Nov. 2006. [41] H. Wang, S. Li, Y. Wang, and J. Zhang, “Self Quotient Image for Face Recognition,” Proc. Int'l Conf. Image Processing, pp.1397-1400, 2004. [42] Y. Wang, Z. Liu, G. Hua, Z. Wen, Z. Zhang, and D. Samaras, “Face Re-Lighting from a Single Image under Harsh Lighting Conditions,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007. [43] Z. Wen, Z. Liu, and T.S. Huang, “Face Relighting with Radiance Environment Maps,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 158-165, 2003. [44] S. Yan, M. Li, H. Zhang, and Q. Cheng, “Ranking Prior Likelihood Distributions for Bayesian Shape Localization Framework,” Proc. Int'l Conf. Computer Vision, pp. 51-58, 2003. [45] L. Zhang and D. Samaras, “Pose Invariant Face Recognition under Arbitrary Unknown Lighting Using Spherical Harmonics,” Proc. Int'l Workshop Biometric Authentication, pp. 10-23, 2004. [46] L. Zhang and D. Samaras, “Face Recognition from a Single Training Image under Arbitrary Unknown Lighting Using Spherical Harmonics,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 3, pp. 351-363, Mar. 2006. [47] L. Zhang, S. Wang, and D. Samaras, “Face Synthesis and Recognition from a Single Image under Arbitrary Unknown Lighting Using a Spherical Harmonic Basis Morphable Model,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp.209-216, 2005. [48] W. Zhao, R. Chellappa, P.J. Phillips, and A. Rosenfeld, “Face Recognition: A Literature Survey,” ACM Computing Surveys, vol. 35, no. 4, pp. 399-458, 2003. [49] W. Zhao and R. Chellappa, “Illumination Insensitive Face Recognition Using Symmetric Shape-from-Shading,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 286-293, 2000. [50] S. Zhu, C. Guo, Y. Wang, and Z. Xu, “What Are Textons?” Int'l J. Computer Vision, vol. 62, nos. 1/2, pp. 121-143, 2005.