This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose
June 2001 (vol. 23 no. 6)
pp. 643-660

Abstract—We present a generative appearance-based method for recognizing human faces under variation in lighting and viewpoint. Our method exploits the fact that the set of images of an object in fixed pose, but under all possible illumination conditions, is a convex cone in the space of images. Using a small number of training images of each face taken with different lighting directions, the shape and albedo of the face can be reconstructed. In turn, this reconstruction serves as a generative model that can be used to render—or synthesize—images of the face under novel poses and illumination conditions. The pose space is then sampled and, for each pose, the corresponding illumination cone is approximated by a low-dimensional linear subspace whose basis vectors are estimated using the generative model. Our recognition algorithm assigns to a test image the identity of the closest approximated illumination cone (based on Euclidean distance within the image space). We test our face recognition method on 4,050 images from the Yale Face Database B; these images contain 405 viewing conditions (9 poses $\times$ 45 illumination conditions) for 10 individuals. The method performs almost without error, except on the most extreme lighting directions, and significantly outperforms popular recognition methods that do not use a generative model.

[1] P.N. Belhumeur and D.J. Kriegman, “What Is the Set of Images of an Object under All Possible Illumination Conditions?” Int'l J. Computer Vision, vol. 28, no. 3, pp. 245-260, 1998.
[2] P. Belhumeur, D. Kriegman, and A. Yuille, “The Bas-Relief Ambiguity,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 1,060–1,066, 1997.
[3] P.N. Belhumeur, J. Hespanda, and D. Kriegeman, Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[4] D. Beymer, "Face Recognition Under Varying Pose," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994, pp. 756-761.
[5] D. Beymer and T. Poggio, “Face Recognition from One Example View,” Proc. Int'l Conf. Computer Vision, pp. 500-507, 1995.
[6] R. Brunelli and T. Poggio, "Face Recognition: Features vs. Templates," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 10, pp. 1,042-1,053, Oct. 1993.
[7] R. Chellappa, C. Wilson, and S. Sirohey, "Human and Machine Recognition of Faces: A Survey," Proc. IEEE, vol. 83, no. 5, pp. 705-740, 1995.
[8] H. Chen, P. Belhumeur, and D. Jacobs, “In Search of Illumination Invariants,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 254-261, 2000.
[9] Q. Chen, H. Wu, and M. Yachida, "Face Detection by Fuzzy Pattern Matching," Int'l Conf. Computer Vision, 1995, pp. 591-596.
[10] T.F. Cootes, G.J. Edwards, and C.J. Taylor, “Active Appearance Models,” Proc. European Conf. Computer Vision, pp. 484-498, June 1998.
[11] T. Cootes, K. Walker, and C. Taylor, “View-Based Active Appearance Models,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 227-232, 2000.
[12] I.J. Cox, J. Ghosn, and P.N. Yianilos, Feature-Based Face Recognition Using Mixture-Distance Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 209-216, 1996.
[13] I. Craw, D. Tock, and A. Bennet, “Finding Face Features,” Proc. Int'l Conf. Computer Vision, pp. 92-96, 1992.
[14] G. Edwards, T. Cootes, and C. Taylor, “Advances in Active Appearance Models,” Proc. Int'l Conf. Computer Vision, pp. 137-142, 1999.
[15] R. Epstein, P. Hallinan, and A. Yuille, “5 Plus or Minus 2 Eigenimages Suffice: An Empirical Investigation of Low-Dimensional Lighting Models,” Proc. IEEE Workshop Physics-Based Modeling in Computer Vision, pp. 108-116, 1995.
[16] R.T. Frankot and R. Chellappa, “A Method for Enforcing Integrabilty in Shape from Shading Algorithms,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 10, no. 4, pp. 439-451, Apr. 1988.
[17] T. Fromherz, “Face Recognition: A Summary of 1995-1997,” Int'l Computer Science Inst. ICSI TR-98-027, Univ. of California, Berkeley, 1998.
[18] A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman, “Illumination-Based Image Synthesis: Creating Novel Images of Human Faces under Differing Pose and Lighting,” Proc. IEEE Workshop Multiple-View Modeling and Analysis of Visual Scenes, pp. 47-54, 1999.
[19] A.S. Georghiades, P.N. Belhumeur, and D.J. Kriegman, From Few to Many: Generative Models for Recognition under Variable Pose and Illumination Proc. Fourth IEEE Int'l Conf. Automatic Face and Gesture Recognition, 2000.
[20] A. Georghiades, D. Kriegman, and P. Belhumeur, “Illumination Cones for Recognition under Variable Lighting: Faces,” Proc. Computer Vision and Pattern Recognition Conf., pp. 52-59, 1998.
[21] A. Goldstein, L. Harmon, and A. Lesk, “Identification of Human Faces,” Proc. IEEE, vol. 59, no. 5, pp. 748-760, May 1971.
[22] V. Govindaraju, “Locating Human Faces in Photographs,” Int'l J. Computer Vision, vol. 19, no. 2 pp. 129-146, Aug. 1996.
[23] P. Hallinan, "A Low-Dimensional Representation of Human Faces for Arbitrary Lighting Conditions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994, pp. 995-999.
[24] P. Hallinan, "A Low-Dimensional Representation of Human Faces for Arbitrary Lighting Conditions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994, pp. 995-999.
[25] P. Hallinan, G. Gordon, A. Yuille, and D. Mumford, Two- and Three-Dimensional Patterns of the Face. A.K. Peters, 1999.
[26] L. Harmon, M. Kaun, R. Lasch, and P. Ramig, “Machine Identification of Human Faces,” Pattern Recognition, vol. 13, no. 2, pp. 97-110, 1981.
[27] L. Harmon, S. Kuo, P. Ramig, and U. Raudkivi, “Identification of Human Face Profiles by Computer,” Pattern Recognition, vol. 10, pp. 301-312, 1978.
[28] H. Hayakawa, “Photometric Stereo under a Light-Source with Arbitrary Motion,” J. Optical Soc. Am. A, vol. 11, no. 11, pp. 3079-3089, Nov. 1994.
[29] B. Horn, Computer Vision. Cambridge, Mass.: MIT Press, 1986.
[30] F. Huang, Z. Zhou, H. Zhang, and T. Chen, “Pose Invariant Face Recognition,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 245-250, 2000.
[31] D. Jacobs, "Linear Fitting with Missing Data," Proc. IEEE CVPR, 1997.
[32] P. Juell and R. Marsh, “A Hierarchical Neural-Network for Human Face Detection,” Pattern Recognition, vol. 29, no. 5, pp. 781-787, May 1996.
[33] T. Kanade, “Picture Processing by Computer Complex and Recognition of Human Faces,” PhD thesis, Kyoto Univ., 1973.
[34] T. Kanade, Computer Recognition of Human Faces. Stuttgart, Germany: Birkhauser Verlag, 1977.
[35] G. Kaufman and K. Breeding, “The Automatic Recognition of Human Faces from Profile Silhouettes,” IEEE Trans. Systems, Man, and Cybernetics, vol. 6, pp. 113-121, Feb. 1976.
[36] L. Sirovitch and M. Kirby, “Low-Dimensional Procedure for the Characterization of Human Faces,” J. Optical Soc. Am. A, vol. 2, pp. 519-524, 1987.
[37] J. Lambert, Photometria Sive de Mensura et Gradibus Luminus, Colorum et Umbrae. Eberhard Klett, 1760.
[38] A. Lanitis, C.J. Taylor, and T.F. Cootes, "A Unified Approach to Coding and Interpreting Face Images," Int'l Conf. Computer Vision, 1995, pp. 368-373.
[39] A. Lanitis, C.J. Taylor, and T.F. Cootes, “Automatic Interpretation and Coding of Face Images using Flexible Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 743-756, July 1997.
[40] C. Lee, J. Kim, and K. Park, “Automatic Human Face Location in a Complex Background Using Motion and Color Information,” Pattern Recognition, vol. 29, no. 11, pp. 1877-1889, Nov. 1996.
[41] T. Leung, M. Burl, and P. Perona, "Finding Faces in Cluttered Scenes Using Labeled Random Graph Matching," Int'l Conf. Computer Vision, 1995, pp. 637-644.
[42] S.Z. Li and J. Lu, “Face Recognition Using the Nearest Feature Line Method,” IEEE Trans. Neural Networks, vol. 10, no. 2, pp. 439-443, Mar. 1999.
[43] Y. Li, S. Gong, and H. Liddell, “Support Vector Regression and Classification Based Multi-View Face Detection and Recognition,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 300-305, 2000.
[44] B. Moghaddam and A. Pentland, "Probabilistic Visual Learning for Object Detection," Int'l Conf. Computer Vision, 1995, pp. 786-793.
[45] B. Moghaddam and A. Pentland, “Probabilistic Visual Learning for Object Representation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 696-710, July 1997.
[46] Y. Moses, Y. Adini, and S. Ullman, “Face Recognition: The Problem of Compensating for Changes in Illumination Direction,” Proc. Third European Conf. Computer Vision, pp. 286-296, J.-O. Eklundh, ed., May 1994.
[47] H. Murase and S.K. Nayar, “Visual Learning and Recognition of 3-D Objects from Appearance,” Int'l J. Computer Vision, vol. 14, pp. 5-24, 1995.
[48] S. Nayar and H. Murase, “Dimensionality of Illumination Manifolds in Appearance Matching,” Proc. Int'l Workshop Object Representations for Computer Vision, p. 165, 1996.
[49] A. O'Toole, T. Vetter, N. Toje, and H. Bulthoff, “Sex Classification is Better with Three-Dimensional Head Structure than with Texture,” Perception, vol. 26, pp. 75-84, 1997.
[50] A. Pentland, "Looking at People: Sensing for Ubiquitous and Wearable Computing," IEEE Trans. Pattern Analysis and Machine Intelligence, IEEE CS Press, Los Alamitos, Calif., Jan. 2000, pp. 107-118.
[51] A. Pentland, B. Moghaddam, and Starner, "View-Based and Modular Eigenspaces for Face Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994, pp. 84-91.
[52] P.J. Phillips, H. Moon, P. Rauss, and S.A. Rizvi, "The FERET Evaluation Methodology for Face-Recognition Algorithms," Computer Vision and Pattern Recognition, pp. 137-143, 1997.
[53] P. Phillips, H. Moon, P. Rauss, and S. Risvi, “The FERET September 1996 Database Evaluation Procedure,” Audio and Video-Based Biometric Person Authentication, 1997.
[54] P. Phillips, H. Wechsler, J. Huang, and P. Rauss, “The FERET Database and Evaluation Procedure for Face-Recognition Algorithms,” Image and Visual Computing, vol. 16, no. 5, 1998.
[55] T. Poggio and K. Sung, “Example-Based Learning for View-Based Human Face Detection,” Proc. Image Understanding Workshop, vol. II, pp. 843-850, 1994.
[56] T. Riklin-Raviv and A. Shashua, “The Quotient Image: Class Based Recognition and Synthesis under Varying Illumination,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 566-571, 1999.
[57] H. Rowley, S. Baluja, and T. Kanade, "Neural Network-Based Face Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 1, Jan. 1998, pp. 23-38.
[58] H.A. Rowley, S. Baluja, and T. Kanade, “Rotation Invariant Neural Network-Based Face Detection,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1998.
[59] A. Samal and P.A. Iyengar,“Automatic recognition and analysis of human faces and facial expressions: A survey,” Pattern Recognition, vol. 25, no. 1, pp. 65-77, 1992.
[60] A. Samil and P. Iyengar, “Human Face Detection Using Silhouettes,” Pattern Recognition and Artificial Intelligence, vol. 9, pp. 845-867, 1995.
[61] A. Shashua, “Geometry and Photometry in 3D Visual Recognition,” PhD dissertation, Dept. of Brain and Cognitive Sciences, Massachusetts Inst. of Technology, Cambridge, Nov. 1992.
[62] A. Shashua, “On Photometric Issues in 3D Visual Recognition from a Single 2D Image,” Int'l J. Computer Vision, vol. 21, nos. 1-2, pp. 99-122, Jan. 1997.
[63] H. Shum, K. Ikeuchi, and R. Reddy, Principal Component Analysis with Missing Data and Its Applications to Polyhedral Object Modeling IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 9, pp. 854-867, Sept. 1995.
[64] W. Silver, “Determining Shape and Reflectance Using Multiple Images,” PhD thesis, Massachusetts Inst. of Technology, Cambridge, 1980.
[65] C. Tomasi and T. Kanade, "Shape and Motion From Image Streams Under Orthography: A Factorization Method," Int'l J. Computer Vision, vol. 9, no. 2, pp. 137-154, 1992.
[66] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-96, 1991.
[67] T. Vetter, “Synthesis of Novel Views from a Single Face Image,” Int'l J. Computer Vision, vol. 28, no. 2, pp. 103-116, June 1998.
[68] T. Vetter and T. Poggio, "Linear Object Classes and Image Synthesis from Single Example Image," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 733-741, July 1997.
[69] L. Wiskott, J.M. Fellous, N. Kruger, and C. von der Malsburg, Face Recognition by Elastic Bunch Graph Matching IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 775-779, July 1997.
[70] R. Woodham, “Analysing Images of Curved Surfaces,” Artificial Intelligence, vol. 17, pp. 117-140, 1981.
[71] M. Yang, N. Ahuja, and D. Kriegman, “Mixture of Linear Subspaces for Face Detection,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 196-201, 2000.
[72] K. Yow and R. Cipolla, “Feature-Based Human Face Detection,” Image Visual Computing, vol. 15, no. 9, pp. 713-735, Sept. 1997.
[73] Y. Yu and J. Malik, “Recovering Photometric Properties of Architectural Scenes from Photographs,” Proc. SIGGRAPH '98, pp. 207-217, July 1998.
[74] A. Yuille and D. Snow, “Shape and Albedo from Multiple Images Using Integrability,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 158-164, 1997.
[75] W. Zhao and R. Chellapa, “SFS Based View Synthesis for Robust Face Recognition,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 285-292, 2000.
[76] W. Zhao, R. Chellappa, and P. Phillips, “Subspace Linear Discriminant Analysis for Face Recognition,” Center for Automation Research CAR-TR-914, Univ. of Maryland, College Park, 1999.

Index Terms:
Face recognition, image-based rendering, appearance-based vision, face modeling, illumination and pose modeling, lighting, illumination cones, generative models.
Citation:
Athinodoros S. Georghiades, Peter N. Belhumeur, David J. Kriegman, "From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 6, pp. 643-660, June 2001, doi:10.1109/34.927464
Usage of this product signifies your acceptance of the Terms of Use.