This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class
June 2002 (vol. 24 no. 6)
pp. 748-763

The classical way of attempting to solve the face (or object) recognition problem is by using large and representative data sets. In many applications, though, only one sample per class is available to the system. In this contribution, we describe a probabilistic approach that is able to compensate for imprecisely localized, partially occluded, and expression-variant faces even when only one single training sample per class is available to the system. To solve the localization problem, we find the subspace (within the feature space, e.g., eigenspace) that represents this error for each of the training images. To resolve the occlusion problem, each face is divided into k local regions which are analyzed in isolation. In contrast with other approaches where a simple voting space is used, we present a probabilistic method that analyzes how "good" a local match is. To make the recognition system less sensitive to the differences between the facial expression displayed on the training and the testing images, we weight the results obtained on each local area on the basis of how much of this local area is affected by the expression displayed on the current test image.

[1] Y. Adini, Y. Moses, and S. Ullman, “Face Recognition: The Problem of Compensating for Changes in Illumination Direction,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 721-732, July 1997.
[2] A. Barron, J. Rissanen, and B. Yu, “The Minimum Description Length Principle in Coding and Modeling,” IEEE Trans. Information Theory, vol. 44, no. 6, pp. 2,743-2,760, Oct. 1998.
[3] P.N. Belhumeur, J. Hespanda, and D. Kriegeman, Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 711-720, July 1997.
[4] P.N. Belhumeur and D.J. Kriegman, “What Is the Set of Images of an Object under All Possible Illumination Conditions?” Int'l J. Computer Vision, vol. 28, no. 3, pp. 245-260, 1998.
[5] D. Beymer and T. Poggio, “Face Recognition from One Example View,” Science, vol. 272, no. 5250, 1996.
[6] M.J. Black, D.J. Fleet, and Y. Yacoob, “Robustly Estimating Changes in Image Appearance,” Computer Vision and Image Understanding, vol. 78, no. 1, pp. 8–31, 2000.
[7] R. Brunelli and T. Poggio, "Face Recognition: Features vs. Templates," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 10, pp. 1,042-1,053, Oct. 1993.
[8] R. Brunelli and D. Falavigna,"Person Identification Using Multiple Cues," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 10, 1995, pp. 955-966.
[9] R. Brunelli, “Estimation of Pose and Illumination Direction for Face Processing,” Image and Vision Computing, vol. 15, pp. 741-748, 1997.
[10] R. Brunelli and T. Poggio, “Template Matching: Matched Spatial Filters and Beyond,” Pattern Recognition, vol. 30, no. 5, pp. 751-768, 1997.
[11] R. Campbell, “The Lateralization of Emotion: A Critical Review,” Int'l J. Psychology, vol. 17, pp. 211-219, 1982.
[12] H. Chen, P. Belhumeur, and D. Jacobs, “In Search of Illumination Invariants,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 254-261, 2000.
[13] D. Clemens and D. Jacobs, "Space and time bounds on model indexing," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 10, pp. 1,007-1,018, 1991.
[14] I. Craw and P. Cameron, “Face Recognition by Computers,” Proc. British Machine Vision Conf., pp. 498-507, 1992.
[15] I. Craw et al., "How Should We Represent Faces for Automatic Recognition?," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 8, Aug. 1999, pp. 725-736.
[16] C. Darwin, The Expression of the Emotions in Man and Animals. London: John Murray, 1872. Reprinted by Univ. of Chicago Press, 1965.
[17] A.P. Dempster, N.M. Laird, and D.B. Rubin, “Maximum Likelihood from Incomplete Data via the EM Algorithm,” J. Royal Statistical Soc., vol. 30, no. 1, pp. 1-38, 1977.
[18] G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, and T.J. Sejnowski, “Classifying Facial Actions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, Oct. 1999.
[19] T. Evgeniou, M. Pontil, and T. Poggio, “Regularization Networks and Support Vector Machines,” Advances in Computational Math., vol. 13, no. 1, pp. 1-50, 2000.
[20] P. Ekman and W. Friesen, Facial Action Coding System: A Technique for the Measurements of Facial Movements. Consulting Psychologists Press, 1978.
[21] K. Fukunaga, Introduction to Statistical Pattern Recognition, second edition. Academic Press, 1990.
[22] J. Hager, “Asymmetries in Facial Expression,” Emotion in the Human Face, P. Ekman, ed. pp. 318-352, Cambridge Univ. Press, 1982.
[23] C.-Y. Huang, O. Camps, and T. Kanungo, "Object Recognition Using Appearance-Based Parts and Relations," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 877-883, May 1997.
[24] A.K. Jain and B. Chandrasekaran, “Dimensionality and Sample Size Considerations in Pattern Recognition Practice,” Handbook of Statistics, P.R. Krishnaiah and L.N. Kanal, eds., vol. 2, pp. 835-855, 1982.
[25] A.K. Jain, Y. Zhong, and S. Lakshmanan, Object Matching Using Deformable Templates IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 3, pp. 267-278, Mar. 1996.
[26] A.K. Jain, R.P.W. Duin, and J. Mao, Statistical Pattern Recognition: A Review IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[27] R.A. Jacobs, M.I. Jordan, and G.E. Hinton, “Adaptative Mixture of Local Experts,” Neural Computer, vol. 3, pp. 79-87, 1991.
[28] M. Kirby and L. Sirovich,“Application of Karhunen-Loève procedure for the characterization of human faces,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 1, pp. 103-108, Jan. 1990.
[29] B. Kolb and L. Taylor, “Facial Expression, Emotion, and Hemispheric Organization,” Cognitive Neuroscience of Emotion, R.D. Lane and L. Nadel, eds., pp. 62-83, Oxford Univ. Press, 2000.
[30] M. Lades, J.C. Vorbruggen, J. Buhmann, J. Lange, C. von der Malsburg, R.P. Wurtz, and W. Konen, “Distortion Invariant Object Recognition in the Dynamic Link Architecture,” IEEE Trans. Computers, vol. 42, no. 3, pp. 300-311, Mar. 1993.
[31] A. Lanitis, C.J. Taylor, and T.F. Cootes, “Automatic Interpretation and Coding of Face Images using Flexible Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 743-756, July 1997.
[32] M.J. Lyons, J. Budynek, and S. Akamatsu, “Automatic Classification of Single Facial Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 12, pp. 1357-1362, 1999.
[33] M.J. Lyons, R. Campbell, A. Plante, M. Coleman, M. Kamachi, and S. Akamatsu, “The Noh Mask Effect: Vertical Viewpoint Dependence of Facial Expression Perception,” The Royal Soc. Proc.: Biological Sciences, 2000.
[34] A.M. Martínez and R. Benavente, “The AR Face Database,” CVC Technical Report no. 24, June 1998.
[35] A.M. Martinez and A.C. Kak, PCA versus LDA IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 228-233, Feb. 2001.
[36] A. M. Martínez, “Recognition of Partially Occluded and/or Imprecisely Localized Faces Using a Probabilistic Approach,” Proc. Computer Vision and Pattern Recognition, vol. 1, pp. 712-717, June 2000.
[37] A.M. Martínez, “Semantic Access of Frontal Face Images: The Expression-Invariant Problem,” Proc. IEEE Content Based Access of Images and Video Libraries, pp. 55-59, June 2000.
[38] A.M. Martínez and J. Vitrià, “Learning Mixture Models Using a Genetic Version of the EM Algorithm,” Pattern Recognition Letters, vol. 21, pp. 759-769, 2000.
[39] G. McLachlan and K. Basford, Mixture Models: Inference and Applications to Clustering. Marcel Dekker, 1988.
[40] G. McLachlan and T. Krishnan, The EM Algorithm and Extensions. Wiley, 1997.
[41] S.K. Mishra and V.V. Raghavan, “An Empirical Study of the Performance of Heuristic Methods for Clustering,” Pattern Recognition in Practice, E.S. Gelsema and L.N. Kanal, eds., pp. 425-436, North Holland, 1994.
[42] T. Mitchell, Machine Learning, McGraw-Hill, 1997.
[43] B. Moghaddam and A. Pentland, “Probabilistic Visual Learning for Object Representation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 696-710, July 1997.
[44] B. Moghaddam, W. Wahid, and A. Pentland, “Beyond Eigenfaces: Probabalistic Matching for Face Recognition,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 30-35, Nara, Japan, Apr. 1998.
[45] M. Moscovitch and J. Olds, “Asymmetries in Spontaneous Facial Expressions and Their Possible Relation to Their Hemispheric Specialization,” Neuropsychologia, vol. 20, pp. 71-81, 1982.
[46] K. Ohba and K. Ikeuchi, “Detectability, Uniqueness, and Reliability of Eigen-Windows for Stable Verification of Partially Occluded Objects,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 9, pp. 1,043-1,048, Sept. 1997.
[47] M. Pantic and L.J.M. Rothkrantz, “Automatic Analysis of Facial Expressions: The State of the Art,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424-1445, Dec. 1996.
[48] P.S. Penev and J.J. Atick, “Local Feature Analysis: A General Statistical Theory for Object Representation,” Network: Computation in Neural Systems, vol. 7, no. 3, pp. 477-500, 1996.
[49] P.S. Penev and L. Sirovich, “The Global Dimensionality of Face Space,” Proc. IEEE Face and Gesture Recognition, pp. 264-270, 2000.
[50] A. Pentland, T. Starner, N. Etcoff, N. Masoiu, O. Oliyide, and M. Turk, “Experiments with Eigenfaces,” Proc. Workshop Int'l Joint Conf. Artificial Intelligence, Looking at People, 1993.
[51] P.J. Phillips, H. Moon, P. Rauss, and S.A. Rizvi, “The FERET Evaluation Methodology for Face-Recognition Algorithms,” Proc. First Int'l Conf. Audio and Video-Based Biometric Person Authentification, 1997.
[52] S. Richardson and P. Green, “On Bayesian Analysis of Mixtures with Unknown Numbers of Components,” J. Royal Statistics Soc. B, vol. 59, pp. 731-792, 1997.
[53] K. Rose, “Deterministic Annealing for Clustering, Compression, Classification, Regression and Related Optimization Problems,” Proc. IEEE, vol. 86, pp. 2,210-2,239, 1998.
[54] H. Rowley, S. Baluja, and T. Kanade, "Neural Network-Based Face Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 1, Jan. 1998, pp. 23-38.
[55] H. Sackheim, R.C. Gur, and M.C. Saucy, “Emotions Are Expressed More Intensively on the Left Side of the Face,” Science, vol. 202, pp. 434-436, 1978.
[56] P.Y. Simard, Y. LeCun, and J. Denker, "Efficient Pattern Recognition Using a New Transformation Distance," Advances in Neural Information Processing Systems, pp. 50-58.San Mateo, Calif.: Morgan Kaufman, 1993.
[57] P. Simard, B. Victorri, Y. LeCun, and J. Denker, “Tangent Prop–A Formalism for Specifying Selected Invariances in an Adaptative Network,” Advances in Neural Information Processing Systems 4, J.E. Moody, S.J. Hanson, and R.P. Lippmann, eds. pp. 651-655, Morgan Kaufmann, 1992.
[58] L. Sirovich and M. Kirby, “Low-Dimensional Procedure for the Characterization of Human Faces,” J. Optical Soc. Am. A, vol. 4, pp. 519-524, 1987.
[59] K.K. Sung and T. Poggio, "Example-Based Learning for View-Based Human Face Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 1, pp. 39-50, Jan. 1998.
[60] M.J. Swain and B.H. Ballard, “Color Indexing,” Int'l J. Computer Vision, vol. 7, no. 1, pp. 11-32, 1991.
[61] Y.-I. Tian, T. Kanade, and J.F. Cohn, “Recognizing Action Units for Facial Expression Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 97-115, Feb. 2001.
[62] G.V. Trunk, “A Problem of Dimensionality: A Simple Example,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 1, no. 3, pp. 306-307, 1979.
[63] M. Turk and A. Pentland, “Eigenfaces for Recognition,” J. Cognitive Neuroscience, vol. 3, no. 1, pp. 71-86, 1991.
[64] S. Ullman, High-Level Vision: Object Recognition and Visual Cognition. MIT Press, 1996.
[65] V.N. Vapnik, Statistical Learning Theory, John Wiley&Sons, 1998.
[66] T. Vetter and T. Poggio, "Linear Object Classes and Image Synthesis from Single Example Image," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 733-741, July 1997.
[67] T. Vetter and N.F. Troje, “Separation of Texture and Shape in Images of Faces for Image Coding and Synthesis,” J. Optical Soc. Am. A, vol. 14, no. 9, pp. 2152-2161, 1997.
[68] M. Whindham and A. Cutler, “Information Ratios for Validating Mixture Analysis,” Int'l J. Computer Vision, vol. 28, no. 2, pp. 103-116, 1998.
[69] L. Wiskott and C. von der Malsburg, “Recognizing Faces by Dynamic Link Matching,” Proc. Int'l Conf. Artificial Neural Networks, pp. 347-352, 1995.
[70] L. Wiskott, J. Fellous, N. Krüger, and C. von der Malsburg, “Face Recognition by Elastic Bunch Graph Matching,” Intelligent Biometric Techniques in Fingerprint and Face Recognition, L.C. Jain et al. eds., Springer-Verlag, 1999.
[71] J. Yang and A. Waibel, “A Real-Time Face Tracker,” Workshop Applications of Computer Vision, pp. pp. 142-147, 1996.
[72] M.-H. Yang, D.J. Kriegman, and N. Ahuja, "Detecting Faces in Images: A Survey," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 1, Jan. 2002, pp. 34-58.
[73] A.L. Yuille, P.W. Hallinan, and D.S. Cohen, "Feature extraction from faces using deformable templates," Int'l J. Computer Vision, vol. 8, no. 2, 133-144, 1992.
[74] W. Zhao and R. Chellapa, “SFS Based View Synthesis for Robust Face Recognition,” Proc. IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 285-292, 2000.

Index Terms:
Face recognition, occlusions, expression-variant faces, learning from undersampled distributions, facial asymmetry, localization of faces, principal components analysis, pattern recognition.
Citation:
Aleix M. Martínez, "Recognizing Imprecisely Localized, Partially Occluded, and Expression Variant Faces from a Single Sample per Class," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 748-763, June 2002, doi:10.1109/TPAMI.2002.1008382
Usage of this product signifies your acceptance of the Terms of Use.