This Article 
 Bibliographic References 
 Add to: 
Coding, Analysis, Interpretation, and Recognition of Facial Expressions
July 1997 (vol. 19 no. 7)
pp. 757-763

Abstract—We describe a computer vision system for observing facial motion by using an optimal estimation optical flow method coupled with geometric, physical and motion-based dynamic models describing the facial structure. Our method produces a reliable parametric representation of the face's independent muscle action groups, as well as an accurate estimate of facial motion.

Previous efforts at analysis of facial expression have been based on the Facial Action Coding System (FACS), a representation developed in order to allow human psychologists to code expression from static pictures. To avoid use of this heuristic coding scheme, we have used our computer vision system to probabilistically characterize facial motion and muscle activation in an experimental population, thus deriving a new, more accurate, representation of human facial expressions that we call FACS+. Finally, we show how this method can be used for coding, analysis, interpretation, and recognition of facial expressions.

[1] J.N. Bassili, "Facial Motion in the Perception of Faces and of Emotional Expression," J. Experimental Psychology, vol. 4, pp. 373-379, 1978.
[2] J.N. Bassili, "Emotion Recognition: The Role of Facial Motion and the Relative Importance of Upper and Lower Areas of the Face," J. Personality and Social Psychology, vol. 37, pp. 2,049-2,059, 1979.
[3] K.-J. Bathe, Finite Element Procedures in Engineering Analysis. Prentice-Hall, 1982.
[4] M.J. Black and Y. Yacoob, "Tracking and Recognizing Rigid and Non-Rigid Facial Motions Using Local Parametric Model of Image Motion," Proc. Int'l Conf. Computer Vision, pp. 374-381,Cambridge, Mass., 1995.
[5] R.G. Brown, Introduction to Random Signal Analysis and Kalman Filtering. John Wiley&Sons Inc., 1983.
[6] V. Bruce, Recognising Faces. Lawrence Erlbaum Associates, 1988.
[7] J.S. Bruner and R. Taguiri, "The Perception of People," Handbook of Social Psyschology. Addison-Wesley, 1954.
[8] P. Ekman, "The Argument and Evidence About Universals in Facial Expressions of Emotion," Handbook of Social Psychophysiology, H. Wagner and A. Manstead, eds. Lawrence Erlbaum, 1989.
[9] P. Ekman and W.V. Friesen, Facial Action Coding System.Palo Alto, Calif.: Consulting Psychologists Press, Inc., 1978.
[10] P. Ekman, T. Huang, T. Sejnowski and J. Hager, eds., "Final Report to NSF of the Planning Workshop on Facial Expression Understanding," technical report, Nat'l Science Foundation, Human Interaction Lab., Univ. of California, San Francsco, 1993.
[11] I. Essa, "Analysis, Interpretation, and Synthesis of Facial Expressions," PhD thesis, Massachusetts Inst. of Technology, MIT Media Laboratory, Cambridge, Mass., 1994.
[12] I. Essa, S. Basu, T. Darrell, and A. Pentland, "Modeling, Tracking and Interactive Animation of Faces and Heads Using Input From Video," Proc. Computer Animation Conf. 1996, pp. 68-79, June 1996.
[13] I. Essa, T. Darrell, and A. Pentland, "Tracking Facial Motion," Proc. Workshop Motion of Nonrigid and Articulated Objects, pp. 36-42.Cambridge, Mass., 1995.
[14] I. Essa and A. Pentland, "Facial Expression Recognition Using a Dynamic Model and Motion Energy," Proc. Int'l Conf. Computer Vision, pp. 360-367,Cambridge, Mass., 1995.
[15] I. Essa, S. Sclaroff, and A. Pentland, "A Unified Approach for Physical and Geometric Modeling for Graphics and Animation," Computer Graphics Forum, The Int'l J. Eurographics Assoc., vol. 2, no. 3, 1992.
[16] B. Friedland, Control System Design: An Introduction to State-Space Methods. McGraw-Hill, 1986.
[17] C.E. Izard, "Facial Expressions and the Regulation of Emotions," J. Personality and Social Psychology, vol. 58, no. 3, pp. 487-498, 1990.
[18] H. Li, P. Roivainen, and R. Forchheimer, "3D Motion Estimation in Model-Based Facial Image Coding," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 545-555, June 1993.
[19] K. Mase, "Recognition of Facial Expressions for Optical Flow," IEICE Trans., Special Issue on Computer Vision and Its Applications, vol. 74, no. 10, 1991.
[20] K. Mase and A. Pentland, "Lipreading by Optical Flow," Systems and Computers, vol. 22, no. 6, pp. 67-76, 1991.
[21] D. Metaxas and D. Terzopoulos, “Shape and Non-Rigid Motion Estimation through Physics-Based Synthesis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 580-591, June 1993.
[22] B. Moghaddam and A. Pentland, "Face Recognition Using View-Based and Modular Eigenspaces," Automatic Systems for the Identification and Inspection of Humans, vol. 2,277, SPIE, 1994.
[23] C. Pelachaud, N. Badler, and M. Viaud, "Final Report to NSF of the Standards for Facial Animation Workshop," technical report, Nat'l Science Foundation, Univ. of Pennsylvania, 1994.
[24] A. Pentland, B. Moghaddam, and Starner, "View-Based and Modular Eigenspaces for Face Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994, pp. 84-91.
[25] A. Pentland and S. Sclaroff, "Closed-Form Solutions for Physically-Based Shape Modeling and Recognition," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 7, pp. 715-729, July 1991.
[26] S. Pieper, J. Rosen, and D. Zeltzer, "Interactive Graphics for Plastic Surgery: A Task Level Analysis and Implementation," Computer Graphics, Special Issue: ACM Siggraph, 1992 Symp. Interactive 3D Graphics, pp. 127-134, 1992.
[27] S.M. Platt and N.I. Badler, "Animating Facial Expression," ACM SIGGRAPH Conf. Proc., vol. 15, no. 3, pp. 245-252, 1981.
[28] E.P. Simoncelli, "Distributed Representation and Analysis of Visual Motion," PhD thesis, Massachusetts Inst. of Tech nology, 1993.
[29] D. Terzopoulos and K. Waters, "Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 569-579, 1993.
[30] S.A. Wainwright, W.D. Biggs, J.D. Curry, and J.M. Gosline, Mechanical Design in Organisms. Princeton Univ. Press, 1976.
[31] J.Y.A. Wang and E. Adelson, "Layered Representation for Motion Analysis," Proc. Computer Vision and Pattern Recognition Conf., 1993.
[32] K. Waters and D. Terzopoulos, "Modeling and Animating Faces Using Scanned Data," J. Visualization and Computer Animation, vol. 2, pp. 123-128, 1991.
[33] Y. Yacoob and L. Davis, "Computing Spatio-Temporal Representations of Human Faces," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 70-75,Seattle, 1994.

Index Terms:
Facial expression analysis, expression recognition, face processing, emotion recognition, facial analysis, motion analysis, perception of action, vision-based HCI.
Irfan A. Essa, Alex P. Pentland, "Coding, Analysis, Interpretation, and Recognition of Facial Expressions," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 757-763, July 1997, doi:10.1109/34.598232
Usage of this product signifies your acceptance of the Terms of Use.