This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships
October 2007 (vol. 29 no. 10)
pp. 1683-1699
A system that could automatically analyze the facial actions in real time has applications in a wide range of different fields. However, developing such a system is always challenging due to the richness, ambiguity, and the dynamic nature of facial actions. Although a number of research groups attempt to recognize facial action units (AUs) by either improving facial feature extraction techniques, or the AU classification techniques, these methods often recognize AUs or certain AU combinations individually and statically, ignoring the semantic relationships among AUs and the dynamics of AUs. Hence, these approaches cannot always recognize AUs reliably, robustly, and consistently.In this paper, we propose a novel approach that systematically accounts for the relationships among AUs and their temporal evolutions for AU recognition. Specifically, we use a dynamic Bayesian network (DBN) to model the relationships among different AUs. The DBN provides a coherent and unified hierarchical probabilistic framework to represent probabilistic relationships among various AUs and to account for the temporal changes in facial action development. Within our system, robust computer vision techniques are used to obtain AU measurements. And such AU measurements are then applied as evidence to the DBN for inferring various AUs. The experiments show that the integration of AU relationships and AU dynamics with AU measurements yields significant improvement of AU recognition, especially for spontaneous facial expressions and under more realistic environment including illumination variation, face pose variation, and occlusion.

[1] P. Ekman and W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, 1978.
[2] K. Scherer and P. Ekman, Handbook of Methods in Nonverbal Behavior Research. Cambridge Univ. Press, 1982.
[3] J.J. Lien, T. Kanade, J.F. Cohn, and C. Li, “Detection, Tracking, and Classification of Action Units in Facial Expression,” J. Robotics and Autonomous System, vol. 31, pp. 131-146, 2000.
[4] G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, and T.J. Sejnowski, “Classifying Facial Actions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, Oct. 1999.
[5] B. Fasel and J. Luettin, “Recognition of Asymmetric Facial Action Unit Activities and Intensities,” Proc. Int'l Conf. Pattern Recognition, vol. 1, pp. 1100-1103, 2000.
[6] E. Smith, M.S. Bartlett, and J.R. Movellan, “Computer Recognition of Facial Actions: A Study of Co-Articulation Effects,” Proc. Eighth Ann. Joint Symp. Neural Computation, 2001.
[7] J.J. Bazzo and M.V. Lamar, “Recognizing Facial Actions Using Gabor Wavelets with Neutral Face Average Difference,” Proc. Sixth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp.505-510, 2004.
[8] M.S. Bartlett, G. Littlewort, M.G. Frank, C. Lainscsek, I. Fasel, and J.R. Movellan, “Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 568-573, 2005.
[9] I. Cohen, N. Sebe, A. Garg, L. Chen, and T. Huang, “Facial Expression Recognition from Video Sequences: Temporal and Static Modeling,” Computer Vision and Image Understanding, vol. 91, nos. 1-2, pp. 160-187, 2003.
[10] J.F. Cohn, L.I. Reed, Z. Ambadar, J. Xiao, and T. Moriyama, “Automatic Analysis and Recognition of Brow Actions and Head Motion in Spontaneous Facial Behavior,” Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, vol. 1, pp. 610-616, 2004.
[11] Y. Tian, T. Kanade, and J.F. Cohn, “Recognizing Action Units for Facial Expression Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 97-115, Feb. 2001.
[12] Y. Tian, T. Kanade, and J.F. Cohn, “Evaluation of Gabor-Wavelet-Based Facial Action Unit Recognition in Image Sequences of Increasing Complexity,” Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 218-223, 2002.
[13] M.F. Valstar, I. Patras, and M. Pantic, “Facial Action Unit Detection Using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, Workshop Vision for Human-Computer Interaction, 2005.
[14] Y. Zhang and Q. Ji, “Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp.699-714, May 2005.
[15] H. Gu and Q. Ji, “Facial Event Classification with Task-Oriented Dynamic Bayesian Network,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 870-875, 2004.
[16] A. Kapoor, Y. Qi, and R.W. Picard, “Fully Automatic Upper Facial Action Recognition,” Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures, pp. 195-202, 2003.
[17] R. El Kaliouby and P.K. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition Workshop, 2004.
[18] M. Pantic and L.J.M. Rothkrantz, “Facial Action Recognition for Facial Expression Analysis from Static Face Images,” IEEE Trans. Systems, Man, and Cybernetics–Part B: Cybernetics, vol. 34, no. 3, pp.1449-1461, June 2004.
[19] I. Cohen, F.G. Cozman, N. Sebe, M.C. Cirelo, and T.S. Huang, “Semisupervised Learning of Classifiers: Theory, Algorithms, and Their Application to Human-Computer Interaction,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 12, pp. 1553-1567, Dec. 2004.
[20] P. Ekman, W.V. Friesen, and J.C. Hager, “Facial Action Coding System: The Manual,” Research Nexus Division, Network Information Research Corp., Salt Lake City, 2002.
[21] A. Lanitis, C.J. Taylor, and T.F. Cootes, “Automatic Interpretation and Coding of Face Images Using Flexible Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 743-756, July 1997.
[22] J.F. Cohn and A. Zlochower, “A Computerized Analysis of Facial Expression: Feasibility of Automated Discrimination,” Am. Psychological Soc., 1995.
[23] M. Valstar, M. Pantic, and I. Patras, “Motion History for Facial Action Detection in Video,” Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, vol. 1, pp. 635-640, 2004.
[24] C. Huang and Y. Huang, “Facial Expression Recognition Using Model-Based Feature Extraction and Action Parameters Classification,” J. Visual Comm. and Image Representation, vol. 8, no. 3, pp.278-290, 1997.
[25] J.N. Bassili, “Emotion Recognition: The Role of Facial Movement and the Relative Importance of Upper and Lower Areas of the Face,” J. Personality and Social Psychology, vol. 37, pp. 2049-2058, 1979.
[26] P. Wang and Q. Ji, “Learning Discriminant Features for Multi-View Face and Eye Detection,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 373-379, 2005.
[27] P. Viola and M. Jones, “Robust Real-Time Object Detection,” Int'l J. Computer Vision, vol. 57, no. 2, pp. 137-154, May 2004.
[28] P. Wang, M.B. Green, Q. Ji, and J. Wayman, “Automatic Eye Detection and Its Validation,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, Workshop Face Recognition Grand Challenge Experiments, vol. 3, 2005.
[29] P.J. Phillips, P.J. Flynn, T. Scruggs, K.W. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek, “Overview of the Face Recognition Grand Challenge,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 947-954, 2005.
[30] T. Lee, “Image Representation Using 2D Gabor Wavelets,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 10, pp.959-971, Oct. 1996.
[31] Z. Zhang, M. Lyons, M. Schuster, and S. Akamatsu, “Comparison between Geometry-Based and Gabor-Wavelets-Based Facial Expression Recognition Using Multi-Layer Perceptron,” Proc. Third IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 454-459, 1998.
[32] T. Kanade, J.F. Cohn, and Y. Tian, “Comprehensive Database for Facial Expression Analysis,” Proc. Fourth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 46-53, 2000.
[33] G. Schwarz, “Estimating the Dimension of a Model,” The Annals of Statistics, vol. 6, pp. 461-464, 1978.
[34] D. Heckerman, D. Geiger, and D.M. Chickering, “Learning Bayesian Networks: The Combination of Knowledge and Statistical Data,” Machine Learning, vol. 20, no. 3, pp. 197-243, 1995.
[35] D. Heckerman, “A Tutorial on Learning with Bayesian Networks,” Technical Report MSR-TR-95-06, Microsoft Research, 1995.
[36] D. Spiegelhalter and S. Lauritzen, “Sequential Updating of Conditional Probabilities on Directed Graphical Structures,” Networks, vol. 20, pp. 579-605, 1990.
[37] K.B. Korb and A.E. Nicholson, Bayesian Artificial Intelligence. Chapman and Hall/CRC, 2004.
[38] U. Kjaerulff, “DHUGIN: A Computational System for Dynamic Time-Sliced Bayesian Networks,” Int'l J. Forecasting–Special Issue on Probability Forecasting, vol. 11, pp. 89-111, 1995.
[39] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-Based Database for Facial Expression Analysis,” Proc. IEEE Int'l Conf. Multimedia and Expo, July 2005.
[40] M. Stewart Bartlett, G. Littlewort, J. Movellan, and M.S. Frank, “Auto FACS Coding,” http://mplab.ucsd.edu/grants/project1/research fully-auto-facs-coding.html, 2007.
[41] I. Young, L. van Vliet, and M. van Ginkel, “Recursive Gabor Filtering,” IEEE Trans. Signal Processing, vol. 50, no. 11, pp. 2798-2805, 2002.

Index Terms:
Facial Action Unit Recognition, Facial Expression Analysis, Facial Action Coding System, Bayesian Networks
Citation:
Yan Tong, Wenhui Liao, Qiang Ji, "Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1683-1699, Oct. 2007, doi:10.1109/TPAMI.2007.1094
Usage of this product signifies your acceptance of the Terms of Use.