The Community for Technology Leaders
RSS Icon
Issue No.02 - February (2010 vol.32)
pp: 258-273
Yan Tong , GE Global Research Center, Niskayuna
Jixu Chen , Rensselaer Polytechnic Institute, Troy
Qiang Ji , Rensselaer Polytechnic Institute, Troy
Facial expression is a natural and powerful means of human communication. Recognizing spontaneous facial actions, however, is very challenging due to subtle facial deformation, frequent head movements, and ambiguous and uncertain facial motion measurements. Because of these challenges, current research in facial expression recognition is limited to posed expressions and often in frontal view. A spontaneous facial expression is characterized by rigid head movements and nonrigid facial muscular movements. More importantly, it is the coherent and consistent spatiotemporal interactions among rigid and nonrigid facial motions that produce a meaningful facial expression. Recognizing this fact, we introduce a unified probabilistic facial action model based on the Dynamic Bayesian network (DBN) to simultaneously and coherently represent rigid and nonrigid facial motions, their spatiotemporal dependencies, and their image measurements. Advanced machine learning methods are introduced to learn the model based on both training data and subjective prior knowledge. Given the model and the measurements of facial motions, facial action recognition is accomplished through probabilistic inference by systematically integrating visual measurements with the facial action model. Experiments show that compared to the state-of-the-art techniques, the proposed system yields significant improvements in recognizing both rigid and nonrigid facial motions, especially for spontaneous facial expressions.
Facial action unit recognition, face pose estimation, facial action analysis, facial action coding system, Bayesian networks.
Yan Tong, Jixu Chen, Qiang Ji, "A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 2, pp. 258-273, February 2010, doi:10.1109/TPAMI.2008.293
[1] M. Pantic and M. Bartlett, “Machine Analysis of Facial Expressions,” Face Recognition, K. Delac and M. Grgic, eds., pp. 377-416, I-Tech Education and Publishing, 2007.
[2] P. Ekman and W.V. Friesen, Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, 1978.
[3] K. Scherer and P. Ekman, Handbook of Methods in Nonverbal Behavior Research. Cambridge Univ. Press, 1982.
[4] J.F. Cohn and K. Schmidt, “The Timing of Facial Motion in Posed and Spontaneous Smiles,” Int'l J. Wavelets, Multiresolution, and Information Processing, vol. 2, pp. 1-12, Mar. 2004.
[5] M. Pantic and I. Patras, “Dynamics of Facial Expression: Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences,” IEEE Trans. Systems, Man, and Cybernetics—Part B: Cybernetics, vol. 36, no. 2, pp. 433-449, Apr. 2006.
[6] Y. Tong, W. Liao, and Q. Ji, “Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp.1683-1699, Oct. 2007.
[7] M. Pantic and L.J.M. Rothkrantz, “Automatic Analysis of Facial Expressions: The State of the Art,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424-1445, Dec. 2000.
[8] B. Fasel and J. Luettin, “Automatic Facial Expression Analysis: A Survey,” Pattern Recognition, vol. 36, no. 1, pp. 259-275, 2003.
[9] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J. Taylor, “Emotion Recognition in Human-Computer Interaction,” IEEE Signal Processing Magazine, vol. 18, no. 1, pp. 32-80, Jan. 2001.
[10] Y. Tian, T. Kanade, and J. Cohn, “Facial Expression Analysis,” Handbook of Face Recognition, S. Li and A. Jain, eds., Springer, 2004.
[11] M. Pantic, A. Pentland, A. Nijholt, and T.S. Huang, “Human Computing Machine Understanding of Human Behavior: A Survey,” Artificial Intelligence for Human Computing, T.S. Huang, A. Nijholt, M. Pantic, and A. Pentland, eds., Springer Verlag, 2007.
[12] J. Wang, L. Yin, X. Wei, and Y. Sun, “3D Facial Expression Recognition Based on Primitive Surface Feature Distribution,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 2, pp.1399-1406, 2006.
[13] Y. Chang, M. Vieira, M. Turk, and L. Velho, “Automatic 3D Facial Expression Analysis in Videos,” Proc. Analysis and Modelling of Faces and Gestures, pp. 293-307, 2005.
[14] B. Braathen, M.S. Bartlett, G.C. Littlewort, E. Smith, and J.R. Movellan, “An Approach to Automatic Recognition of Spontaneous Facial Actions,” Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 345-350, 2002.
[15] M.S. Bartlett, G.C. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, and J.R. Movellan, “Automatic Recognition of Facial Actions in Spontaneous Expressions,” J. Multimedia, vol. 1, no. 6, pp. 22-35, Sept. 2006.
[16] F. Dornaika and F. Davoine, “Simultaneous Facial Action Tracking and Expression Recognition Using a Particle Filter,” Proc. Int'l Conf. Computer Vision, vol. 2, pp. 1733-1738, 2005.
[17] Y. Zhang and Q. Ji, “Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp.699-714, May 2005.
[18] M.F. Valstar, I. Patras, and M. Pantic, “Facial Action Unit Detection Using Probabilistic Actively Learned Support Vector Machines on Tracked Facial Point Data,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, Workshop Vision for Human-Computer Interaction, June 2005.
[19] R. El Kaliouby and P. Robinson, “Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures,” Real-Time Vision for HCI, pp. 181-200, Springer Verlag, 2005.
[20] B. Bascle and A. Blake, “Separability of Pose and Expression in Facial Tracking and Animation,” Proc. Int'l Conf. Computer Vision, pp. 323-328, 1998.
[21] Z. Zhu and Q. Ji, “Robust Real-Time Face Pose and Facial Expression Recovery,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 681-688, 2006.
[22] M. Anisetti, V. Bellandi, E. Damiani, and F. Beverina, “3D Expressive Face Model-Based Tracking Algorithm,” Proc. Signal Processing, Pattern Recognition, and Applications, pp. 111-116, 2006.
[23] M.A.O. Vasilescu and D. Terzopoulos, “Multilinear Analysis of Image Ensembles: Tensorfaces,” Proc. European Conf. Computer Vision, pp. 447-460, 2002.
[24] T.K. Marks, J. Hershey, J.C. Roddey, and J.R. Movellan, “Joint Tracking of Pose, Expression, and Texture Using Conditionally Gaussian Filters,” Advances in Neural Information Processing Systems, vol. 17, pp. 889-896, The MIT Press, 2005.
[25] A. Kapoor, Y. Qi, and R.W. Picard, “Fully Automatic Upper Facial Action Recognition,” Proc. IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures, pp. 195-202, 2003.
[26] J.F. Cohn, L.I. Reed, Z. Ambadar, J. Xiao, and T. Moriyama, “Automatic Analysis and Recognition of Brow Actions and Head Motion in Spontaneous Facial Behavior,” Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, vol. 1, pp. 610-616, 2004.
[27] N. Sebe, M. Lew, I. Cohen, S. Yafei, T. Gevers, and T. Huang, “Authentic Facial Expression Analysis,” Proc. Sixth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 517-522, 2004.
[28] M.F. Valstar, M. Pantic, Z. Ambadar, and J.F. Cohn, “Spontaneous vs. Posed Facial Behavior: Automatic Analysis of Brow Actions,” Proc. Eighth Int'l Conf. Multimodal Interfaces, pp. 162-170, 2006.
[29] G. Littlewort, M.S. Bartlett, and K. Lee, “Faces of Pain: Automated Measurement of Spontaneous Facial Expressions of Genuine and Posed Pain,” Proc. Ninth Int'l Conf. Multimodal Interfaces, pp. 15-21, 2007.
[30] Z. Zeng, Y. Fu, G. Roisman, Z. Wen, Y. Hu, and T.S. Huang, “Spontaneous Emotional Facial Expression Detection,” J. Multimedia, vol. 1, no. 5, pp. 1-8, 2006.
[31] S. Ioannou, A. Raouzaiou, V. Tzouvaras, T. Mailis, K. Karpouzis, and S. Kollias, “Emotion Recognition through Facial Expression Analysis Based on a Neurofuzzy Method,” Neural Networks, vol. 18, no. 4, pp. 423-435, 2005.
[32] S. Lucey, A.B. Ashraf, and J. Cohn, “Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face,” Face Recognition Book, K. Kurihara, ed., Pro Literatur Verlag, Apr. 2007.
[33] J. Russell and J. Fernandez-Dols, The Psychology of Facial Expression. Cambridge Univ. Press, 1997.
[34] J.N. Bassili, “Emotion Recognition: The Role of Facial Movement and the Relative Importance of Upper and Lower Areas of the Face,” J. Personality and Social Psychology, vol. 37, no. 11, pp. 2049-2058, 1979.
[35] P. Ekman and E. Rosenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford Univ. Press, 2005.
[36] P. Ekman, W.V. Friesen, and J.C. Hager, Facial Action Coding System: The Manual. Research Nexus Division, Network Information Research Corp., 2002.
[37] P. Wang and Q. Ji, “Multi-View Face and Eye Detection Using Discriminant Features,” Computer Vision and Image Understanding, vol. 105, no. 2, pp. 99-111, Feb. 2007.
[38] Z. Zhu and Q. Ji, “Robust Pose Invariant Facial Feature Detection and Tracking in Real-Time,” Proc. Int'l Conf. Pattern Recognition, vol. 1, pp. 1092-1095, 2006.
[39] Y. Tong, Y. Wang, Z. Zhu, and Q. Ji, “Robust Facial Feature Tracking under Varying Face Pose and Facial Expression,” Pattern Recognition, vol. 40, no. 11, pp. 3195-3208, Nov. 2007.
[40] K. Murphy, “Inference and Learning in Hybrid Bayesian Networks,” Technical Report CSD-98-990, Dept. of Computer Science, Univ. of California Berkeley, 1998.
[41] P. Ekman, Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. Norton, 1985.
[42] K. Schmidt and J. Cohn, “Dynamics of Facial Expression: Normative Characteristics and Individual Differences,” Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 728-731, 2001.
[43] S. Nishio, K. Koyama, and T. Nakamura, “Temporal Differences in Eye and Mouth Movements Classifying Facial Expressions of Smiles” Proc. Third IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 206-211, Apr. 1998.
[44] T. Dean and K. Kanazawa, “Probabilistic Temporal Reasoning,” Proc. Seventh Nat'l Conf. Artificial Intelligence, pp. 524-528, 1988.
[45] T. Kanade, J.F. Cohn, and Y. Tian, “Comprehensive Database for Facial Expression Analysis,” Proc. Fourth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 46-53, 2000.
[46] E. Douglas-Cowie, R. Cowie, and M. Schroeder, “The Description of Naturally Occurring Emotional Speech,” Proc. 15th Int'l Congress of Phonetic Sciences, 2003.
[47] Multiple Aspects of Discourse Research Lab, http:/madre, 2009.
[48] G. Schwarz, “Estimating the Dimension of a Model,” Annals of Statistics, vol. 6, pp. 461-464, 1978.
[49] D. Heckerman, D. Geiger, and D.M. Chickering, “Learning Bayesian Networks: The Combination of Knowledge and Statistical Data,” Machine Learning, vol. 20, no. 3, pp. 197-243, 1995.
[50] S.J. Russell and P. Norvig, Artificial Intelligence: A Modern Approach. Prentice-Hall, 1995.
[51] Y. Tong and Q. Ji, “Learning Bayesian Networks with Qualitative Constraints,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2008.
[52] C.P. de Campos, Y. Tong, and Q. Ji, “Exploiting Qualitative Constraints for Learning Bayesian Network Parameters,” Proc. European Conf. Computer Vision, 2008.
[53] K.B. Korb and A.E. Nicholson, Bayesian Artificial Intelligence. Chapman and Hall/CRC, 2004.
[54] K. Murphy, “The Bayes Net Toolbox for Matlab,” Computing Science and Statistics, vol. 33, pp. 331-350, 2001.
30 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool