The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.11 - November (2010 vol.32)
pp: 1940-1954
Sander Koelstra , Queen Mary University of London, London
Maja Pantic , Imperial College, London
Ioannis (Yiannis) Patras , Queen Mary University of London, London
ABSTRACT
In this work, we propose a dynamic texture-based approach to the recognition of facial Action Units (AUs, atomic facial gestures) and their temporal models (i.e., sequences of temporal segments: neutral, onset, apex, and offset) in near-frontal-view face videos. Two approaches to modeling the dynamics and the appearance in the face region of an input video are compared: an extended version of Motion History Images and a novel method based on Nonrigid Registration using Free-Form Deformations (FFDs). The extracted motion representation is used to derive motion orientation histogram descriptors in both the spatial and temporal domain. Per AU, a combination of discriminative, frame-based GentleBoost ensemble learners and dynamic, generative Hidden Markov Models detects the presence of the AU in question and its temporal segments in an input image sequence. When tested for recognition of all 27 lower and upper face AUs, occurring alone or in combination in 264 sequences from the MMI facial expression database, the proposed method achieved an average event recognition accuracy of 89.2 percent for the MHI method and 94.3 percent for the FFD method. The generalization performance of the FFD method has been tested using the Cohn-Kanade database. Finally, we also explored the performance on spontaneous expressions in the Sensitive Artificial Listener data set.
INDEX TERMS
Facial image analysis, facial expression, dynamic texture, motion.
CITATION
Sander Koelstra, Maja Pantic, Ioannis (Yiannis) Patras, "A Dynamic Texture-Based Approach to Recognition of Facial Actions and Their Temporal Models", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 11, pp. 1940-1954, November 2010, doi:10.1109/TPAMI.2010.50
REFERENCES
[1] E. Aarts, "Ambient Intelligence Drives Open Innovation," ACM Interactions, vol. 12, no. 4, pp. 66-68, 2005.
[2] K. Anderson and P. McOwan, "A Real-Time Automated System for Recognition of Human Facial Expressions," IEEE Trans. Systems, Man, and Cybernetics, vol. 36, no. 1, pp. 96-105, Feb. 2006.
[3] M. Bartlett, G. Littlewort-Ford, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 568-573, 2005.
[4] M. Bartlett, G. Littlewort-Ford, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, "Fully Automatic Facial Action Recognition in Spontaneous Behavior," Proc. IEEE Conf. Face and Gesture Recognition, pp. 223-230, 2006.
[5] Y. Chang, C. Hu, R. Feris, and M. Turk, "Manifold-Based Analysis of Facial Expression," J. Image and Vision Computing, vol. 24, no. 6, pp. 605-614, 2006.
[6] D. Chetverikov and R. Péteri, "A Brief Survey of Dynamic Texture Description and Recognition," Proc. Conf. Computer Recognition Systems, vol. 5, pp. 17-26, 2005.
[7] I. Cohen, N. Sebe, F. Cozman, M. Cirelo, and T. Huang, "Learning Bayesian Network Classifiers for Facial Expression Recognition Both Labeled and Unlabeled Data," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 595-601, 2003.
[8] J. Davis and A. Bobick, "The Representation and Recognition of Human Movement Using Temporal Templates," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 928-934, 1997.
[9] E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M. McRorie, J. Martin, L. Devillers, S. Abrilian, A. Batliner, N. Amir, and K. Karpouzis, "The HUMAINE Database: Addressing the Collection and Annotation of Naturalistic and Induced Emotional Data," Lecture Notes in Computer Science, vol. 4738, pp. 488-500, Springer, 2007.
[10] P. Ekman, W. Friesen, and J. Hager, The Facial Action Coding System: A Technique for the Measurement of Facial Movement. A Human Face, 2002.
[11] P. Ekman and E. Rosenberg, What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford Univ. Press, 2005.
[12] J. Friedman, T. Hastie, and R. Tibshirani, "Additive Logistic Regression: A Statistical View of Boosting," The Annals of Statistics, vol. 28, no. 2, pp. 337-407, 2000.
[13] S. Gokturk, J. Bouguet, C. Tomasi, and B. Girod, "Model-Based Face Tracking for Viewindependent Facial Expression Recognition," Proc. IEEE Conf. Face and Gesture Recognition, pp. 272-278, 2002.
[14] G. Guo and C. Dyer, "Learning from Examples in the Small Sample Case—Face Expression Recognition," IEEE Trans. Systems, Man, and Cybernetics, vol. 35, no. 3, pp. 477-488, June 2005.
[15] T. Kanade, J. Cohn, and Y. Tian, "Comprehensive Database for Facial Expression Analysis," Proc. IEEE Conf. Face and Gesture Recognition, pp. 46-53, 2000.
[16] S. Koelstra and M. Pantic, "Non-Rigid Registration Using Free-Form Deformations for Recognition of Facial Actions and Their Temporal Dynamics," Proc. IEEE Conf. Face and Gesture Recognition, pp. 1-8, 2008.
[17] I. Kotsia and I. Pitas, "Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines," IEEE Trans. Image Processing, vol. 16, no. 1, pp. 172-187, Jan. 2007.
[18] G. Littlewort, M. Bartlett, I. Fasel, J. Susskind, and J. Movellan, "Dynamics of Facial Expression Extracted Automatically from Video," Image and Vision Computing, vol. 24, no. 6, pp. 615-625, 2006.
[19] Z. Lu, W. Xie, J. Pei, and J. Huang, "Dynamic Texture Recognition by Spatio-Temporal Multiresolution Histograms," Proc. IEEE Workshop Motion and Video Computing, vol. 2, pp. 241-246, 2005.
[20] S. Lucey, A. Ashraf, and J. Cohn, "Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face," Face Recognition, K. Delac and M. Grgic, eds., pp. 275-286, I-Tech Education and Publishing, 2007.
[21] M. Pantic and M. Bartlett, "Machine Analysis of Facial Expressions," Face Recognition, K. Delac and M. Grgic, eds., pp. 377-416, I-Tech Education and Publishing, 2007.
[22] M. Pantic and I. Patras, "Detecting Facial Actions and Their Temporal Segments in Nearly Frontal-View Face Image Sequences," Proc. IEEE Conf. Systems, Man, and Cybernetics, vol. 4, pp. 3358-3363, 2005.
[23] M. Pantic and I. Patras, "Dynamics of Facial Expressions—Recognition of Facial Actions and Their Temporal Segments from Face Profile Image Sequences," IEEE Trans. Systems, Man, and Cybernetics, vol. 36, no. 2, pp. 433-449, Apr. 2006.
[24] M. Pantic, A. Pentland, A. Nijholt, and T. Huang, "Human Computing and Machine Understanding of Human Behavior: A Survey," Lecture Notes on Artificial Intelligence, vol. 4451, pp. 47-71, Springer, 2007.
[25] M. Pantic and L. Rothkrantz, "Automatic Analysis of Facial Expressions—The State of the Art," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424-1445, Dec. 2000.
[26] M. Pantic and L. Rothkrantz, "Facial Action Recognition for Facial Expression Analysis from Static Face Images," IEEE Trans. Systems, Man, and Cybernetics, vol. 34, no. 3, pp. 1449-1461, June 2004.
[27] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, "Web-Based Database for Facial Expression Analysis," Proc. IEEE Conf. Multimedia and Expo, pp. 317-321, 2005.
[28] R. Polana and R. Nelson, "Temporal Texture and Activity Recognition," Motion-Based Recognition, pp. 87-115, 1997.
[29] D. Rueckert, L. Sonoda, C. Hayes, D. Hill, M. Leach, and D. Hawkes, "Nonrigid Registration Using Free-Form Deformations: Application to Breast MR Images," IEEE Trans. Medical Imaging, vol. 18, no. 8, pp. 712-721, Aug. 1999.
[30] P. Saisan, G. Doretto, Y. Wu, and S. Soatto, "Dynamic Texture Recognition," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 58-63, 2001.
[31] Y. Tian, T. Kanade, and J. Cohn, "Recognizing Action Units for Facial Expression Analysis," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 97-115, Feb. 2001.
[32] Y. Tian, T. Kanade, and J. Cohn, "Evaluation of Gabor-Wavelet-Based Facial Action Unit Recognition in Image Sequences of Increasing Complexity," Proc. IEEE Conf. Face and Gesture Recognition, pp. 218-223, 2002.
[33] Y. Tong, W. Liao, and Q. Ji, "Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1683-1699, Oct. 2007.
[34] M. Valstar and M. Pantic, "Fully Automatic Facial Action Unit Detection and Temporal Analysis," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 3, no. 149, 2006.
[35] M. Valstar and M. Pantic, "Combined Support Vector Machines and Hidden Markov Models for Modeling Facial Action Temporal Dynamics," Lecture Notes on Computer Science, vol. 4796, pp. 118-127, Springer, 2007.
[36] M. Valstar, M. Pantic, and I. Patras, "Motion History for Facial Action Detection from Face Video," Proc. IEEE Conf. Systems, Man, and Cybernetics, pp. 635-640, 2004.
[37] D. Vukandinovic and M. Pantic, "Fully Automatic Facial Feature Point Detection Using Gabor Feature Based Boosted Classifiers," Proc. IEEE Conf. Systems, Man, and Cybernetics, vol. 2, pp. 1692-1698, 2005.
[38] Z. Wen and T. Huang, "Capturing Subtle Facial Motions in 3D Face Tracking," Proc. Int'l Conf. Computer Vision, vol. 2, pp. 1343-1350, 2003.
[39] J. Whitehill and C. Omlin, "Haar Features for FACS AU Recognition," Proc. IEEE Int'l Conf. Face and Gesture Recognition, pp. 97-101, 2006.
[40] Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A Survey of Affect Recognition Methods: Audio, Visual and Spontaneous Expressions," Proc. ACM Conf. Multimodal Interfaces, pp. 126-133, 2007.
[41] Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
[42] Y. Zhang and Q. Ji, "Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequence," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 699-714, May 2005.
[43] G. Zhao and M. Pietikäinen, "Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 915-928, June 2007.
[44] G. Zhao and M. Pietikäinen, "Boosted Multi-Resolution Spatiotemporal Descriptors for Facial Expression Recognition," Pattern Recognition Letters, vol. 30, no. 12, pp. 1117-1127, Sept. 2009.
6 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool