Subscribe
Issue No.06 - June (2013 vol.35)
pp: 1357-1369
Ognjen Rudovic , Imperial College London, London, UK
Maja Pantic , Imperial College, London, UK
Ioannis Patras , Queen Mary University, London, UK
ABSTRACT
We propose a method for head-pose invariant facial expression recognition that is based on a set of characteristic facial points. To achieve head-pose invariance, we propose the Coupled Scaled Gaussian Process Regression (CSGPR) model for head-pose normalization. In this model, we first learn independently the mappings between the facial points in each pair of (discrete) nonfrontal poses and the frontal pose, and then perform their coupling in order to capture dependences between them. During inference, the outputs of the coupled functions from different poses are combined using a gating function, devised based on the head-pose estimation for the query points. The proposed model outperforms state-of-the-art regression-based approaches to head-pose normalization, 2D and 3D Point Distribution Models (PDMs), and Active Appearance Models (AAMs), especially in cases of unknown poses and imbalanced training data. To the best of our knowledge, the proposed method is the first one that is able to deal with expressive faces in the range from $(-45^\circ)$ to $(+45^\circ)$ pan rotation and $(-30^\circ)$ to $(+30^\circ)$ tilt rotation, and with continuous changes in head pose, despite the fact that training was conducted on a small set of discrete poses. We evaluate the proposed method on synthetic and real images depicting acted and spontaneously displayed facial expressions.
INDEX TERMS
Face recognition, Head, Solid modeling, Active appearance model, Estimation, Training, Magnetic heads,Gaussian process regression, Multiview/pose-invariant facial expression/emotion recognition, head-pose estimation
CITATION
Ognjen Rudovic, Maja Pantic, Ioannis Patras, "Coupled Gaussian processes for pose-invariant facial expression recognition", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 6, pp. 1357-1369, June 2013, doi:10.1109/TPAMI.2012.233
REFERENCES
 [1] M. Pantic, A. Nijholt, A. Pentland, and T. Huang, "Human-Centred Intelligent Human-Computer Interaction (HCI2): How Far Are We from Attaining It?" Int'l J. Autonomous and Adaptive Comm. Systems, vol. 1, no. 2, pp. 168-187, 2008. [2] A. Vinciarelli, M. Pantic, and H. Bourlard, "Social Signal Processing: Survey of an Emerging Domain," Image and Vision Computing J., vol. 27, no. 12, pp. 1743-1759, 2009. [3] Z. Zeng, M. Pantic, G.I. Roisman, and T.S. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009. [4] M.S. Bartlett, G. Littlewort, M.G. Frank, C. Lainscsek, I.R. Fasel, and J.R. Movellan, "Automatic Recognition of Facial Actions in Spontaneous Expressions," J. Multimedia, vol. 1, no. 6, pp. 22-35, 2006. [5] Y. Zhang and Q. Ji, "Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 699-714, May 2005. [6] Y. Tong, W. Liao, and Q. Ji, "Facial Action Unit Recognition by Exploiting Their Dynamic and Semantic Relationships," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1683-1699, Oct. 2007. [7] Z. Zhu and Q. Ji, "Robust Real-Time Face Pose and Facial Expression Recovery," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 681-688, 2006, [8] Y. Tong, J. Chen, and Q. Ji, "A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 2, pp. 258-273, Feb. 2010. [9] S. Moore and R. Bowden, "Local Binary Patterns for Multi-View Facial Expression Recognition," Computer Vision and Image Understanding, vol. 115, no. 4, pp. 541-558, 2011. [10] W. Zheng, H. Tang, Z. Lin, and T.S. Huang, "Emotion Recognition from Arbitrary View Facial Images," Proc. European Conf. Computer Vision, pp. 490-503, 2010, [11] H. Tang, M. Hasegawa-Johnson, and T.S. Huang, "Non-Frontal View Facial Expression Recognition Based on Ergodic Hidden Markov Model Supervectors," Proc. Int'l Conf. Multimedia and Expo, pp. 1202-1207, 2010, [12] M.J. Black and Y. Yacoob, "Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion," Int'l J. Computer Vision, vol. 25, pp. 23-48, 1997. [13] S. Kumano, K. Otsuka, J. Yamato, E. Maeda, and Y. Sato, "Pose-Invariant Facial Expression Recognition Using Variable-Intensity Templates," Int'l J. Computer Vision, vol. 83, no. 2, pp. 178-194, 2009. [14] P. Ekman and W.V. Friesen, Unmasking the Face: A Guideline to Recognising Emotions from Facial Clues, vol. 3. Prentice Hall, 1978. [15] C.M. Bishop, Pattern Recognition and Machine Learning. Springer, 2007. [16] Y. Sun and L. Yin, "Facial Expression Recognition Based on 3D Dynamic Range Model Sequences," Proc. European Conf. Computer Vision, pp. 58-71, 2008. [17] L.A. Jeni, A. Lrincz, T. Nagy, Z. Palotai, J. Sebk, Z. Szab, and D. Takcs, "3D Shape Estimation in Video Sequences Provides High Precision Evaluation of Facial Expressions," Image and Vision Computing, vol. 30, pp. 785-795, 2012. [18] J. Wang, L. Yin, X. Wei, and Y. Sun, "3D Facial Expression Recognition Based on Primitive Surface Feature Distribution," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1399-1406, 2006, [19] R.E. Kaliouby and P. Robinson, "Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures," Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, p. 154. 2004, [20] W.-K. Liao and I. Cohen, "Belief Propagation Driven Method for Facial Gestures Recognition in Presence of Occlusions," Proc. IEEE Conf. Computer Vision and Pattern Recognition Workshops, p. 158. 2006, [21] S. Lucey, A.B. Ashraf, and J. Cohn, Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face. I-Tech Education & Publishing, 2007. [22] J. Sung and D. Kim, "Real-Time Facial Expression Recognition Using STAAM and Layered GDA Classifier," Image and Vision Computing J., vol. 27, no. 9, pp. 1313-1325, 2009. [23] F. Dornaika and J. Orozco, "Real Time 3D Face and Facial Feature Tracking," J. Real-Time Image Processing, vol. 2, no. 1, pp. 35-44, 2007. [24] R. Gross, I. Matthews, and S. Baker, "Generic vs. Person Specific Active Appearance Models," Image and Vision Computing J., vol. 23, pp. 1080-1093, 2005. [25] X. Chai, S. Shan, X. Chen, and W. Gao, "Locally Linear Regression for Pose-Invariant Face Recognition," IEEE Trans. Image Processing, vol. 16, no. 7, pp. 1716-1725, July 2007. [26] M.A.O. Vasilescu and D. Terzopoulos, "Multilinear Analysis of Image Ensembles: Tensorfaces," Proc. European Conf. Computer Vision, pp. 447-460, 2002. [27] Y. Hu, Z. Zeng, L. Yin, X. Wei, X. Zhou, and T.S. Huang, "Multi-View Facial Expression Recognition," Proc. Int'l Conf. Automatic Face and Gesture Recognition, pp. 1-6, 2008, [28] W. Zheng, H. Tang, Z. Lin, and T. Huang, "A Novel Approach to Expression Recognition from Non-Frontal Face Images," Proc. IEEE Int'l Conf. Computer Vision, pp. 1901-1908, 2009. [29] O. Rudovic, I. Patras, and M. Pantic, "Coupled Gaussian Process Regression for Pose-Invariant Facial Expression Recognition," Proc. European Conf. Computer Vision, pp. 350-363, 2010. [30] M. Valstar, B. Martinez, X. Binefa, and M. Pantic, "Facial Point Detection Using Boosted Regression and Graph Models," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2729-2736, 2010. [31] E. Murphy-Chutorian and M.M. Trivedi, "Head Pose Estimation in Computer Vision: A Survey," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 4, pp. 607-626, Apr. 2009. [32] M.F. Valstar and M. Pantic, "Fully Automatic Recognition of the Temporal Phases of Facial Actions," IEEE Trans. Systems, Man, and Cybernetics, vol. 42, no. 1, pp. 28-43, Feb. 2012. [33] B. Schölkopf, A.J. Smola, R.C. Williamson, and P.L. Bartlett, "New Support Vector Algorithms," Neural Computation, vol. 12, pp. 1207-1245, 2000. [34] K. Grochow, S.L. Martin, A. Hertzmann, and Z. Popović, "Style-Based Inverse Kinematics," Proc. ACM Int'l Conf. Computer Graphics and Interactive Techniques, pp. 522-531, 2004. [35] C.E. Rasmussen and C.K.I. Williams, Gaussian Processes for Machine Learning. The MIT Press, 2005. [36] S.J. Julier and J.K. Uhlmann, "A Non-Divergent Estimation Algorithm in the Presence of Unknown Correlations," Proc. Am. Control Conf., pp. 2369-2373, 1997. [37] V. Tresp and M. Taniguchi, "Combining Estimators Using Non-Constant Weighting Functions," Proc. Neural Information Processing Systems Conf., pp. 419-426, 1995. [38] V. Tresp, "A Bayesian Committee Machine," Neural Computing, vol. 12, no. 11, pp. 2719-2741, 2000. [39] P. Boyle and M. Frean, "Dependent Gaussian Processes," Neural Information Processing Systems, pp. 217-224, 2005. [40] K. Yu, V. Tresp, and A. Schwaighofer, "Learning Gaussian Processes from Multiple Tasks," Proc. 22nd Int'l Conf. Machine Learning, pp. 1012-1019, 2005. [41] E.V. Bonilla, K.M.A. Chai, and C.K.I. Williams, "Multi-Task Gaussian Process Prediction," Neural Information Processing Systems, 2008. [42] M. Alvarez and N. Lawrence, "Sparse Convolved Multiple Output Gaussian Processes," Proc. Neural Information Processing Systems Conf., pp. 57-64, 2008. [43] L. Bo and C. Sminchisescu, "Twin Gaussian Processes for Structured Prediction," In'l J. Computer Vision, vol. 87, nos. 1/2, pp. 28-52, 2010. [44] S. Yu, B. Krishnapuram, R. Rosales, and R.B. Rao, "Bayesian Co-Training," J. Machine Learning Research, vol. 12, pp. 2649-2680, 2011. [45] R. Gross, I. Matthews, J. Cohn, T. Kanade, and S. Baker, "Multi-Pie," Image and Vision Computing J., vol. 28, no. 5, pp. 807-813, 2010. [46] G. McKeown, M.F. Valstar, R. Cowie, and M. Pantic, "The Semaine Corpus of Emotionally Coloured Character Interactions," Proc. Int'l Conf. Multimedia and Expo, pp. 1079-1084, 2010. [47] C.-C. Chang and C.-J. Lin, "LIBSVM: A Library for Support Vector Machines," ACM Trans. Intelligent Systems and Technology, vol. 2, pp. 1-27, 2011. [48] T. Cootes and C. Taylor, "Active Shape Models—Smart Snakes," Proc. British Machine Vision Conf., pp. 266-275, 1992. [49] Y. Hu, Z. Zeng, L. Yin, X. Wei, J. Tu, and T. Huang, "A Study of Non-Frontal-View Facial Expressions Recognition," Proc. Int'l Conf. Pattern Recognition, pp. 1-4, 2008.
32 ms
(Ver 2.0)

Marketing Automation Platform