The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.11 - November (2009 vol.31)
pp: 2106-2111
Jacob Whitehill , University of California, San Diego, La Jolla
Gwen Littlewort , University of California, San Diego, La Jolla
Ian Fasel , University of Arizona, Tucson
Marian Bartlett , University of California, San Diego, La Jolla
Javier Movellan , University of California, San Diego, La Jolla
ABSTRACT
Machine learning approaches have produced some of the highest reported performances for facial expression recognition. However, to date, nearly all automatic facial expression recognition research has focused on optimizing performance on a few databases that were collected under controlled lighting conditions on a relatively small number of subjects. This paper explores whether current machine learning methods can be used to develop an expression recognition system that operates reliably in more realistic conditions. We explore the necessary characteristics of the training data set, image registration, feature representation, and machine learning algorithms. A new database, GENKI, is presented which contains pictures, photographed by the subjects themselves, from thousands of different people in many different real-world imaging conditions. Results suggest that human-level expression recognition accuracy in real-life illumination conditions is achievable with machine learning technology. However, the data sets currently used in the automatic expression recognition literature to evaluate progress may be overly constrained and could potentially lead research into locally optimal algorithmic solutions.
INDEX TERMS
Face and gesture recognition, machine learning, computer vision.
CITATION
Jacob Whitehill, Gwen Littlewort, Ian Fasel, Marian Bartlett, Javier Movellan, "Toward Practical Smile Detection", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.31, no. 11, pp. 2106-2111, November 2009, doi:10.1109/TPAMI.2009.42
REFERENCES
[1] Y.-L. Tian, T. Kanade, and J. Cohn, “Facial Expression Analysis,” Handbook of Face Recognition, S.Z. Li and A.K. Jain, eds., Springer, Oct. 2003.
[2] B. Fasel and J. Luettin, “Automatic Facial Expression Analysis: Survey,” Pattern Recognition, vol. 36, pp. 259-275, 2003.
[3] M. Pantic and L. Rothkrantz, “Automatic Analysis of Facial Expressions: The State of the Art,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424-1445, Dec. 2000.
[4] Z. Zeng, M. Pantic, G. Roisman, and T. Huang, “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
[5] M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, “Fully Automatic Facial Action Recognition in Spontaneous Behavior,” Proc. Automatic Facial and Gesture Recognition, 2006.
[6] Y. Wang, H. Ai, B. Wu, and C. Huang, “Real Time Facial Expression Recognition with Adaboost,” Proc. 17th Int'l Conf. Pattern Recognition, 2004.
[7] M. Pantic and J. Rothkrantz, “Facial Action Recognition for Facial Expression Analysis from Static Face Images,” IEEE Trans. Systems, Man and Cybernetics, vol. 34, no. 3, pp. 1449-1461, June 2004.
[8] Y. Tian, T. Kanade, and J. Cohn, “Recognizing Action Units for Facial Expression Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 2, pp. 97-115, Feb. 2001.
[9] A. Kapoor, Y. Qi, and R. Picard, “Fully Automatic Upper Facial Action Recognition,” IEEE Int'l Workshop Analysis and Modeling of Faces and Gestures, 2003.
[10] I. Kotsia and I. Pitas, “Facial Expression Recognition in Image Sequences Using Geometric Deformation Features and Support Vector Machines,” IEEE Trans. Image Processing, vol. 16, no. 1, pp. 172-187, Jan. 2007.
[11] Z. Wen and T. Huang, “Capturing Subtle Facial Motions in 3D Face Tracking,” Proc. IEEE Int'l Conf. Computer Vision, 2003.
[12] N. Sebe, Y. Sun, E. Bakker, M. Lew, I. Cohen, and T. Huang, “Towards Authentic Emotion Recognition,” Proc. IEEE Int'l Conf. Multimedia and Expo, 2004.
[13] J. Cohn and K. Schmidt, “The Timing of Facial Motion in Posed and Spontaneous Smiles,” Int'l J. Wavelets, Multiresolution, and Information Processing, vol. 2, pp. 1-12, 2004.
[14] I. Cohen, N. Sebe, L. Chen, A. Garg, and T. Huang, “Facial Expression Recognition from Video Sequences: Temporal and Static Modelling,” Computer Vision and Image Understanding, special issue on face recognition, vol. 91, pp. 160-187, 2003.
[15] Y. Zhang and Q. Ji, “Active and Dynamic Information Fusion for Facial Expression Understanding from Image Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 699-714, May 2005.
[16] P. Yang, Q. Liu, and D. Metaxas, “Boosting Coded Dynamic Features for Facial Action Units and Facial Expression Recognition,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005.
[17] Y. Tong, W. Liao, and Q. Ji, “Facial Action Unit Recognition by Exploiting their Dynamic and Semantic Relationships,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 10, pp. 1683-1699, Oct. 2007.
[18] G. Zhao and M. Pietikäinen, “Dynamic Texture Recognition Using Local Binary Patterns with an Application to Facial Expressions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 6, pp. 915-928, June 2007.
[19] Z. Zhu and Q. Ji, “Robust Real-Time Face Pose and Facial Expression Recovery,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2006.
[20] T. Kanade, J. Cohn, and Y.-L. Tian, “Comprehensive Database for Facial Expression Analysis,” Proc. Fourth IEEE Int'l Conf. Automatic Face and Gesture Recognition, pp. 46-53, Mar. 2000.
[21] T. Sim, S. Baker, and M. Bsat, “The CMU Pose, Illumination, and Expression Database,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 12, pp. 1615-1618, Dec. 2003.
[22] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-Based Database for Facial Expression Analysis,” Proc. IEEE Int'l Conf. Multimedia and Expo, 2005.
[23] A. OToole, J. Harms, S. Snow, D. Hurst, M. Pappas, and H. Abdi, “A Video Database of Moving Faces and People,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 812-816, May 2005.
[24] G. Donato, M. Bartlett, J. Hager, P. Ekman, and T. Sejnowski, “Classifying Facial Actions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, Oct. 1999.
[25] G. Littlewort, M. Bartlett, I. Fasel, J. Susskind, and J. Movellan, “Dynamics of Facial Expression Extracted Automatically from Video,” Image and Vision Computing, vol. 24, no. 6, pp. 615-625, 2006.
[26] P. Ekman and W. Friesen, “Pictures of Facial Affect,” Photographs, available from Human Interaction Laboratory, Univ. of California, San Francisco, 1976.
[27] C. Cortes and M. Mohri, “Confidence Intervals for the Area under the ROC Curve,” Proc. Advances in Neural Information Processing Systems, 2004.
[28] I. Fasel, B. Fortenberry, and J. Movellan, “A Generative Framework for Real Time Object Detection and Classification,” Computer Vision and Image Understanding, vol. 98, pp. 182-210, 2005.
[29] A. Pnevmatikakis, A. Stergiou, E. Rentzeperis, and L. Polymenakos, “Impact of Face Registration Errors on Recognition,” Proc. Third Int'l Federation for Information Processing Conf. Artificial Intelligence Applications & Innovations, 2006.
[30] J. Movellan, “Tutorial on Gabor Filters,” technical report, MPLab Tutorials, Univ. of California, San Diego, 2005.
[31] P. Viola and M. Jones, “Robust Real-Time Face Detection,” Int'l J. Computer Vision, vol. 57, pp. 137-154, 2004.
[32] D. Lowe, “Object Recognition from Local Scale-Invariant Features,” Proc. Int'l Conf. Computer Vision, 1999.
[33] K. Levi and Y. Weiss, “Learning Object Detection from a Small Number of Examples: The Importance of Good Features,” Proc. 2004 IEEE Conf. Computer Vision and Pattern Recognition, 2004.
[34] T. Ojala, M. Pietikäinen, and D. Harwood, “A Comparative Study of Texture Measures with Classification Based on Feature Distributions,” Pattern Recognition, vol. 29, no. 1, pp. 51-59, 1996.
[35] J. Friedman, T. Hastie, and R. Tibshirani, “Additive Logistic Regression: A Statistical View of Boosting,” Annals of Statistics, vol. 28, no. 2, pp. 337-407, 2000.
[36] Y. Freund and R. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting,” Proc. European Conf. Computational Learning Theory, pp. 23-37, 1995.
[37] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Springer Verlag, 2001.
[38] G. Heusch, Y. Rodriguez, and S. Marcel, “Local Binary Patterns as an Image Preprocessing for Face Authentication,” Proc. Seventh Int'l Conf. Automatic Face and Gesture Recognition, 2006.
[39] M. Bartlett, G. Littlewort, M. Frank, C. Lainscsek, I. Fasel, and J. Movellan, “Automatic Recognition of Facial Actions in Spontaneous Expressions,” J.Multimedia, vol. 1, p. 22, 2006.
[40] I. Fenwick and M. Rice, “Reliability of Continuous Measurement Copy-Testing Methods,” J. Advertising Research, vol. 13, pp. 23-29, 1991.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool