Toward Machine Emotional Intelligence: Analysis of Affective Physiological State
October 2001 (vol. 23 no. 10)
pp. 1175-1191
DOI Bookmark:
http://doi.ieeecomputersociety.org/10.1109/34.954607
—The ability to recognize emotion is one of the hallmarks of emotional intelligence, an aspect of human intelligence that has been argued to be even more important than mathematical and verbal intelligences. This paper proposes that machine intelligence needs to include emotional intelligence and demonstrates results toward this goal: developing a machine's ability to recognize human affective state given four physiological signals. We describe difficult issues unique to obtaining reliable affective data and collect a large set of data from a subject trying to elicit and experience each of eight emotional states, daily, over multiple weeks. This paper presents and compares multiple algorithms for feature-based recognition of emotional state from this data. We analyze four physiological signals that exhibit problematic day-to-day variations: The features of different emotions on the same day tend to cluster more tightly than do the features of the same emotion on different days. To handle the daily variations, we propose new features and algorithms and compare their performance. We find that the technique of seeding a Fisher Projection with the results of Sequential Floating Forward Search improves the performance of the Fisher Projection and provides the highest recognition rates reported to date for classification of affect from physiology: 81 percent recognition accuracy on eight classes of emotion, including neutral. [1] A.R. Damasio, Descartes' Error: Emotion, Reason, and the Human Brain. New York: Gosset/Putnam Press, 1994.[2] J. LeDoux, The Emotional Brain. New York: Simon&Schuster, 1996.[3] P. Salovey and J.D. Mayer, “Emotional Intelligence,” Imagination, Cognition and Personality, vol. 9, no. 3, pp. 185-211, 1990.[4] D. Goleman, Emotional Intelligence. New York: Bantam Books, 1995.[5] B. Reeves and C. Nass, The Media Equation. Cambridge Univ. Press, Center for the Study of Language and Information, 1996.[6] M. Sigman and L. Capps, Children with Autism: A Developmental Perspective. Cambridge, Mass.: Harvard Univ. Press, 1997[7] R. Picard, Affective Computing. MIT Press, 1997.[8] J. Healey, R.W. Picard, and F. Dabek, “A New Affect-Perceiving Interface and Its Application to Personalized Music Selection,” Proc. Workshop Perceptual User Interfaces, Nov. 1998.[9] R.W. Picard and J. Healey, “Affective Wearables,” Personal Technologies, vol. 1, no. 4, pp. 231-240, 1997.[10] K.R. Scherer, “Speech and Emotional States,” Speech Evaluation in Psychiatry, J.K. Darby, ed., chapter 10, pp. 189-220, Grune and Stratton, Inc., 1981.[11] R. Banse and K.R. Scherer, “Acoustic Profiles in Vocal Emotion Expression,” J. Personality and Social Psychology, vol. 70, no. 3, pp. 614-636, 1996.[12] T. Polzin, “Detecting Verbal and Non-Verbal Cues in the Communication of Emotions,” PhD thesis, School of Computer Science, June 2000.[13] J. Hansen, Comm. during ICASSP '99 Panel on Speech Under Stress. Proc. Int'l Conf. Acoustics, Speech, and Signal Processing '99, 1999.[14] J.N. Bassili, “Emotion Recognition: The Role of Facial Movement and the Relative Importance of Upper and Lower Areas of the Face,” J. Personality and Social Psychology, vol. 37, pp. 2049-2058, 1979.[15] Y. Yacoob and L.S. Davis, “Recognizing Human Facial Expression from Long Image Sequences Using Optical Flow,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 6, pp. 636-642, June 1996.[16] I. Essa and A. Gardner, “Prosody Analysis for Speaker Affect Determination,” Proc. Workshop Perceptual User Interfaces '97, pp. 45-46, Oct. 1997.[17] I.A. Essa, “Analysis, Interpretation and Synthesis of Facial Expressions,” PhD thesis, Mass. Inst. of Technology, Media Lab, Cambridge, Mass., Feb. 1995.[18] J.F. Cohn, A.J. Zlochower, J. Lien, and T. Kanade, “Automated Face Analysis by Feature Point Tracking has High Concurrent Validity with Manual FACS Coding,” Psychophysiology, vol. 36, pp. 35-43, 1999.[19] M. Bartlett, J.C. Hager, P. Ekman, and T.J. Sejnowski, “Measuring Facial Expressions by Computer Image Analysis,” Psychophysiology, vol. 36, pp. 253-263, 1999.[20] G. Donato, M.S. Bartlett, J.C. Hager, P. Ekman, and T.J. Sejnowski, “Classifying Facial Actions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 10, pp. 974-989, Oct. 1999.[21] L.C. DeSilva, T. Miyasato, and R. Nakatsu, “Facial Emotion Recognition Using Multi-Modal Information,” Proc. IEEE Int'l Conf. Information, Comm., and Signal Processing, pp. 397-401, Sept. 1997.[22] T.S. Huang, L.S. Chen, and H. Tao, “Bimodal Emotion Recognition by Man and Machine,” Proc. ATR Workshop Virtual Communication Environments, Apr. 1998.[23] L.S. Chen, T.S. Huang, T. Miyasato, and R. Nakatsu, “Multimodal Human Emotion/Expression Recognition,” Proc. Third Int'l Conf. Automatic Face and Gesture Recognition, pp. 366-371, Apr. 1998.[24] J.T. Cacioppo and L.G. Tassinary, “Inferring Psychological Significance from Physiological Signals,” Am. Psychologist, vol. 45, pp. 16-28, Jan. 1990.[25] W. James, William James: Writings 1878-1899, pp. 350-365, The Library of Am., 1992. Originally published in 1890.[26] W.B. Cannon, “The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory,” Am. J. Psychology, vol. 39, pp. 106-124, 1927.[27] S. Schachter, “The Interaction of Cognitive and Physiological Determinants of Emotional State,” Advances in Experimental Psychology, L. Berkowitz, ed., vol. 1, pp. 49-80, 1964.[28] P. Ekman, R.W. Levenson, and W.V. Friesen, “Autonomic Nervous System Activity Distinguishes Among Emotions,” Science, vol. 221, pp. 1208-1210, Sept. 1983.[29] W.M. Winton, L. Putnam, and R. Krauss, “Facial and Autonomic Manifestations of the Dimensional Structure of Emotion,” J. Experimental Social Psychology, vol. 20, pp. 195-216, 1984.[30] A.J. Fridlund and C.E. Izard, “Electromyographic Studies of Facial Expressions of Emotions and Patterns of Emotions,” Social Psychophysiology: A Sourcebook, J.T. Cacioppo and R.E. Petty, eds., pp. 243-286, 1983.[31] J.T. Cacioppo, G.G. Berntson, J.T. Larsen, K.M. Poehlmann, and T.A. Ito, “The Psychophysiology of Emotion,” Handbook of Emotions, M. Lewis and J.M. Haviland-Jones, eds., pp. 173-191, 2000.[32] D. Keltner and P. Ekman, “The Psychophysiology of Emotion,” Handbook of Emotions, M. Lewis and J.M. Haviland-Jones, eds., pp. 236-249, 2000.[33] D.M. Clynes, Sentics: The Touch of the Emotions. Anchor Press/Doubleday, 1977.[34] C.E. Izard, “Four Systems for Emotion Activation: Cognitive and Noncognitive Processes,” Psychological Rev., vol. 100, no. 1, pp. 68-90, 1993.[35] H. Hama and K. Tsuda, “Finger-Pressure Waveforms Measured on Clynes' Sentograph Distinguished Among Emotions,” Perceptual and Motor Skills, vol. 70, pp. 371-376, 1990.[36] H. Schlosberg, “Three Dimensions of Emotion,” Psychological Rev., vol. 61, pp. 81-88, Mar. 1954.[37] P.J. Lang, “The Emotion Probe: Studies of Motivation and Attention,” Am. Psychologist, vol. 50, no. 5, pp. 372-385, 1995.[38] J.A. Healey, “Wearable and Automotive Systems for Affect Recognition from Physiology,” Technical Report 526, PhD thesis, Mass. Inst. Technology, Cambridge, Mass., May 2000.[39] E. Vyzas and R.W. Picard, “Affective Pattern Classification,” Proc. AAAI 1998 Fall Symp., Emotional and Intelligent: The Tangled Knot of Cognition, Oct. 1998.[40] E. Vyzas and R.W. Picard, “Offline and Online Recognition of Emotion Expression from Physiological Data,” Proc. Workshop Emotion-Based Agent Architectures, Third Int'l Conf. Autonomous Agents, pp. 135-142, May 1999.[41] A.V. Oppenheim and R.W. Schafer, Discrete-Time Signal Processing.Englewood Cliffs, N.J.: Prentice Hall, 1989.[42] D.T. Lyyken, R. Rose, B. Luther, and M. Maley, “Correcting Psychophysiological Measures for Individual Differences in Range,” Psychophysiological Bulletin, vol. 66, pp. 481-484, 1966.[43] D.T. Lyyken and P.H. Venables, “Direct Measurement of Skin Conductance: A Proposal for Standardization,” Psychophysiology, vol. 8, no. 5, pp. 656-672, 1971.[44] M.E. Dawson, A.M. Schell, and D.L. Filion, “The Electrodermal System,” Principles of Psychophysiology: Physical, Social, and Inferential Elements, J.T. Cacioppo and L.G. Tassinary, eds., pp. 295-324, 1990.[45] P. Pudil, J. Novovicová, and J. Kittler, "Floating search methods in feature selection," Pattern Recognition Letters, vol. 15, pp. 1,119-1,125, 1994.[46] A. Jain and D. Zongker, Feature Selection: Evaluation, Application, and Small Sample Performance IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 153-158, Feb. 1997.[47] P. Pudil, J. Novovicova, and J. Kittler, “Simultaneous Learning of Decision Rules and Important Attributes for Classification Problems in Image Analysis,” Image and Vision Computing, vol. 12, pp. 193-198, Apr. 1994.[48] R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis. Wiley-Interscience, 1973.[49] T.M. Cover and P.E. Hart, “Nearest Neighbor Pattern Classification,” IEEE Trans. Information Theory, vol. 13, pp. 21-27, Jan. 1967.[50] J. Healey and R.W. Picard, “Digital Processing of Affective Signals,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, 1998.[51] J. Scheirer, J. Klein, R. Fernandez, and R.W. Picard, “Frustrating the User on Purpose: A Step Toward Building an Affective Computer,” Interaction with Computers, 2001, to appear.
Citation:
R.W. Picard, E. Vyzas, J. Healey, "Toward Machine Emotional Intelligence: Analysis of Affective Physiological State," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001, doi:10.1109/34.954607
Usage of this product signifies your acceptance of the
Terms of Use.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb