This Article 
 Bibliographic References 
 Add to: 
Emotion Recognition Based on Physiological Changes in Music Listening
December 2008 (vol. 30 no. 12)
pp. 2067-2083
This paper investigates the potential of physiological signals as reliable channels for emotion recognition. All essential stages of an automatic recognition system are discussed, from the recording of a physiological dataset to a feature-based multiclass classification. In order to collect a physiological dataset from multiple subjects over many weeks, we used a musical induction method which spontaneously leads subjects to real emotional states, without any deliberate lab setting. Four-channel biosensors were used to measure electromyogram, electrocardiogram, skin conductivity and respiration changes. A wide range of physiological features from various analysis domains, including time/frequency, entropy, geometric analysis, subband spectra, multiscale entropy, etc., is proposed in order to find the best emotion-relevant features and to correlate them with emotional states. The best features extracted are specified in detail and their effectiveness is proven by classification results. Classification of four musical emotions (positive/high arousal, negative/high arousal, negative/low arousal, positive/low arousal) is performed by using an extended linear discriminant analysis (pLDA). Furthermore, by exploiting a dichotomic property of the 2D emotion model, we develop a novel scheme of emotion-specific multilevel dichotomous classification (EMDC) and compare its performance with direct multiclass classification using the pLDA. Improved recognition accuracy of 95\% and 70\% for subject-dependent and subject-independent classification, respectively, is achieved by using the EMDC scheme.

[1] W. James, The Principles of Psychology. Holt, 1890.
[2] W.B. Cannon, “The James-Lange Theory of Emotions: A Critical Examination and an Alternative Theory,” Am. J. Psychology, vol. 39, pp. 106-127, 1927.
[3] R. Cowie, E. Douglas-Cowie, N. Tsapatsoulis, G. Votsis, S. Kollias, W. Fellenz, and J.G. Taylor, “Emotion Recognition in Human-Computer Interaction,” IEEE Signal Processing Magazine, vol. 18, pp. 32-80, 2001.
[4] P. Ekman, “The Argument and Evidence about Universals in Facial Expressions of Emotion,” Handbook of Social Psychophysiology, pp. 143-164, John Wiley & Sons, 1989.
[5] R.W. Levenson, P. Ekman, P. Heider, and W.V. Friesen, “Emotion and Autonomic Nervous System Activity in the Minangkabau of West Sumatra,” J. Personality and Social Psychology, vol. 62, pp. 972-988, 1992.
[6] J.T. Cacioppo, D.J. Klein, G.G. Bemston, and E. Hatfield, “The Psychophysiology of Emotion,” Handbook of Emotions, M. Lewis and J. Haviland, eds., pp. 119-142, Guilford Press, 1993.
[7] R.W. Levenson, P. Ekman, and W.V. Friesen, “Voluntary Facial Action Generates Emotion-Specific Autonomic Nervous System Activity,” Psychophysiology, vol. 27, pp. 363-384, 1990.
[8] G. Stemmler, M. Heldmann, C.A. Pauls, and T. Scherer, “Constraints for Emotion Specificity in Fear and Anger: The Context Counts,” Psychophysiology, vol. 38, pp. 275-291, 2001.
[9] S. Schachter and J.E. Singer, “Cognitive, Social, and Physiological Determinants of Emotional State,” Psychological Rev., vol. 69, pp.379-399, 1962.
[10] G. Stemmler, “The Autonomic Differentiation of Emotions Revisited: Convergent and Discriminant Validation,” Psychophysiology, vol. 26, pp. 617-632, 1989.
[11] M.P. Kastner and R.G. Crowder, “Perception of the Major/Minor Distinction: IV. Emotional Connotations in Young Children,” Music Perception, vol. 8, pp. 189-201, 1990.
[12] P. Kivy, Sound Sentiment: An Essay on the Musical Emotions. Temple Univ. Press, 1989.
[13] C.L. Krumhansl, “An Exploratory Study of Musical Emotions and Psychophysiology,” Canadian J. Experimental Psychology, vol. 51, pp. 336-352, 1997.
[14] L.B. Meyer, Emotion and Meaning in Music. Univ. of Chicago Press, 1956.
[15] D. Vaitl, W. Vehrs, and S. Sternagel, “Prompts-Leitmotif-Emotion: Play It Again, Richard Wagner,” The Structure of Emotion: Psychophysiological, Cognitive, and Clinical Aspects, pp. 169-189, Hogrefe & Huber, 1993.
[16] W.B. Davis and M.H. Thaut, “The Influence of Preferred Relaxing Music on Measures of State Anxiety, Relaxation, and Physiological Responses,” J. Music Therapy, vol. 26, no. 4, pp. 168-187, 1989.
[17] C.E. Guzzetta, “Effects of Relaxation and Music Therapy on Patients in a Coronary Care Unit with Presumptive Acute Myocardial Infarction,” Heart and Lung: J. Critical Care, vol. 18, no. 6, pp. 609-616, 1989.
[18] J. Healey and R.W. Picard, “Digital Processing of Affective Signals,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 3749-3752, 1998.
[19] R. Picard, E. Vyzas, and J. Healy, “Toward Machine Emotional Intelligence: Analysis of Affective Physiological State,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001.
[20] F. Nasoz, K. Alvarez, C. Lisetti, and N. Finkelstein, “Emotion Recognition from Physiological Signals for Presence Technologies,” Int'l J. Cognition, Technology, and Work, special issue on presence, vol. 6, no. 1, 2003.
[21] J.J. Gross and R.W. Levenson, “Emotion Elicitation Using Films,” Cognition and Emotion, vol. 9, pp. 87-108, 1995.
[22] A. Haag, S. Goronzy, P. Schaich, and J. Williams, “Emotion Recognition Using Bio-Sensors: First Steps Towards an Automatic System,” Proc. Ninth Int'l Conf. Reliable Software Technologies, pp.36-48, 2004.
[23] Center for the Study of Emotion and Attention (CSEA-NIMH), The International Affective Picture System: Digitized Photographs, Center for Research in Psychophysiology, Univ. of Florida, 1995.
[24] K.H. Kim, S.W. Bang, and S.R. Kim, “Emotion Recognition System Using Short-Term Monitoring of Physiological Signals,” Medical & Biological Eng. and Computing, vol. 42, pp. 419-427, 2004.
[25] P. Lang, “The Emotion Probe: Studies of Motivation and Attention,” Am. Psychologist, vol. 50, no. 5, pp. 372-385, 1995.
[26] J. Tooby and L. Cosmides, “The Past Explains the Present: Emotional Adaptations and the Structure of Ancestral Environments,” Ethology and Sociobiology, vol. 11, pp. 375-424, 1990.
[27] H. Scholsberg, “Three Dimensions of Emotion,” Psychological Rev., vol. 61, pp. 81-88, 1954.
[28] B.L. Fredricson and R.W. Levenson, “Positive Emotions Speed Recovery from the Cardiovascular Sequelae of Negative Emotions,” Cognition and Emotion, vol. 12, no. 2, pp. 191-220, 1998.
[29] J.E. LeDoux, The Amygdala: Neurobiological Aspects of Emotion, Memory, and Mental Dysfunction, pp. 339-351. Wiley-Liss, 1992.
[30] D.M. Sloan, “Emotion Regulation in Action: Emotional Reactivity in Experiential Avoidance,” Behavior Research and Therapy, vol. 4, pp. 1257-1270, 2004.
[31] B. Melin and U. Lundberg, “A Biopsychosocial Approach to Work-Stress and Musculoskeletal Disorders,” J. Psychophysiology, vol. 11, no. 3, pp. 238-247, 1997.
[32] H.G. McCurdy, “Consciousness and the Galvanometer,” Psychological Rev., vol. 57, pp. 322-327, 1950.
[33] A. Jain, R. Duin, and J. Mao, “Statistical Pattern Recognition: A Review,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[34] R.D. King, C. Feng, and A. Shutherland, “StatLog: Comparison of Classification Algorithms on Large Real-World Problems,” Applied Artificial Intelligence, vol. 9, no. 3, pp. 259-287, May/June 1995.
[35] M.B. Mcllory and M.D. Cheithin, Clinical Cardiology, fifth ed. VLANGE Medical Book, 1990.
[36] J. Pan and W. Tompkins, “A Real-Time QRS Detection Algorithm,” IEEE Trans. Biomedical Eng., vol. 32, no. 3, pp. 230-323, 1985.
[37] P.W. Kamen, H. Krum, and A.M. Tonkin, “Poincare Plot of Heart Rate Variability Allows Quantitative Display of Parasympathetic Nervous Activity,” Clinical Science, vol. 91, pp. 201-208, 1996.
[38] J. Richmann and J. Moorman, “Physiological Time Series Analysis Using Approximate Entropy and Sample Entropy,” Am. J. Physiology—Heart and Circulatory Physiology, vol. 278, p. H2039, 2000.
[39] M. Costa, A.L. Goldberger, and C.-K. Peng, “Multiscale Entropy Analysis of Biological Signals,” Physical Rev. E, vol. 71, no. 021906, 2005.
[40] R. Thuraisingham and G. Gottwald, “On Multiscale Entropy Analysis for Physiological Data,” Physica A, 2005.
[41] A. Malliani, “The Pattern of Sympathovagal Balance Explored in the Frequency Domain,” News in Physiological Science, vol. 14, pp.111-117, 1999.
[42] A. Jain and D. Zongker, “Feature Selection: Evaluation, Application, and Small Sample Performance,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 153-163, Feb. 1997.
[43] F.J. Ferri, P. Pudil, M. Hatef, and J. Kittler, “Comparative Study of Techniques for Large-Scale Feature Selection,” Pattern Recognition in Practice IV: Multiple Paradigms, Comparative Studies, and Hybrid Systems, S.E. Gelsema and L.N. Kanal, eds., pp. 403-413, 1994.
[44] D.L. Swets and J. Weng, “Using Discriminant Eigenfeatures for Image Retrieval,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 8, pp. 831-836, Aug. 1996.
[45] T. Hastie and R. Tibshirani, “Classification by Pairwise Coupling,” The Annals of Statistics, vol. 26, no. 1, pp. 451-471, 1998.
[46] J. Friedman, “Another Approach to Polychotomous Classification,” technical report, Dept. of Statistics, Stanford Univ., 1996.
[47] L. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms. John Wiley & Sons, 2004.

Index Terms:
Signal analysis, synthesis, and processing, Methodologies and techniques, User/Machine Systems, Theory and methods, Interaction styles, Human-centered computing, Classifier design and evaluation, Feature evaluation and selection, Pattern analysis, Signal processing, Robotics
Jonghwa Kim, Elisabeth André, "Emotion Recognition Based on Physiological Changes in Music Listening," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008, doi:10.1109/TPAMI.2008.26
Usage of this product signifies your acceptance of the Terms of Use.