This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Multimodal Emotion Recognition in Response to Videos
April-June 2012 (vol. 3 no. 2)
pp. 211-223
ASCII Text
x 
 
T. Pun, M. Pantic, M. Soleymani, "Multimodal Emotion Recognition in Response to Videos," IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 211-223, April-June, 2012.
 
 
BibTex
x
 
@article{ 10.1109/T-AFFC.2011.37,
author = {T. Pun and M. Pantic and M. Soleymani},
title = {Multimodal Emotion Recognition in Response to Videos},
journal ={IEEE Transactions on Affective Computing},
volume = {3},
number = {2},
issn = {1949-3045},
year = {2012},
pages = {211-223},
doi = {http://doi.ieeecomputersociety.org/10.1109/T-AFFC.2011.37},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Affective Computing
TI - Multimodal Emotion Recognition in Response to Videos
IS - 2
SN - 1949-3045
SP211
EP223
EPD - 211-223
A1 - T. Pun,
A1 - M. Pantic,
A1 - M. Soleymani,
PY - 2012
KW - support vector machines
KW - behavioural sciences computing
KW - electroencephalography
KW - emotion recognition
KW - sensor fusion
KW - user-independent emotion recognition
KW - multimodal emotion recognition
KW - video response
KW - user-independent emotion recognition method
KW - affective tags
KW - electroencephalogram
KW - pupillary response
KW - gaze distance
KW - extrinsic emotional content
KW - emotional video clips
KW - median arousal
KW - valence scores
KW - bodily response classification
KW - one-participant-out cross validation
KW - modality fusion strategy
KW - support vector machine
KW - Videos
KW - Emotion recognition
KW - Physiology
KW - Tagging
KW - Motion pictures
KW - Electroencephalography
KW - Multimedia communication
KW - affective computing.
KW - Emotion recognition
KW - EEG
KW - pupillary reflex
KW - pattern classification
VL - 3
JA - IEEE Transactions on Affective Computing
ER -
 
T. Pun, Comput. Sci. Dept., Univ. of Geneva, Carouge, Switzerland
M. Pantic, Dept. of Comput., Imperial Coll. London, London, UK
M. Soleymani, Comput. Sci. Dept., Univ. of Geneva, Carouge, Switzerland
This paper presents a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram (EEG), pupillary response and gaze distance. We first selected 20 video clips with extrinsic emotional content from movies and online resources. Then, EEG responses and eye gaze data were recorded from 24 participants while watching emotional video clips. Ground truth was defined based on the median arousal and valence scores given to clips in a preliminary study using an online questionnaire. Based on the participants' responses, three classes for each dimension were defined. The arousal classes were calm, medium aroused, and activated and the valence classes were unpleasant, neutral, and pleasant. One of the three affective labels of either valence or arousal was determined by classification of bodily responses. A one-participant-out cross validation was employed to investigate the classification performance in a user-independent approach. The best classification accuracies of 68.5 percent for three labels of valence and 76.4 percent for three labels of arousal were obtained using a modality fusion strategy and a support vector machine. The results over a population of 24 participants demonstrate that user-independent emotion recognition can outperform individual self-reports for arousal assessments and do not underperform for valence assessments.

[1] J. Kim and E. André, "Emotion Recognition Based on Physiological Changes in Music Listening," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008.
[2] J. Rottenberg, R.D. Ray, and J.J. Gross, Emotion Elicitation Using Films, pp. 9-28, series in affective science. Oxford Univ. Press, 2007.
[3] M. Soleymani, G. Chanel, J.J.M. Kierkels, and T. Pun, "Affective Characterization of Movie Scenes Based on Content Analysis and Physiological Changes," Int'l J. Semantic Computing, vol. 3, no. 2, pp. 235-254, June 2009.
[4] R.W. Picard and S.B. Daily, "Evaluating Affective Interactions: Alternatives to Asking What Users Feel," Proc. CHI Workshop Evaluating Affective Interfaces: Innovative Approaches, 2005.
[5] K.R. Scherer, "Studying the Emotion-Antecedent Appraisal Process: An Expert System Approach," Cognition & Emotion, vol. 7, no. 3, pp. 325-355, 1993.
[6] A. Ortony, G.L. Clore, and A. Collins, The Cognitive Structure of Emotions. Cambridge Univ. Press, July 1988.
[7] D. Sander, D. Grandjean, and K.R. Scherer, "A Systems Approach to Appraisal Mechanisms in Emotion," Neural Networks, vol. 18, no. 4, pp. 317-352, May 2005.
[8] W. Wirth and H. Schramm, "Media and Emotions," Comm. Research Trends, vol. 24, no. 3, pp. 3-39, 2005.
[9] K.R. Scherer, "What Are Emotions? And How Can They Be Measured?" Social Science Information, vol. 44, no. 4, pp. 695-729, Dec. 2005.
[10] A. Hanjalic and L.-Q. Xu, "Affective Video Content Representation and Modeling," IEEE Trans. Multimedia, vol. 7, no. 1, pp. 143-154, http://dx.doi.org/10.1109TMM.2004.840618 , Feb. 2005.
[11] M. Tkalčič, U. Burnik, and A. Košir, "Using Affective Parameters in a Content-Based Recommender System for Images," User Modeling and User-Adapted Interaction, vol. 20, pp. 279-311, Oct. 2010.
[12] M.K. Shan, F.F. Kuo, M.F. Chiang, and S.Y. Lee, "Emotion-Based Music Recommendation by Affinity Discovery from Film Music," Expert Systems Applications, vol. 36, no. 4, pp. 7666-7674, Sept. 2009.
[13] H. Joho, J.M. Jose, R. Valenti, and N. Sebe, "Exploiting Facial Expressions for Affective Video Summarisation," Proc. ACM Int'l Conf. Image and Video Retrieval, 2009.
[14] C.L. Lisetti and F. Nasoz, "Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals," EURASIP J. Applied Signal Processing, vol. 2004, no. 1, pp. 1672-1687, Jan. 2004.
[15] K. Takahashi, "Remarks on Emotion Recognition from Bio-Potential Signals," Proc. Second Int'l Conf. Autonomous Robots and Agents, 2005.
[16] S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J.-S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, "Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos," Proc. Int'l Conf. Brain Informatics, Y. Yao, R. Sun, T. Poggio, J. Liu, N. Zhong, and J. Huang, eds., ch. 9, pp. 89-100, 2010.
[17] H. Joho, J. Staiano, N. Sebe, and J. Jose, "Looking at the Viewer: Analysing Facial Activity to Detect Personal Highlights of Multimedia Contents," Multimedia Tools and Applications, pp. 1-19, Oct. 2010.
[18] I. Arapakis, I. Konstas, and J.M. Jose, "Using Facial Expressions and Peripheral Physiological Signals as Implicit Indicators of Topical Relevance," Proc. 17th ACM Int'l Conf. Multimedia, pp. 461-470, 2009.
[19] G. Irie, T. Satou, A. Kojima, T. Yamasaki, and K. Aizawa, "Affective Audio-Visual Words and Latent Topic Driving Model for Realizing Movie Affective Scene Classification," IEEE Trans. Multimedia, vol. 12, no. 6, pp. 523-535, Oct. 2010.
[20] M. Soleymani, J.J.M. Kierkels, G. Chanel, and T. Pun, "A Bayesian Framework for Video Affective Representation," Proc. Int'l Conf. Affective Computing and Intelligent Interaction, pp. 1-7, Sept. 2009.
[21] P. Lang, M. Bradley, and B. Cuthbert, "International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual," Technical Report A-8, Univ. of Florida, Gainesville, 2008.
[22] P.J. Lang, M.K. Greenwald, M.M. Bradley, and A.O. Hamm, "Looking at Pictures: Affective, Facial, Visceral, and Behavioral Reactions," Psychophysiology, vol. 30, no. 3, pp. 261-273, 1993.
[23] J. Wang and Y. Gong, "Recognition of Multiple Drivers' Emotional State," Proc. 19th Int'l Conf. Pattern Recognition, 2008.
[24] J.A. Healey, "Wearable and Automotive Systems for Affect Recognition from Physiology," PhD dissertation, MIT, 2000.
[25] G. Chanel, J.J.M. Kierkels, M. Soleymani, and T. Pun, "Short-Term Emotion Assessment in a Recall Paradigm," Int'l J. Human-Computer Studies, vol. 67, no. 8, pp. 607-627, Aug. 2009.
[26] V. Kolodyazhniy, S.D. Kreibig, J.J. Gross, W.T. Roth, and F.H. Wilhelm, "An Affective Computing Approach to Physiological Emotion Specificity: Toward Subject-Independent and Stimulus-Independent Classification of Film-Induced Emotions," Psychophysiology, vol. 7, no. 48, pp. 908-922, 2011.
[27] M.M. Bradley, L. Miccoli, M.A. Escrig, and P.J. Lang, "The Pupil as a Measure of Emotional Arousal and Autonomic Activation," Psychophysiology, vol. 45, no. 4, pp. 602-607, July 2008.
[28] T. Partala and V. Surakka, "Pupil Size Variation as an Indication of Affective Processing," Int'l J. Human-Computer Studies, vol. 59, nos. 1/2, pp. 185-198, 2003.
[29] Y. Gao, A. Barreto, and M. Adjouadi, "Monitoring and Processing of the Pupil Diameter Signal for Affective Assessment of a Computer User," Proc. 13th Int'l Conf. Human-Computer Interaction. Part I: New Trends, pp. 49-58, 2009.
[30] J.J.M. Kierkels, M. Soleymani, and T. Pun, "Queries and Tags in Affect-Based Multimedia Retrieval," Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 1436-1439, 2009.
[31] M. Pantic and A. Vinciarelli, "Implicit Human-Centered Tagging," IEEE Signal Processing Magazine, vol. 26, no. 6, pp. 173-180, Nov. 2009.
[32] J.A. Russell, "Culture and the Categorization of Emotions," Psychological Bull., vol. 110, no. 3, pp. 426-450, 1991.
[33] J.A. Russell and A. Mehrabian, "Evidence for a Three-Factor Theory of Emotions," J. Research in Personality, vol. 11, no. 3, pp. 273-294, Sept. 1977.
[34] M. Soleymani, J. Davis, and T. Pun, "A Collaborative Personalized Affective Video Retrieval System," Proc. Third Int'l Conf. Affective Computing and Intelligent Interaction and Workshops, Sept. 2009.
[35] A. Schaefer, F. Nils, X. Sanchez, and P. Philippot, "Assessing the Effectiveness of a Large Database of Emotion-Eliciting Films: A New Tool for Emotion Researchers," Cognition & Emotion, vol. 24, no. 7, pp. 1153-1172, 2010.
[36] J. Lichtenauer, M. Valstar, J. Shen, and M. Pantic, "Cost-Effective Solution to Synchronized Audio-Visual Capture Using Multiple Sensors," Proc. Sixth IEEE Int'l Conf. Advanced Video and Signal Based Surveillance, pp. 324-329, 2009.
[37] R. Adolphs, D. Tranel, and A.R. Damasio, "Dissociable Neural Systems for Recognizing Emotions," Brain and Cognition, vol. 52, no. 1, pp. 61-69, June 2003.
[38] A.R. Damasio, T.J. Grabowski, A. Bechara, H. Damasio, L.L.B. Ponto, J. Parvizi, and R.D. Hichwa, "Subcortical and Cortical Brain Activity during the Feeling of Self-Generated Emotions," Nature Neuroscience, vol. 3, no. 10, pp. 1049-1056, Oct. 2000.
[39] R.J. Davidson, "Affective Neuroscience and Psychophysiology: Toward a Synthesis," Psychophysiology, vol. 40, no. 5, pp. 655-665, Sept. 2003.
[40] L.I. Aftanas, N.V. Reva, A.A. Varlamov, S.V. Pavlov, and V.P. Makhnev, "Analysis of Evoked EEG Synchronization and Desynchronization in Conditions of Emotional Activation in Humans: Temporal and Topographic Characteristics," Neuroscience and Behavioral Physiology, vol. 34, no. 8, pp. 859-867, Oct. 2004.
[41] P.D. Welch, "The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Averaging over Short, Modified Periodograms," IEEE Trans. Audio and Electroacoustics, vol. 15, no. 2, pp. 70-73, June 1967.
[42] S.K. Sutton and R.J. Davidson, "Prefrontal Brain Asymmetry: A Biological Substrate of the Behavioral Approach and Inhibition Systems," Psychological Science, vol. 8, no. 3, pp. 204-210, 1997.
[43] V.F. Pamplona, M.M. Oliveira, and G.V.G. Baranoski, "Photorealistic Models for Pupil Light Reflex and Iridal Pattern Deformation," ACM Trans. Graphics, vol. 28, no. 4, pp. 1-12, 2009.
[44] A. Longtin and J. Milton, "Modelling Autonomous Oscillations in the Human Pupil Light Reflex Using Nonlinear Delay-Differential Equations," Bull. Math. Biology, vol. 51, no. 5, pp. 605-624, Sept. 1989.
[45] H. Bouma and L.C.J. Baghuis, "Hippus of the Pupil: Periods of Slow Oscillations of Unknown Origin," Vision Research, vol. 11, no. 11, pp. 1345-1351, 1971.
[46] F.H. Kanfer, "Verbal Rate, Eyeblink, and Content in Structured Psychiatric Interviews," J. Abnormal and Social Psychology, vol. 61, no. 3, pp. 341-347, 1960.
[47] D. Ruta and B. Gabrys, "An Overview of Classifier Fusion Methods," Computing and Information Systems, vol. 7, no. 1, pp. 1-10, 2000.
[48] L.I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms. Wiley-Interscience, July 2004.
[49] J.C. Platt, "Probabilities for SV Machines," Advances in Large Margin Classifier, pp. 61-74, MIT Press, 2000.
[50] T.F. Wu, C.J. Lin, and R.C. Weng, "Probability Estimates for Multi-Class Classification by Pairwise Coupling," J. Machine Learning Research, vol. 5, pp. 975-1005, 2004.
[51] C. Chang and C. Lin, "LIBSVM: A Library for Support Vector Machines," Science, vol. 2, pp. 1-39, 2001.
[52] G. Chanel, C. Rebetez, M. Bétrancourt, and T. Pun, "Emotion Assessment from Physiological Signals for Adaptation of Game Difficulty," IEEE Trans. Systems, Man, and Cybernetics, Part A: Systems and Humans, vol. 41, no. 6, pp. 1052-1063, Nov. 2011.
[53] S.D. Kreibig, G. Schaefer, and T. Brosch, Psychophysiological Response Patterning in Emotion: Implications for Affective Computing. ch. 2.4, pp. 105-130, Oxford Univ. Press, 2010.
[54] D. Watson, L.A. Clark, and A. Tellegen, "Development and Validation of Brief Measures of Positive and Negative Affect: The PANAS Scales," J. Personality and Social Psychology, vol. 54, no. 6, pp. 1063-1070, June 1988.

Index Terms:
support vector machines,behavioural sciences computing,electroencephalography,emotion recognition,sensor fusion,user-independent emotion recognition,multimodal emotion recognition,video response,user-independent emotion recognition method,affective tags,electroencephalogram,pupillary response,gaze distance,extrinsic emotional content,emotional video clips,median arousal,valence scores,bodily response classification,one-participant-out cross validation,modality fusion strategy,support vector machine,Videos,Emotion recognition,Physiology,Tagging,Motion pictures,Electroencephalography,Multimedia communication,affective computing.,Emotion recognition,EEG,pupillary reflex,pattern classification
Citation:
T. Pun, M. Pantic, M. Soleymani, "Multimodal Emotion Recognition in Response to Videos," IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 211-223, April-June 2012, doi:10.1109/T-AFFC.2011.37
Usage of this product signifies your acceptance of the Terms of Use.