The Community for Technology Leaders
Green Image
Issue No. 02 - April-June (2012 vol. 3)
ISSN: 1949-3045
pp: 211-223
M. Pantic , Dept. of Comput., Imperial Coll. London, London, UK
M. Soleymani , Comput. Sci. Dept., Univ. of Geneva, Carouge, Switzerland
T. Pun , Comput. Sci. Dept., Univ. of Geneva, Carouge, Switzerland
ABSTRACT
This paper presents a user-independent emotion recognition method with the goal of recovering affective tags for videos using electroencephalogram (EEG), pupillary response and gaze distance. We first selected 20 video clips with extrinsic emotional content from movies and online resources. Then, EEG responses and eye gaze data were recorded from 24 participants while watching emotional video clips. Ground truth was defined based on the median arousal and valence scores given to clips in a preliminary study using an online questionnaire. Based on the participants' responses, three classes for each dimension were defined. The arousal classes were calm, medium aroused, and activated and the valence classes were unpleasant, neutral, and pleasant. One of the three affective labels of either valence or arousal was determined by classification of bodily responses. A one-participant-out cross validation was employed to investigate the classification performance in a user-independent approach. The best classification accuracies of 68.5 percent for three labels of valence and 76.4 percent for three labels of arousal were obtained using a modality fusion strategy and a support vector machine. The results over a population of 24 participants demonstrate that user-independent emotion recognition can outperform individual self-reports for arousal assessments and do not underperform for valence assessments.
INDEX TERMS
support vector machines, behavioural sciences computing, electroencephalography, emotion recognition, sensor fusion, user-independent emotion recognition, multimodal emotion recognition, video response, user-independent emotion recognition method, affective tags, electroencephalogram, pupillary response, gaze distance, extrinsic emotional content, emotional video clips, median arousal, valence scores, bodily response classification, one-participant-out cross validation, modality fusion strategy, support vector machine, Videos, Emotion recognition, Physiology, Tagging, Motion pictures, Electroencephalography, Multimedia communication, affective computing., Emotion recognition, EEG, pupillary reflex, pattern classification
CITATION
M. Pantic, M. Soleymani, T. Pun, "Multimodal Emotion Recognition in Response to Videos", IEEE Transactions on Affective Computing, vol. 3, no. , pp. 211-223, April-June 2012, doi:10.1109/T-AFFC.2011.37
216 ms
(Ver )