The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - Jan.-March (2012 vol.3)
pp: 18-31
S. Koelstra , Sch. of Electron. Eng. & Comput. Sci., Queen Mary Univ. of London, London, UK
C. Muhl , Human Media Interaction Group, Univ. of Twente, Enschede, Netherlands
M. Soleymani , Comput. Sci. Dept., Univ. of Geneva, Carouge, Switzerland
Jong-Seok Lee , Sch. of Integrated Technol., Yonsei Univ., Incheon, South Korea
A. Yazdani , Multimedia Signal Process. Group, Ecole Polytechniaue Fed. de Lausanne (EPFL), Lausanne, Switzerland
T. Ebrahimi , Multimedia Signal Process. Group, Ecole Polytechniaue Fed. de Lausanne (EPFL), Lausanne, Switzerland
T. Pun , Comput. Sci. Dept., Univ. of Geneva, Carouge, Switzerland
A. Nijholt , Human Media Interaction Group, Univ. of Twente, Enschede, Netherlands
I. Patras , Sch. of Electron. Eng. & Comput. Sci., Queen Mary Univ. of London, London, UK
ABSTRACT
We present a multimodal data set for the analysis of human affective states. The electroencephalogram (EEG) and peripheral physiological signals of 32 participants were recorded as each watched 40 one-minute long excerpts of music videos. Participants rated each video in terms of the levels of arousal, valence, like/dislike, dominance, and familiarity. For 22 of the 32 participants, frontal face video was also recorded. A novel method for stimuli selection is proposed using retrieval by affective tags from the last.fm website, video highlight detection, and an online assessment tool. An extensive analysis of the participants' ratings during the experiment is presented. Correlates between the EEG signal frequencies and the participants' ratings are investigated. Methods and results are presented for single-trial classification of arousal, valence, and like/dislike ratings using the modalities of EEG, peripheral physiological signals, and multimedia content analysis. Finally, decision fusion of the classification results from different modalities is performed. The data set is made publicly available and we encourage other researchers to use it for testing their own affective state estimation methods.
INDEX TERMS
Web sites, electroencephalography, emotion recognition, image classification, information retrieval, multimedia computing, neurophysiology, state estimation, video signal processing, state estimation methods, DEAP, emotion analysis, multimodal data set, human affective states, electroencephalogram, peripheral physiological signals, music videos, arousal, dominance, familiarity, frontal face video, stimuli selection, Web site, video highlight detection, online assessment tool, EEG signal frequencies, single-trial classification, multimedia content analysis, decision fusion, Videos, Databases, Electroencephalography, Motion pictures, Multimedia communication, Visualization, Face, affective computing., Emotion classification, EEG, physiological signals, signal processing, pattern classification
CITATION
S. Koelstra, C. Muhl, M. Soleymani, Jong-Seok Lee, A. Yazdani, T. Ebrahimi, T. Pun, A. Nijholt, I. Patras, "DEAP: A Database for Emotion Analysis ;Using Physiological Signals", IEEE Transactions on Affective Computing, vol.3, no. 1, pp. 18-31, Jan.-March 2012, doi:10.1109/T-AFFC.2011.15
REFERENCES
[1] M.K. Shan, F.F. Kuo, M.F. Chiang, and S.Y. Lee, “Emotion-Based Music Recommendation by Affinity Discovery from Film Music,” Expert Systems with Applications, vol. 36, no. 4, pp. 7666-7674, May 2009.
[2] M. Tkalčič, U. Burnik, and A. Košir, “Using Affective Parameters in a Content-Based Recommender System for Images,” User Modeling and User-Adapted Interaction, vol. 20, pp. 1-33-33, Sept. 2010.
[3] J.J.M. Kierkels, M. Soleymani, and T. Pun, “Queries and Tags in Affect-Based Multimedia Retrieval,” Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 1436-1439, 2009.
[4] M. Soleymani, J. Lichtenauer, T. Pun, and M. Pantic, “A Multi-Modal Database for Affect Recognition and Implicit Tagging,” IEEE Trans. Affective Computing, vol. 3, no. 1, pp. 42-55, Jan.-Mar. 2012.
[5] A. Savran, K. Ciftci, G. Chanel, J.C. Mota, L.H. Viet, B. Sankur, L. Akarun, A. Caplier, and M. Rombaut, “Emotion Detection in the Loop from Brain Signals and Facial Images,” Proc. eNTERFACE, July 2006.
[6] P. Ekman, W.V. Friesen, M. O'Sullivan, A. Chan, I. Diacoyanni-Tarlatzis, K. Heider, R. Krause, W.A. LeCompte, T. Pitcairn, and P.E. Ricci-Bitti, “Universals and Cultural Differences in the Judgments of Facial Expressions of Emotion,” J. Personality and Social Psychology, vol. 53, no. 4, pp. 712-717, Oct. 1987.
[7] W.G. Parrott, Emotions in Social Psychology: Essential Readings. Psychology Press, 2001.
[8] R. Plutchik, “The Nature of Emotions,” Am. Scientist, vol. 89, p. 344, 2001.
[9] J.A. Russell, “A Circumplex Model of Affect,” J. Personality and Social Psychology, vol. 39, no. 6, pp. 1161-1178, 1980.
[10] M.M. Bradley and P.J. Lang, “Measuring Emotion: The Self-Assessment Manikin and the Semantic Differential,” J. Behavior Therapy Experimental Psychiatry, vol. 25, no. 1, pp. 49-59, Mar. 1994.
[11] M. Pantic, M. Valstar, R. Rademaker, and L. Maat, “Web-Based Database for Facial Expression Analysis,” Proc. Int'l Conf. Multimedia and Expo, pp. 317-321, 2005.
[12] E. Douglas-Cowie, R. Cowie, and M. Schröder, “A New Emotion Database: Considerations, Sources and Scope,” Proc. Int'l Symp. Computer Architecture, pp. 39-44, 2000.
[13] H. Gunes and M. Piccardi, “A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior,” Proc. 18th Int'l Conf. Pattern Recognition, vol. 1, pp. 1148-1153, 2006.
[14] G. Fanelli, J. Gall, H. Romsdorfer, T. Weise, and L. Van Gool, “A 3-D Audio-Visual Corpus of Affective Communication,” IEEE Trans. Multimedia, vol. 12, no. 6, pp. 591-598, Oct. 2010.
[15] M. Grimm, K. Kroschel, and S. Narayanan, “The Vera am Mittag German Audio-Visual Emotional Speech Database,” Proc. Int'l Conf. Multimedia and Expo, pp. 865-868, 2008.
[16] J.A. Healey, “Wearable and Automotive Systems for Affect Recognition from Physiology,” PhD dissertation, MIT, 2000.
[17] J.A. Healey and R.W. Picard, “Detecting Stress during Real-World Driving Tasks Using Physiological Sensors,” IEEE Trans. Intelligent Transportation Systems, vol. 6, no. 2, pp. 156-166, June 2005.
[18] P. Lang, M. Bradley, and B. Cuthbert, “International Affective Picture System (IAPS): Affective Ratings of Pictures and Instruction Manual,” Technical Report A-8, Univ. of Florida, 2008.
[19] Z. Zeng, M. Pantic, G.I. Roisman, and T.S. Huang, “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
[20] P. Lang, M. Greenwald, M. Bradely, and A. Hamm, “Looking at Pictures—Affective, Facial, Visceral, and Behavioral Reactions,” Psychophysiology, vol. 30, no. 3, pp. 261-273, May 1993.
[21] J. Kim and E. André, “Emotion Recognition Based on Physiological Changes in Music Listening,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008.
[22] J. Wang and Y. Gong, “Recognition of Multiple Drivers' Emotional State,” Proc. Int'l Conf. Pattern Recognition, pp. 1-4, 2008.
[23] C.L. Lisetti and F. Nasoz, “Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals,” EURASIP J. Applied Signal Processing, vol. 2004, no. 1, pp. 1672-1687, Jan. 2004.
[24] G. Chanel, J. Kierkels, M. Soleymani, and T. Pun, “Short-Term Emotion Assessment in a Recall Paradigm,” Int'l J. Human-Computer Studies, vol. 67, no. 8, pp. 607-627, Aug. 2009.
[25] J. Kierkels, M. Soleymani, and T. Pun, “Queries and Tags in Affect-Based Multimedia Retrieval,” Proc. Int'l Conf. Multimedia and Expo, pp. 1436-1439, June 2009.
[26] M. Soleymani, G. Chanel, J.J.M. Kierkels, and T. Pun, “Affective Characterization of Movie Scenes Based on Content Analysis and Physiological Changes,” Int'l J. Semantic Computing, vol. 3, no. 2, pp. 235-254, June 2009.
[27] A. Yazdani, J.-S. Lee, and T. Ebrahimi, “Implicit Emotional Tagging of Multimedia Using EEG Signals and Brain Computer Interface,” Proc. SIGMM Workshop Social Media, pp. 81-88, 2009.
[28] P. Ekman, W. Friesen, M. Osullivan, A. Chan, I. Diacoyannitarlatzis, K. Heider, R. Krause, W. Lecompte, T. Pitcairn, P. Riccibitti, K. Scherer, M. Tomita, and A. Tzavaras, “Universals and Cultural-Differences in the Judgments of Facial Expressions of Emotion,” J. Personality and Social Psychology, vol. 53, no. 4, pp. 712-717, Oct. 1987.
[29] A. Hanjalic and L.-Q. Xu, “Affective Video Content Representation and Modeling,” IEEE Trans. Multimedia, vol. 7, no. 1, pp. 143-154, Feb. 2005.
[30] H.L. Wang and L.-F. Cheong, “Affective Understanding in Film,” IEEE Trans. Circuits and Systems for Video Technology, vol. 16, no. 6, pp. 689-704, June 2006.
[31] M. Soleymani, J. Kierkels, G. Chanel, and T. Pun, “A Bayesian Framework for Video Affective Representation,” Proc. Int'l Conf. Affective Computing and Intelligent Interaction, pp. 1-7, Sept. 2009.
[32] D. Liu, “Automatic Mood Detection from Acoustic Music Data,” Proc. Int'l Conf. Music Information Retrieval, pp. 13-17, 2003.
[33] L. Lu, D. Liu, and H.-J. Zhang, “Automatic Mood Detection and Tracking of Music Audio Signals,” IEEE Trans. Audio, Speech, and Language Processing, vol. 14, no. 1, pp. 5-18, Jan. 2006.
[34] Y.-H. Yang and H.H. Chen, “Music Emotion Ranking,” Proc. Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 1657-1660, 2009.
[35] S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J.-S. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, “Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos,” Proc. Brain Informatics, pp. 89-100, 2010.
[36] M.E. Tipping, “Sparse Bayesian Learning and the Relevance Vector Machine,” J. Machine Learning Research, vol. 1, pp. 211-244, June 2001.
[37] J.D. Morris, “SAM: The Self-Assessment Manikin an Efficient Cross-Cultural Measurement of Emotional Response,” J. Advertising Research, vol. 35, no. 8, pp. 63-68, 1995.
[38] M. Bradley and P. Lang, “International Affective Digitized Sounds (IADS): Stimuli, Instruction Manual and Affective Ratings,” Technical Report B-2, The Center for Research in Psychophysiology, Univ. of Florida, 1999.
[39] N. Lazar, “Combining Brains: A Survey of Methods for Statistical Pooling of Information,” NeuroImage, vol. 16, no. 2, pp. 538-550, June 2002.
[40] T.M. Loughin, “A Systematic Comparison of Methods for Combining $p$ -Values from Independent Tests,” Computational Statistics and Data Analysis, vol. 47, pp. 467-485, 2004.
[41] R.J. Barry, A.R. Clarke, S.J. Johnstone, C.A. Magee, and J.A. Rushby, “EEG Differences between Eyes-Closed and Eyes-Open Resting Conditions,” Clinical Neurophysiology, vol. 118, no. 12, pp. 2765-2773, Dec. 2007.
[42] R.J. Barry, A.R. Clarke, S.J. Johnstone, and C.R. Brown, “EEG Differences in Children between Eyes-Closed and Eyes-Open Resting Conditions,” Clinical Neurophysiology, vol. 120, no. 10, pp. 1806-1811, Oct. 2009.
[43] W. Klimesch, P. Sauseng, and S. Hanslmayr, “EEG Alpha Oscillations: The Inhibition-Timing Hypothesis,” Brain Research Rev., vol. 53, no. 1, pp. 63-88, Jan. 2007.
[44] H. Cole and W.J. Ray, “EEG Correlates of Emotional Tasks Related to Attentional Demands,” Int'l J. Psychophysiology, vol. 3, no. 1, pp. 33-41, July 1985.
[45] J. Onton and S. Makeig, “High-Frequency Broadband Modulations of Electroencephalographic Spectra,” Frontiers in Human Neuroscience, vol. 3, 2009.
[46] I. Goncharova, D.J. McFarland, J.R. Vaughan, and J.R. Wolpaw, “EMG Contamination of EEG: Spectral and Topographical Characteristics,” Clinical Neurophysiology, vol. 114, no. 9, pp. 1580-1593, Sept. 2003.
[47] E. Harmon-Jones, “Clarifying the Emotive Functions of Asymmetrical Frontal Cortical Activity,” Psychophysiology, vol. 40, no. 6, pp. 838-848, 2003.
[48] R.R. Cornelius, The Science of Emotion. Research and Tradition in the Psychology of Emotion. Prentice Hall, 1996.
[49] D. Sander, D. Grandjean, and K.R. Scherer, “A Systems Approach to Appraisal Mechanisms in Emotion,” Neural Networks, vol. 18, no. 4, pp. 317-352, 2005.
[50] R. McCraty, M. Atkinson, W. Tiller, G. Rein, and A. Watkins, “The Effects of Emotions on Short-Term Power Spectrum Analysis of Heart Rate Variability,” Am. J. Cardiology, vol. 76, no. 14, pp. 1089-1093, 1995.
[51] S.D. Kreibig, “Autonomic Nervous System Activity in Emotion: A Review,” Biological Psychology, vol. 84, no. 3, pp. 394-421, 2010.
[52] G. Chanel, J.J.M. Kierkels, M. Soleymani, and T. Pun, “Short-Term Emotion Assessment in a Recall Paradigm,” Int'l J. Human-Computer Studies, vol. 67, no. 8, pp. 607-627, Aug. 2009.
[53] J. Kim and E. André, “Emotion Recognition Based on Physiological Changes in Music Listening,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008.
[54] P. Rainville, A. Bechara, N. Naqvi, and A.R. Damasio, “Basic Emotions Are Associated with Distinct Patterns of Cardiorespiratory Activity.” Int'l J. Psychophysiology, vol. 61, no. 1, pp. 5-18, July 2006.
[55] P. Kelm, S. Schmiedeke, and T. Sikora, “Feature-Based Video Key Frame Extraction for Low Quality Video Sequences,” Proc. Int'l Workshop Image Analysis for Multimedia Interactive Services, pp. 25-28, May 2009.
[56] Z. Rasheed, Y. Sheikh, and M. Shah, “On the Use of Computable Features for Film Classification,” IEEE Trans. Circuits and Systems for Video Technology, vol. 15, no. 1, pp. 52-64, Jan. 2005.
[57] P. Valdez and A. Mehrabian, “Effects of Color on Emotions,” J. Experimental Psychology, vol. 123, no. 4, pp. 394-409, Dec. 1994.
[58] R.W. Picard, Affective Computing. MIT Press, Sept. 1997.
[59] D. Li, I.K. Sethi, N. Dimitrova, and T. McGee, “Classification of General Audio Data for Content-Based Retrieval,” Pattern Recognition Letters, vol. 22, no. 5, pp. 533-544, 2001.
[60] L. Lu, H. Jiang, and H. Zhang, “A Robust Audio Classification and Segmentation Method,” Proc. Ninth ACM Int'l Conf. Multimedia, pp. 203-211, 2001.
[61] P. Boersma, “Praat, A System for Doing Phonetics by Computer,” Glot Int'l, vol. 5, nos. 9/10, pp. 341-345, 2001.
[62] L. Chen, S. Gunduz, and M. Ozsu, “Mixed Type Audio Classification with Support Vector Machine,” Proc. Int'l Conf. Multimedia and Expo, pp. 781 -784, July 2006.
[63] J.-S. Lee and C.H. Park, “Robust Audio-Visual Speech Recognition Based on Late Integration,” IEEE Trans. Multimedia, vol. 10, no. 5, pp. 767-779, Aug. 2008.
90 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool