This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
ECG Pattern Analysis for Emotion Detection
Jan.-March 2012 (vol. 3 no. 1)
pp. 102-115
F. Agrafioti, Dept. of Electr. & Comput. Eng., Univ. of Toronto, Toronto, ON, Canada
D. Hatzinakos, Dept. of Electr. & Comput. Eng., Univ. of Toronto, Toronto, ON, Canada
A. K. Anderson, Dept. of Psychol., Univ. of Toronto, Toronto, ON, Canada
Emotion modeling and recognition has drawn extensive attention from disciplines such as psychology, cognitive science, and, lately, engineering. Although a significant amount of research has been done on behavioral modalities, less explored characteristics include the physiological signals. This work brings to the table the ECG signal and presents a thorough analysis of its psychological properties. The fact that this signal has been established as a biometric characteristic calls for subject-dependent emotion recognizers that capture the instantaneous variability of the signal from its homeostatic baseline. A solution based on the empirical mode decomposition is proposed for the detection of dynamically evolving emotion patterns on ECG. Classification features are based on the instantaneous frequency (Hilbert-Huang transform) and the local oscillation within every mode. Two experimental setups are presented for the elicitation of active arousal and passive arousal/valence. The results support the expectations for subject specificity, as well as demonstrating the feasibility of determining valence out of the ECG morphology (up to 89 percent for 44 subjects). In addition, this work differentiates for the first time between active and passive arousal, and advocates that there are higher chances of ECG reactivity to emotion when the induction method is active for the subject.

[1] R.W. Picard, Affective Computing. MIT Press, July 2000.
[2] P. Ekman, R.W. Levenson, and W.V. Friesen, “Autonomic Nervous System Activity Distinguishes among Emotions,” Science, vol. 221, pp. 1208-1210, Sept. 1983.
[3] F. Hönig, A. Batliner, and E. Nöth, “Real-Time Recognition of the Affective User State with Physiological Signals,” Proc. Doctoral Consortium Conf. Affective Computing and Intelligent Interaction, pp. 1-8, 2007.
[4] J. Anttonen and V. Surakka, “Emotions and Heart Rate While Sitting on a Chair,” Proc. SIGCHI Conf. Human Factors in Computing Systems, pp. 491-499. 2005,
[5] C.M. Jones and T. Troen, “Biometric Valence and Arousal Recognition,” Proc. 19th Australasian Conf. Computer-Human Interaction, pp. 191-194, 2007.
[6] R.L. Mandryk, T.W. Inkpen, and K.M. Calvert, “Using Psychophysiological Techniques to Measure User Experience with Entertainment Technologies,” Behaviour and Information Technology, vol. 25, no. 2, pp. 141-158, Mar.-Apr. 2006.
[7] R. Sinha, W.R. Lovallo, and O.A. Parsons, “Cardiovascular Differentiation of Emotions,” Psychosomatic Medicine, vol. 54, pp. 422-435, 1992.
[8] R.W. Picard, E. Vyzas, and J. Healey, “Toward Machine Emotional Intelligence: Analysis of Affective Physiological State,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001.
[9] J. Kim and E. André, “Emotion Recognition Based on Physiological Changes in Music Listening,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008.
[10] F. Nasoz, K. Alvarez, C.L. Lisetti, and N. Finkelstein, “Emotion Recognition from Physiological Signals Using Wireless Sensors for Presence Technologies,” Int'l J. Cognition, Technology, and Work, special issue on presence, vol. 6, no. 1, pp. 4-14, Feb. 2004.
[11] K. Kim, S. Bang, and S. Kim, “Emotion Recognition System Using Short-Term Monitoring of Physiological Signals,” Medical and Biological Eng. and Computing, vol. 42, no. 3, pp. 419-427, May 2004.
[12] R.L. Mandryk and M.S. Atkins, “A Fuzzy Physiological Approach for Continuously Modeling Emotion during Interaction with Play Technologies,” Int'l J. Human Computer Studies, vol. 65, no. 4, pp. 329-347, 2007.
[13] P.J. Lang, M.M. Bradley, and B.N. Cuthbert, “Int. Affective Picture System (IAPS): Instruction Manual and Affective Ratings,” Technical Report A-5, 2001.
[14] A. Haag, S. Goronzy, P. Schaich, and J. Williams, “Emotion Recognition Using Bio-Sensors: First S.gif towards an Automatic System,” Affective Dialogue Systems, vol. 3068, pp. 36-48, 2004.
[15] L. Sornmo and P. Laguna, Bioelectrical Signal Processing in Cardiac and Neurological Applications. Elsevier, 2005.
[16] J.T. Catalano, Guide to ECG Analysis. J.B. Lippincott, 1993.
[17] R. Shouldice, C. Heneghan, P. Nolan, and P.G. Nolan, “PR and PP ECG Intervals as Indicators of Autonomic Nervous Innervation of the Cardiac Sinoatrial and Atrioventricular Nodes,” Proc. First IEEE Int'l Conf. Neural Eng., pp. 261-264, Mar. 2003.
[18] R. Virtanen, A. Jula, J.K. Salminen, L.M. Voipio-Pulkki, H. Helenius, T. Kuusela, and J. Airaksinen, “Anxiety and Hostility Are Associated with Reduced Baroreflex Sensitivity and Increased Beat-to-Beat Blood Pressure Variability,” Psychosomatic Medicine, vol. 65, pp. 751-756, 2003.
[19] S. Booth-Kewley and H.S. Friedman, “Psychological Predictors of Heart Disease: A Quantitative Review,” Psychological Bull., vol. 101, no. 3, pp. 343-362, 1987.
[20] G.H.E. Gendolla, M. Richter, and A. Friedrich, “Task Difficulty Effects on Cardiac Activity,” Psychophysiology, no. 45, pp. 869-875, 2008.
[21] J. Blascovich, M.D. Seery, C.A. Mugridge, R.K. Norris, and M. Weisbuch, “Predicting Athletic Performance from Cardiovascular Indexes of Challenge and Threat,” J. Experimental Social Psychology, vol. 40, no. 5, pp. 683-688, 2004.
[22] H. Ue, I. Masuda, Y. Yoshitake, Y. Inazumi, and T. Moritani, “Assessment of Cardiac Autonomic Nervous Activities by Means of ECG R-R Interval Power Spectral Analysis and Cardiac Depolarization-Repolarization Process,” Annals of Noninvasive Electrocardiology, vol. 5, no. 4, pp. 336-345, 2000.
[23] W.L. Wasmund, E.C. Westerholm, D.E. Watenpaugh, S.L. Wasmund, and M.L. Smith, “Interactive Effects of Mental and Physical Stress on Cardiovascular Control,” J. Applied Physiology, vol. 92, pp. 1828-1834, 2002.
[24] R. Pramila, J. Sims, R. Brackin, and N. Sarkar, “Online Stress Detection Using Psychophysiological Signals for Implicit Human-Robot Cooperation,” Robotica, vol. 20, no. 6, pp. 673-685, 2002.
[25] M. Dambacher, W. Eichinger, K. Theisen, and A.W. Frey, “RT and Systolic Blood Pressure Variability after Sympathetic Stimulation during Positive Tilt in Healthy Volunteers,” Proc. Computers in Cardiology, pp. 573-576, Sept. 1994.
[26] G. Andrassy, A. Szabo, G. Ferencz, Z. Trummer, E. Simon, and A. Tahy, “Mental Stress May Induce QT-Interval Prolongation and T-Wave Notching,” Annals of Noninvasive Electrocardiology, vol. 12, no. 3, pp. 251-259, 2007.
[27] A.F. Folino, G. Buja, P. Turrini, L. Oselladore, and A. Nava, “The Effects of Sympathetic Stimulation Induced by Mental Stress on Signal Averages Electrocardiogram,” Int'l J. Cardiology, vol. 48, pp. 279-285, 1995.
[28] A. Szabo, “The Combined Effects of Orthostatic and Mental Stress on Heart Rate, T-Wave Amplitude, and Pulse Transit Time,” European J. Applied Physiology, vol. 67, no. 6, pp. 540-544, 1993.
[29] H. Scher, J.J. Furedy, and R.J. Heslegrave, “Phasic T-Wave Amplitude and Heart Rate Changes as Indices of Mental Effort and Task Incentive,” Psychophysiology, vol. 21, no. 3, pp. 326-333, 1984.
[30] F. Agrafioti, F.M. Bui, and D. Hatzinakos, “Medical Biometrics: The Perils of Ignoring Time Dependency,” Proc. IEEE Third Int'l Conf. Biometrics: Theory, Applications, and Systems, pp. 1-6, Sept. 2009.
[31] S.A. Israel, J.M. Irvine, A. Cheng, M.D. Wiederhold, and B.K. Wiederhold, “ECG to Identify Individuals,” Pattern Recognition, vol. 38, no. 1, pp. 133-142, 2005.
[32] R. Hoekema, G. Uijen, and A. van Oosterom, “Geometrical Aspect of the Interindividual Variaility of Multilead ECG Recordings,” IEEE Trans. Biomedical Eng., vol. 48, no. 5, pp. 551-559, May 2001.
[33] L.S. Green, R.L. Lux, C.W. Hawsand, R. Williams, S. Hunt, and M. Burgess, “Effects of Age, Sex, and Body Habitus on QRS and ST-T Potential Maps of 1100 Normal Subjects,” Circulation, vol. 85, pp. 244-253, 1985.
[34] G. Kozmann, R.L. Lux, and L.S. Green, “Sources of Variability in Normal Body Surface Potential Maps,” Circulation, vol. 17, pp. 1077-1083, 1989.
[35] B.P. Simon and C. Eswaran, “An ECG Classifier Designed Using Modified Decision Based Neural Network,” Computers and Biomedical Research, vol. 30, pp. 257-272, 1997.
[36] C. Zong and M. Chetouani, “Hilbert-Huang Transform Based Physiological Signals Analysis for Emotion Recognition,” Proc. IEEE Int'l Symp. Signal Processing and Information Technology, pp. 334-339, Dec. 2009.
[37] N.E. Huang, Z. Shen, R.R. Long, M.L. Wu, Q. Zheng, N.C. Yen, and C.C. Tung, “The Empirical Mode Decomposition and Hilbert Spectrum for Nonlinear and Nonstationary Time Series Analysis,” Proc. Royal Soc. London, vol. 454, pp. 903-995, 1998.
[38] G. Rilling, P. Flandrin, P. Gonalves, and J.M. Lilly, “Bivariate Empirical Mode Decomposition,” IEEE Signal Processing Letters, vol. 14, no. 12, pp. 936-939, Dec. 2007.
[39] J. Pan and W.J. Tompkins, “A Real-Time QRS Detection Algorithm,” IEEE Trans. Biomedical Eng., vol. 32, no. 3, pp. 230-236, Mar. 1985.
[40] P.E. McSharry, G.D. Clifford, L. Tarassenko, and L.A. Smith, “A Dynamical Model for Generating Synthetic Electrocardiogram Signals,” IEEE Trans. Biomedical Eng., vol. 50, no. 3, pp. 289-294, Mar. 2003.
[41] M. Blanco-Velasco, B. Weng, and K.E. Barner, “ECG Signal Denoising and Baseline Wander Correction Based on the Empirical Mode Decomposition,” Computers in Biology and Medicine, vol. 38, pp. 1-13, Jan. 2008.
[42] A. Arafat and K. Hasan, “Automatic Detection of ECG Wave Boundaries Using Empirical Mode Decomposition,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 461-464, Apr. 2009.
[43] A.J. Nimunkar and W.J. Tompkins, “R-Peak Detection and Signal Averaging for Simulated Stress ECG Using EMD,” Proc. 29th Ann. IEEE Int'l Conf. Eng. in Medicine and Biology Soc., pp. 1261-1264, Aug. 2007.
[44] P. Flandrin, P. Goncalves, and G. Rilling, “Detrending and Denoising with Empirical Mode Decompositions,” Proc. XII EUSIPCO, Sept. 2004.
[45] M.K.L. Molla, T. Tanaka, T.M. Rutkowski, and A. CiChocki, “Separation of EOG Artifacts from EEG Signals Using Bivatiate EMD,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 562-565, Mar. 2010.
[46] R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schroder, “FEELTRACE: An Instrument for Recording Perceived Emotion in Real Time,” Proc. ISCA Tutorial and Research Workshop Speech and Emotion, pp. 19-24, 2000,
[47] H. Rau, “Responses of the T-Wave Amplitude as a Function of Active and Passive Tasks and Beta-Adrenergic Blockade,” Psychophysiology, vol. 28, no. 2, pp. 231-239, 1991.
[48] S.M.A. Bhuiyan, R.R. Adhami, and J.F. Khan, “A Novel Approach of Fast and Adaptive Bidimensional Empirical Mode Decomposition,” Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 1313-1316, Apr. 2008.
[49] S.M.A. Bhuiyan, R.R. Adhami, and J.F. Khan, “Fast and Adaptive Bidimensional Empirical Mode Decomposition Using Order-Statistics Filter Based Envelope Estimation,” EURASIP J. Advances in Signal Processing, vol. 2008, pp. 1-18, 2008.

Index Terms:
psychology,electrocardiography,emotion recognition,Hilbert transforms,medical signal processing,induction method,ECG pattern analysis,emotion detection,emotion modeling,emotion recognition,psychology,cognitive science,engineering,behavioral modality,physiological signal,ECG signal,psychological property,biometric characteristic,instantaneous frequency,Hilbert-Huang transform,local oscillation,ECG morphology,active arousal,passive arousal,ECG reactivity,Electrocardiography,Physiology,Muscles,Emotion recognition,Heart rate variability,Stress,oscillation.,Electrocardiogram,emotion recognition,affective computing,arousal,valence,active stress,passive stress,bivariate empirical mode decomposition,intrinsic mode function,instantaneous frequency
Citation:
F. Agrafioti, D. Hatzinakos, A. K. Anderson, "ECG Pattern Analysis for Emotion Detection," IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 102-115, Jan.-March 2012, doi:10.1109/T-AFFC.2011.28
Usage of this product signifies your acceptance of the Terms of Use.