This Article 
 Bibliographic References 
 Add to: 
The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition
April-June 2012 (vol. 3 no. 2)
pp. 237-249
A. Lanata, Interdept. Res. Center E. Piaggio, Univ. of Pisa, Pisa, Italy
G. Valenza, Interdept. Res. Center E. Piaggio, Univ. of Pisa, Pisa, Italy
E. P. Scilingo, Interdept. Res. Center E. Piaggio, Univ. of Pisa, Pisa, Italy
This paper reports on a new methodology for the automatic assessment of emotional responses. More specifically, emotions are elicited in agreement with a bidimensional spatial localization of affective states, that is, arousal and valence dimensions. A dedicated experimental protocol was designed and realized where specific affective states are suitably induced while three peripheral physiological signals, i.e., ElectroCardioGram (ECG), ElectroDermal Response (EDR), and ReSPiration activity (RSP), are simultaneously acquired. A group of 35 volunteers was presented with sets of images gathered from the International Affective Picture System (IAPS) having five levels of arousal and five levels of valence, including a neutral reference level in both. Standard methods as well as nonlinear dynamic techniques were used to extract sets of features from the collected signals. The goal of this paper is to implement an automatic multiclass arousal/valence classifier comparing performance when extracted features from nonlinear methods are used as an alternative to standard features. Results show that, when nonlinearly extracted features are used, the percentages of successful recognition dramatically increase. A good recognition accuracy (>;90 percent) after 40-fold cross-validation steps for both arousal and valence classes was achieved by using the Quadratic Discriminant Classifier (QDC).

[1] J. Russell and J.M. Carroll, "On the Bipolarity of Positive and Negative Affect," Psychological Bull., vol. 125, no. 1, pp. 3-30, 1999.
[2] D. Watson, D. Wiese, J. Vaidya, and A. Tellegen, "The Two General Activation Systems of Affect: Structural Findings, Evolutionary Considerations, and Psychobiological Evidence," J. Personality and Social Psychology, vol. 76, no. 5, pp. 820-838, 1999.
[3] D. Watson and L. Clark, "On Traits and Temperament: General and Specific Factors of Emotional Experience and Their Relation to the Five-Factor Model," J. Personality, vol. 60, no. 2, pp. 441-476, 1992.
[4] C. Darwin, The Expression of the Emotions in Man and Animals; with an Introduction, Afterword, and Commentaries by Paul Ekman. Oxford Univ. Press, 1872.
[5] P. Ekman, "Universal Facial Expressions of Emotion," Culture and Personality: Contemporary Readings, p. 8, Aldine Pub. Co., 1974.
[6] P. Ekman, "Basic Emotions," Handbook of Cognition and Emotion, pp. 45-60, John Wiley & Sons, 1999.
[7] S. Tompkins, Affect Imagery Consciousness: The Positive Affects, vol. 1. Springer Publishing Company, 1962.
[8] C. Izard, The Face of Emotion, vol. 23. Appleton-Century-Crofts, 1971.
[9] R. Plutchik, "Emotions: A General Psychoevolutionary Theory," Approaches to Emotion, pp. 197-219, L. Erlbaum Assoc., 1984.
[10] P. Ekman, "Cross-Cultural Studies of Facial Expression," Darwin and Facial Expression: A Century of Research in Review, pp. 169-222, Academic Press, 1973.
[11] J. Watson, Behaviorism. Transaction Publishers, 1997.
[12] A. Ortony and T. Turner, "What Is Basic about Basic Emotions," Psychological Rev., vol. 97, no. 3, pp. 315-331, 1990.
[13] W. Wundt, Grundriss der Psychologie [Fundamentals of Psychology], seventh rev. ed. Engelman, 1905.
[14] H. Schlosberg, "Three Dimensions of Emotion," Psychological Rev., vol. 61, no. 2, pp. 81-88, 1954.
[15] C. Osgood, The Measurement of Meaning. Univ. of Illinois Press, 1975.
[16] J. Davitz, The Language of Emotion. Academic Press, 1969.
[17] P. Lang, M. Bradley, and B. Cuthbert, "Emotion, Motivation, and Anxiety: Brain Mechanisms and Psychophysiology," Biological Psychiatry, vol. 44, no. 12, pp. 1248-1263, 1998.
[18] J. Panskepp, Affective Neuroscience: The Foundations of Human and Animal Emotions. Oxford Univ. Press, 1998.
[19] C. Breazeal, "Emotion and Sociable Humanoid Robots," Int'l J. Human-Computer Studies, vol. 59, nos. 1/2, pp. 119-155, 2003.
[20] J. Russell, "A Circumplex Model of Affect," J. Personality and Social Psychology, vol. 39, no. 6, pp. 1161-1178, 1980.
[21] J. Russell and A. Mehrabian, "Evidence for a Three-Factor Theory of Emotions* 1," J. Research in Personality, vol. 11, no. 3, pp. 273-294, 1977.
[22] M. Arnold, An Excitatory Theory of Emotion. McGraw-Hill, 1950.
[23] N. Frijda, The Emotions. Cambridge Univ. Press, 1986.
[24] A. Ortony, G. Clore, and A. Collins, The Cognitive Structure of Emotions, Cambridge Univ. Press, 1990.
[25] K. Scherer and P. Ekman, Approaches to Emotions. Lawrence Erlbaum Assoc., Publisher, 1984.
[26] C. Lisetti and P. Gmytrasiewicz, "Can a Rational Agent Afford to Be Affectless? A Formal Approach," Applied Artificial Intelligence, vol. 16, pp. 1-33, 2002.
[27] K. Scherer, A. Schorr, and T. Johnstone, Appraisal Processes in Emotion: Theory, Methods, Research, Oxford Univ. Press, 2001.
[28] A. Egges, S. Kshirsagar, and N. Magnenat-Thalmann, "A Model for Personality and Emotion Simulation," Proc. Knowledge-Based Intelligent Information and Eng. Systems, pp. 453-461, 2003.
[29] J. Posner, J. Russell, and B. Peterson, "The Circumplex Model of Affect: An Integrative Approach to Affective Neuroscience, Cognitive Development, and Psychopathology," Development and Psychopathology, vol. 17, no. 3, pp. 715-734, 2005.
[30] L. Bialoskorski, J. Westerink, and E. Broek, "Mood Swings: An Affective Interactive Art System," Proc. Third Int'l Conf. Intelligent Technologies for Interactive Entertainment, pp. 181-186, 2009.
[31] C. Lisetti and F. Nasoz, "Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals," J. Applied Signal Processing, vol. 2004, pp. 1672-1687, 2004.
[32] J. Kim and E. André, "Emotion Recognition Based on Physiological Changes in Music Listening," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 12, pp. 2067-2083, Dec. 2008.
[33] J. Janssen, E. Van den Broek, and J. Westerink, "Personalized Affective Music Player," Proc. Third Int'l Conf. Affective Computing and Intelligent Interaction and Workshops, pp. 1-6, 2009.
[34] Y. Lin, C. Wang, T. Jung, T. Wu, S. Jeng, J. Duann, and J. Chen, "EEG-Based Emotion Recognition in Music Listening," IEEE Trans. Biomedical Eng., vol. 57, no. 7, pp. 1798-1806, July 2010.
[35] E. Van den Broek, M. Schut, J. Westerink, and K. Tuinenbreijer, "Unobtrusive Sensing of Emotions (USE)," J. Ambient Intelligence and Smart Environments, vol. 1, no. 3, pp. 287-299, 2009.
[36] E. Van den Broek and J. Westerink, "Considerations for Emotion-Aware Consumer Products," Applied Ergonomics, vol. 40, no. 6, pp. 1055-1064, 2009.
[37] Z. Zeng, M. Pantic, G. Roisman, and T. Huang, "A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
[38] K. Poels and S. Dewitte, "How to Capture the Heart? Reviewing 20 Years of Emotion Measurement in Advertising," J. Advertising Research, vol. 46, no. 1, p. 18, 2006.
[39] E. Leon, G. Clarke, V. Callaghan, and F. Sepulveda, "A User-Independent Real-Time Emotion Recognition System for Software Agents in Domestic Environments," Eng. Applications of Artificial Intelligence, vol. 20, no. 3, pp. 337-345, 2007.
[40] G. Chanel, J. Kierkels, M. Soleymani, and T. Pun, "Short-Term Emotion Assessment in a Recall Paradigm," Int'l J. Human-Computer Studies, vol. 67, no. 8, pp. 607-627, 2009.
[41] J. Healey, "Affect Detection in the Real World: Recording and Processing Physiological Signals," Proc. Third Int'l Conf. Affective Computing and Intelligent Interaction and Workshops, pp. 1-6, 2009.
[42] J. Healey and R. Picard, "Detecting Stress during Real-World Driving Tasks Using Physiological Sensors," IEEE Trans. Intelligent Transportation Systems, vol. 6, no. 2, pp. 156-166, June 2005.
[43] P. Lang, M. Bradley, and B. Cuthbert, "International Affective Picture System IAPS): Digitized Photographs, Instruction Manual and Affective Ratings," Technical Report A-6, Univ.of Florida, 2005.
[44] P. Lang, M. Bradley, and B. Cuthbert, "International Affective Picture System (IAPS): Technical Manual and Affective Ratings," NIMH Center for the Study of Emotion and Attention, 1997.
[45] P. Lang, M. Greenwald, M. Bradley, and A. Hamm, "Looking at Pictures: Affective, Facial, Visceral, and Behavioral Reactions," Psychophysiology, vol. 30, no. 3, pp. 261-273, 1993.
[46] P. Lang et al., "Behavioral Treatment and Bio-Behavioral Assessment: Computer Applications," Technology in Mental Health Care Delivery Systems, pp. 119-137, Ablex Pub. Corp., 1980.
[47] S. Grimm, C. Schmidt, F. Bermpohl, A. Heinzel, Y. Dahlem, M. Wyss, D. Hell, P. Boesiger, H. Boeker, and G. Northoff, "Segregated Neural Representation of Distinct Emotion Dimensions in the Prefrontal Cortex—An fMRI Study," Neuroimage, vol. 30, no. 1, pp. 325-340, 2006.
[48] A. Hariri, V. Mattay, A. Tessitore, F. Fera, and D. Weinberger, "Neocortical Modulation of the Amygdala Response to Fearful Stimuli," Biological Psychiatry, vol. 53, no. 6, pp. 494-501, 2003.
[49] R. Picard, E. Vyzas, and J. Healey, "Toward Machine Emotional Intelligence: Analysis of Affective Physiological State," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001.
[50] C. Lisetti and F. Nasoz, "Using Noninvasive Wearable Computers to Recognize Human Emotions from Physiological Signals," EURASIP J. Applied Signal Processing, vol. 2004, pp. 1672-1687, 2004.
[51] A. Haag, S. Goronzy, P. Schaich, and J. Williams, "Emotion Recognition Using Bio-Sensors: First Steps towards an Automatic System," Affective Dialogue Systems, vol. 1, pp. 36-48, 2004.
[52] K. Kim, S. Bang, and S. Kim, "Emotion Recognition System Using Short-Term Monitoring of Physiological Signals," Medical and Biological Eng. and Computing, vol. 42, no. 3, pp. 419-427, 2004.
[53] S. Yoo, C. Lee, Y. Park, N. Kim, B. Lee, and K. Jeong, "Neural Network Based Emotion Estimation Using Heart Rate Variability and Skin Resistance," Proc. First Int'l Conf. Advances in Natural Computation, pp. 818-824, 2005.
[54] A. Choi and W. Woo, "Physiological Sensing and Feature Extraction for Emotion Recognition by Exploiting Acupuncture Spots," Proc. First Int'l Conf. Affective Computing and Intelligent Interaction, pp. 590-597, 2005.
[55] L. Li and J. Chen, "Emotion Recognition Using Physiological Signals," Pro. 16th Int'l Conf. Advances in Artificial Reality and Tele-Existence, pp. 437-446, 2006.
[56] P. Rani, C. Liu, N. Sarkar, and E. Vanman, "An Empirical Study of Machine Learning Techniques for Affect Recognition in Human-Robot Interaction," Pattern Analysis & Applications, vol. 9, no. 1, pp. 58-69, 2006.
[57] P. Rainville, A. Bechara, N. Naqvi, and A. Damasio, "Basic Emotions Are Associated with Distinct Patterns of Cardiorespiratory Activity," Int'l J. Psychophysiology, vol. 61, no. 1, pp. 5-18, 2006.
[58] J. Zhai and A. Barreto, "Stress Detection in Computer Users Based on Digital Signal Processing of Noninvasive Physiological Variables," Proc. IEEE 28th Ann. Int'l Conf. Eng. in Medicine and Biology Soc., pp. 1355-1358, 2006.
[59] C. Liu, K. Conn, N. Sarkar, and W. Stone, "Physiology-Based Affect Recognition for Computer-Assisted Intervention of Children with Autism Spectrum Disorder," Int'l J. Human-Computer Studies, vol. 66, no. 9, pp. 662-677, 2008.
[60] C. Katsis, N. Katertsidis, G. Ganiatsas, and D. Fotiadis, "Toward Emotion Recognition in Car-Racing Drivers: A Biosignal Processing Approach," IEEE Trans. Systems, Man, and Cybernetics, Part A: Systems and Humans, vol. 38, no. 3, pp. 502-512, May 2008.
[61] G. Yannakakis and J. Hallam, "Entertainment Modeling through Physiology in Physical Play," Int'l J. Human-Computer Studies, vol. 66, no. 10, pp. 741-755, 2008.
[62] C. Katsis, N. Katertsidis, and D. Fotiadis, "An Integrated System Based on Physiological Signals for the Assessment of Affective States in Patients with Anxiety Disorders," Biomedical Signal Processing and Control, vol. 6, pp. 261-268, 2011.
[63] R. Kohavi and F. Provost, "Glossary of Terms," Machine Learning, vol. 30, pp. 271-274, 1998.
[64] F. van der Heiden, R. Duin, D. de Ridder, and D. Tax, Classification, Parameter Estimation, State Estimation: An Engineering Approach Using MatLab. Wiley, 2004.
[65] A. Schlogl, "Time Series Analysis—A Toolbox for the Use with Matlab," 2002.
[66] K. Kroenke, R. Spitzer, and J. Williams, "The phq-9," J. General Internal Medicine, vol. 16, no. 9, pp. 606-613, 2001.
[67] U. Rajendra Acharya, K. Paul Joseph, N. Kannathal, C. Lim, and J. Suri, "Heart Rate Variability: A Review," Medical and Biological Eng. and Computing, vol. 44, no. 12, pp. 1031-1051, 2006.
[68] A. Lanata, E. Scilingo, E. Nardini, G. Loriga, R. Paradiso, and D. De-Rossi, "Comparative Evaluation of Susceptibility to Motion Artifact in Different Wearable Systems for Monitoring Respiratory Rate," IEEE Trans. Information Technology in Biomedicine, vol. 14, no. 2, pp. 378-386, Mar. 2010.
[69] W. Winton, L. Putnam, and R. Krauss, "Facial and Autonomic Manifestations of the Dimensional Structure of Emotion* 1," J. Experimental Social Psychology, vol. 20, no. 3, pp. 195-216, 1984.
[70] P. Lang, "The Emotion Probe," Am. Psychologist, vol. 50, no. 5, pp. 372-385, 1995.
[71] H. McCURDY, "Consciousness and the Galvanometer," Psychological Rev., vol. 57, no. 6, pp. 322-327, 1950.
[72] P. Venables and M. Christie, "Electrodermal Activity," Techniques in Psychophysiology, pp. 3-67, Wiley, 1980.
[73] A. Jain and R. Mao, "Statistical Pattern Recognition: A Review," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[74] B. Kohler, C. Hennig, and R. Orglmeister, "The Principal of Software QRS Detection," IEEE Eng. in Medicine and Biology, vol. 6, no. 1, pp. 42-57, Jan./Feb. 2002.
[75] J. Pan and W. Tompkins, "A Real-Time QRS Detection Algorithm," IEEE Trans. Biomedical Eng., vol. 32, no. 3, pp. 230-236, Mar. 1985.
[76] R. Berger, S. Akselrod, D. Gordon, and R. Cohen, "An Efficient Algorithm for Spectral Analysis of Heart Rate Variability," IEEE Trans. Biomedical Eng., vol. 33, no. 9, pp. 900-904, Sept. 1986.
[77] S. Tiinanen, M. Tulppo, and T. Seppänen, "RSA Component Extraction from Heart Rate Signal by Independent Component Analysis," Proc. Computers in Cardiology, pp. 161-164, 2010.
[78] A. Ishchenko and P. Shev'ev, "Automated Complex for Multiparameter Analysis of the Galvanic Skin Response Signal," Biomedical Eng., vol. 23, no. 3, pp. 113-117, 1989.
[79] H. Akaike, "Fitting Autoregressive Models for Prediction," Annals of the Inst. of Statistical Math., vol. 21, no. 1, pp. 243-247, 1969.
[80] A. Boardman, F. Schlindwein, A. Rocha, and A. Leite, "A Study on the Optimum Order of Autoregressive Models for Heart Rate Variability," Physiological Measurement, vol. 23, pp. 325-336, 2002.
[81] A. Camm et al., "Heart Rate Variability: Standards of Measurement, Physiological Interpretation, and Clinical Use," Circulation, vol. 93, no. 5, pp. 1043-1065, 1996.
[82] S. Koelstra, A. Yazdani, M. Soleymani, C. Mühl, J. Lee, A. Nijholt, T. Pun, T. Ebrahimi, and I. Patras, "Single Trial Classification of EEG and Peripheral Physiological Signals for Recognition of Emotions Induced by Music Videos," Proc. Int'l Conf. Brain Informatics, pp. 89-100, 2010.
[83] J. Mendel, "Tutorial on Higher-Order Statistics (Spectra) in Signal Processing and System Theory: Theoretical Results and Some Applications," Proc. IEEE, vol. 79, no. 3, pp. 278-305, Mar. 1991.
[84] C. Nikias, Higher-Order Spectral Analysis: A Nonlinear Signal Processing Framework. PTR Prentice-Hall, Inc., 1993.
[85] D.R. Brillinger, M. Rosenblatt, and P. Petropulu, "Computation and Interpretation of kth Order Spectra," Spectral Analysis of Time Series, pp. 189-232, Wiley, 1967.
[86] T.N. Chang and S. Sun, "Blind Detection of Photomontage Using Higher Order Statistics," Proc. IEEE Int'l Symp. Circuits and Systems, 2004.
[87] K. Chua, V. Chandran, U. Acharya, and C. Lim, "Cardiac State Diagnosis Using Higher Order Spectra of Heart Rate Variability," J. Medical Eng. & Technology, vol. 32, no. 2, pp. 145-155, 2008.
[88] N. Marwan, M. Carmen Romano, M. Thiel, and J. Kurths, "Recurrence Plots for the Analysis of Complex Systems," Physics Reports, vol. 438, nos. 5/6, pp. 237-329, 2007.
[89] A. Monk and A. Compton, "Review of Modern Physics," The Am. Physical Soc., pp. 173-179, 1939.
[90] J. Eckmann, S. Kamphorst, and D. Ruelle, "Recurrence Plots of Dynamical Systems," Europhysics Letters, vol. 4, p. 973, 1987.
[91] S. Schinkel, O. Dimigen, and N. Marwan, "Selection of Recurrence Threshold for Signal Detection," The European Physical J.-Special Topics, vol. 164, no. 1, pp. 45-53, 2008.
[92] J. Zbilut and C. WebberJr., Recurrence Quantification Analysis. Wiley Online Library, 2006.
[93] C. Peng, S. Havlin, H. Stanley, and A. Goldberger, "Quantification of Scaling Exponents and Crossover Phenomena in Nonstationary Heartbeat Time Series," Chaos: An Interdisciplinary J. Nonlinear Science, vol. 5, no. 1, p. 82, 1995.
[94] I. Jolliffe, Principal Component Analysis. Wiley Online Library, 2002.
[95] R. Duda, P. Hart, and D. Stork, Pattern Classification. Citeseer, 2001.
[96] K. Scherer, "Emotions are Emergent Processes: They Require a Dynamic Computational Architecture," Philosophical Trans. Royal Soc. B: Biological Sciences, vol. 364, no. 1535, pp. 3459-3474, 2009.
[97] V. Marmarelis, Nonlinear Dynamic Modeling of Physiological Systems. Wiley-IEEE Press, 2004.
[98] N. Fragopanagos and J. Taylor, "Emotion Recognition in Human-Computer Interaction," Neural Networks, vol. 18, no. 4, pp. 389-405, 2005.
[99] R. Cowie, E. Douglas-Cowie, K. Karpouzis, G. Caridakis, M. Wallace, and S. Kollias, "Recognition of Emotional States in Natural Human-Computer Interaction," Multimodal User Interfaces, pp. 119-153, Springer, 2008.
[100] D. Grandjean, D. Sander, and K. Scherer, "Conscious Emotional Experience Emerges as a Function of Multilevel, Appraisal-Driven Response Synchronization," Consciousness and Cognition, vol. 17, no. 2, pp. 484-495, 2008.

Index Terms:
skin,electrocardiography,emotion recognition,feature extraction,image classification,medical image processing,physiological models,emotion elicitation,nonlinear dynamics,affective valence,arousal recognition,automatic emotional response assessment,bidimensional spatial localization,experimental protocol,affective states,peripheral physiological signals,electrocardiogram,ECG,electrodermal response,EDR,respiration activity,RSP,International Affective Picture System,IAPS,neutral reference level,standard methods,feature extraction,automatic multiclass arousal classifier,automatic multiclass valence classifier,recognition accuracy,quadratic discriminant classifier,QDC,40-fold cross-validation steps,Electrocardiography,Feature extraction,Electromyography,Support vector machines,Appraisal,Protocols,Emotion recognition,feature extraction.,Emotion recognition,affective computing,nonlinear analysis
A. Lanata, G. Valenza, E. P. Scilingo, "The Role of Nonlinear Dynamics in Affective Valence and Arousal Recognition," IEEE Transactions on Affective Computing, vol. 3, no. 2, pp. 237-249, April-June 2012, doi:10.1109/T-AFFC.2011.30
Usage of this product signifies your acceptance of the Terms of Use.