The Community for Technology Leaders
RSS Icon
Issue No.02 - July-December (2010 vol.1)
pp: 109-118
Christopher G. Courtney , University of Southern California, Los Angeles
Brent J. Lance , US Army Research Laboratory, Aberdeen Proving Ground
Shrikanth S. Narayanan , University of Southern California, Los Angeles
Dongrui Wu , GE Global Research, Niskayuna
Kelvin S. Oie , US Army RDECOM, Aberdeen Proving Ground
Thomas D. Parsons , University of Southern California, Playa Vista
A closed-loop system that offers real-time assessment and manipulation of a user's affective and cognitive states is very useful in developing adaptive environments which respond in a rational and strategic fashion to real-time changes in user affect, cognition, and motivation. The goal is to progress the user from suboptimal cognitive and affective states toward an optimal state that enhances user performance. In order to achieve this, there is need for assessment of both 1) the optimal affective/cognitive state and 2) the observed user state. This paper presents approaches for assessing these two states. Arousal, an important dimension of affect, is focused upon because of its close relation to a user's cognitive performance, as indicated by the Yerkes-Dodson Law. Herein, we make use of a Virtual Reality Stroop Task (VRST) from the Virtual Reality Cognitive Performance Assessment Test (VRCPAT) to identify the optimal arousal level that can serve as the affective/cognitive state goal. Three stimuli presentations (with distinct arousal levels) in the VRST are selected. We demonstrate that when reaction time is used as the performance measure, one of the three stimuli presentations can elicit the optimal level of arousal for most subjects. Further, results suggest that high classification rates can be achieved when a support vector machine is used to classify the psychophysiological responses (skin conductance level, respiration, ECG, and EEG) in these three stimuli presentations into three arousal levels. This research reflects progress toward the implementation of a closed-loop affective computing system.
Affective computing, arousal classification, affect recognition, virtual reality, Stroop task, Yerkes-Dodson Law.
Christopher G. Courtney, Brent J. Lance, Shrikanth S. Narayanan, Dongrui Wu, Kelvin S. Oie, Thomas D. Parsons, "Optimal Arousal Identification and Classification for Affective Computing Using Physiological Signals: Virtual Reality Stroop Task", IEEE Transactions on Affective Computing, vol.1, no. 2, pp. 109-118, July-December 2010, doi:10.1109/T-AFFC.2010.12
[1] K. Astrom and T. Hagglund , Advanced PID Control. ISA Press, 2006.
[2] C. Breazeal , “Emotion and Sociable Humanoid Robots,” Int'l J. Human-Computer Studies, vol. 59, nos. 1/2, pp. 119-155, 2003.
[3] G. Bush , P. Luu , and M.I. Posner , “Cognitive and Emotional Influences in Anterior Cingulate Cortex,” Trends in Cognitive Sciences, vol. 4, no. 6, pp. 215-222, 2000.
[4] C. Busso , Z. Deng , S. Yildirim , M. Bulut , C.M. Lee , A. Kazemzadeh , S. Lee , U. Neumann , and S.S. Narayanan , “Analysis of Emotion Recognition Using Facial Expressions, Speech and Multimodal Information,” Proc. Int'l Conf. Multimodal Interfaces, pp. 205-211, Oct. 2004.
[5] C.-C. Chang and C.-J. Lin , “LIBSVM: A Library for Support Vector Machines,”, 2009.
[6] C. Conati , “Probabilistic Assessment of Users Emotions in Educational Games,” Applied Artificial Intelligence, vol. 16, nos.7/8, pp. 555-575, 2002.
[7] C. Cortes and V. Vapnik , “Support-Vector Network,” Machine Learning, vol. 20, pp. 273-297, 1995.
[8] R. Cowie and R. Cornelius , “Describing the Emotional States That Are Expressed in Speech,” Speech Comm., vol. 40, pp. 5-32, 2003.
[9] S.D. Craig , A.C. Graesser , J. Sullins , and B. Gholson , “Affect and Learning: An Exploratory Look into the Role of Affect in Learning with Autotutor,” J. Educational Media, vol. 29, no. 3, pp. 241-250, 2004.
[10] R.O. Duda , P.E. Hart , and D.G. Stork , Pattern Classification. Wiley-Interscience, 2000.
[11] S.H. Fairclough , “Fundamentals of Physiological Computing,” Interacting with Computers, vol. 21, pp. 133-145, 2009.
[12] B. Fasel and J. Luettin , “Automatic Facial Expression Analysis: A Survey,” Pattern Recognition, vol. 36, no. 1, pp. 259-275, 2003.
[13] T. Fong , I. Nourbakhsh , and K. Dautenhahn , “A Survey of Socially Interactive Robots,” Robotics and Autonomous Systems, vol. 42, nos. 3/4, pp. 143-166, 2003.
[14] K. Gilleade , A. Dix , and J. Allanson , “Affective Videogames and Modes of Affective Gaming: Assist Me, Challenge Me, Emote Me,” Proc. Digital Games Research Assoc. Conf., pp. 16-20, June 2005.
[15] J. Green and A. Arduini , “Hippocampal Activity in Arousal,” J.Neurophysiology, vol. 17, no. 6, pp. 533-557, 1954.
[16] M. Grimm , K. Kroschel , E. Mower , and S.S. Narayanan , “Primitives-Based Evaluation and Estimation of Emotions in Speech,” Speech Comm., vol. 49, pp. 787-800, 2007.
[17] I. Guyon and A. Ellisseeff , “An Introduction to Variable and Feature Selection,” J. Machine Learning Research, vol. 3, pp. 1157-1182, 2003.
[18] J.L. Herman , “Complex PTSD: A Syndrome in Survivors of Prolonged and Repeated Trauma,” J. Traumatic Stress, vol. 5, no. 3, pp. 377-391, 1992.
[19] Y. Hoshikawa and Y. Yamamoto , “Effects of Stroop Color-Word Conflict Test on the Autonomic Nervous System Responses,” AJP—Heart and Circulatory Physiology, vol. 272, no. 3, pp. 1113-1121, 1997.
[20] H. Jasper , “The Ten-Twenty Electrode System of the International Federation,” Recommendations for the Practice of Clinical Neurophysiology, Elsevier Publishing Company, 1983.
[21] R. Kehrein , “The Prosody of Authentic Emotions,” Proc. Speech Prosody Conf., pp. 423-426, Apr. 2002.
[22] J. Kim and E. Andre , “Fusion of Multichannel Biosignals Towards Automatic Emotion Recognition,” Multisensor Fusion and Integration for Intelligent Systems, S. Lee, H. Ko, and H. Hahn, eds., pp. 55-68, Springer-Verlag, 2009.
[23] J. Kittler , “Feature Set Search Algorithms,” Pattern Recognition and Signal Processing, C.H. Chen, ed., pp. 41-60, Springer, 1978.
[24] C.M. Lee and S.S. Narayanan , “Toward Detecting Emotions in Spoken Dialogs,” IEEE Trans. Speech and Audio Processing, vol. 13, no. 2, pp. 293-303, Mar. 2005.
[25] M. Liotti , M.G. Woldorff , R. Perez , and H.S. Mayberg , “An ERP Study of the Temporal Course of the Stroop Color-Word Interference Effect,” Neuropsychologia, vol. 38, no. 5, pp. 701-711, 2000.
[26] C. Lisetti and C. LeRouge , “Affective Computing in Tele-Home Health,” Proc. 37th Hawaii Int'l Conf. System Sciences, p. 8, Jan. 2004.
[27] C. Liu , P. Agrawal , N. Sarkar , and S. Chen , “Dynamic Difficulty Adjustment in Computer Games through Real-Time Anxiety-Based Affective Feedback,” Int'l J. Human-Computer Interaction, vol. 25, no. 6, pp. 506-529, 2009.
[28] C.M. MacLeod , “Half a Century of Research on the Stroop Effect: An Integrative Review,” Psychological Bull., vol. 109, no. 2, pp. 163-203, 1991.
[29] C. Magerkurth , A. Cheok , R. Mandryk , and T. Nilsen , “Pervasive Games: Bringing Computer Entertainment Back to the Real World,” ACM Computers in Entertainment, vol. 3, no. 3, pp. 11-29, 2005.
[30] J.M. Mendel , Uncertain Rule-Based Fuzzy Logic Systems: Introduction and New Directions. Prentice-Hall, 2001.
[31] J.M. Mendel and D. Wu , Perceptual Computing: Aiding People in Making Subjective Judgments. Wiley-IEEE Press, 2010.
[32] A. Metallinou , S. Lee , and S.S. Narayanan , “Decision Level Combination of Multiple Modalities for Recognition and Analysis of Emotional Expression,” Proc. Int'l Conf. Acoustics, Speech, and Signal Processing, pp. 2462-2465, Mar. 2010.
[33] J. Morton and S.M. Chambers , “Selective Attention to Words and Colors,” The Quarterly J. Experimental Psychology, vol. 25, no. 3, pp.387-397, 1973.
[34] M. Nuwer , “Quantitative EEG: I. Techniques and Problems of Frequency Analysis and Topographic Mapping,” J. Clinical Neurophysiology, vol. 5, no. 1, pp. 1-43, 1988.
[35] M. Pantic and L.J. Rothkrantz , “Automatic Analysis of Facial Expressions: The State of the Art,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 12, pp. 1424-1445, Dec. 2000.
[36] J.V. Pardo , P.J. Pardo , K.W. Janer , and M.E. Raichle , “The Anterior Cingulate Cortex Mediates Processing Selection in the Stroop Attentional Conflict Paradigm,” Proc. Nat'l Academy of Sciences USA, vol. 87, no. 1, pp. 256-259, 1990.
[37] T. Parsons , L. Cosand , C. Courtney , A. Iyer , and A. Rizzo , “Neurocognitive Workload Assessment Using the Virtual Reality Cognitive Performance Assessment Test,” Proc. Eighth Int'l Conf. Eng. Psychology and Cognitive Ergonomics, pp. 243-252, 2009.
[38] T. Parsons , C. Courtney , L. Cosand , A. Iyer , A. Rizzo , and K. Oie , “Assessment of Psychophysiological Differences of West Point Cadets and Civilian Controls Immersed within a Virtual Environment,” Proc. Fifth Int'l Conf. Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience, pp. 514-523, 2009.
[39] T. Parsons and A. Rizzo , “Initial Validation of a Virtual Environment for Assessment of Memory Functioning: Virtual Reality Cognitive Performance Assessment Test,” Cyberpsychology and Behavior, vol. 11, pp. 17-25, 2008.
[40] T.D. Parsons , A. Iyer , L. Cosand , C. Courtney , and A.A. Rizzo , “Neurocognitive and Psychophysiological Analysis of Human Performance within Virtual Reality Environments,” Studies Health Technology and Informatics, vol. 142, pp. 247-252, 2009.
[41] T.D. Parsons and A.A. Rizzo , “Affective Outcomes of Virtual Reality Exposure Therapy for Anxiety and Specific Phobias: A Meta-Analysis,” J. Behavior Therapy and Experimental Psychiatry, vol. 39, pp. 250-261, 2008.
[42] R. Picard , Affective Computing. The MIT Press, 1997.
[43] R. Picard , “Affective Medicine: Technology with Emotional Intelligence,” Future of Health Technology, R. Bushko, ed., pp. 69-83, IOS Press, 2002.
[44] R. Picard , S. Papert , W. Bender , B. Blumberg , C. Breazeal , D. Cavallo , T. Machover , M. Resnick , D. Roy , and C. Strohecker , “Affective Learning a Manifesto,” BT Technology J., vol. 22, no. 4, pp. 253-269, 2004.
[45] H. Prendinger and M. Ishizuka , “What Affective Computing and Life-Like Character Technology Can Do for Tele-Home Health Care,” Proc. Workshop HCI and Homecare, Apr. 2004.
[46] P. Rani , C. Liu , N. Sarkar , and E. Vanman , “An Empirical Study of Machine Learning Techniques for Affect Recognition in Human-Robot Interaction,” Pattern Analysis and Application, vol. 9, no. 1, pp. 58-69, 2006.
[47] J.A. Russell , “Core Affect and the Psychological Construction of Emotion,” Psychological Rev., vol. 110, pp. 145-172, 2003.
[48] S. Sanei and J.A. Chambers , EEG Signal Processing. John Wiley & Sons, 2007.
[49] K. Scherer , “What Are Emotions? And How Can They Be Measured?” Social Science Information, vol. 44, no. 4, pp. 695-729, 2005.
[50] B. Scholkopf , A. Smola , R.C. Williamson , and P.L. Bartlett , “New Support Vector Algorithms,” Neural Computation, vol. 12, pp. 1207-1245, 2000.
[51] B. Schuller , M. Lang , and G. Rigoll , “Recognition of Spontaneous Emotions by Speech within Automotive Environment,” Proc. German Ann. Conf. Acoustics, pp. 57-58, Mar. 2006.
[52] H.V. Semltsch , P. Anderer , P. Schuster , and O. Presslich , “A Solution for Reliable and Valid Reduction of Ocular Artifacts, Applied to the P300 ERP,” Psychophysiology, vol. 23, no. 6, pp. 695-703, 1986.
[53] N. Serbedzija and S. Fairclough , “Biocybernetic Loop: From Awareness to Evolution,” Proc. IEEE Congress Evolutionary Computation, pp. 2063-2069, May 2009.
[54] J. Stroop , “Studies of Interference in Serial Verbal Reactions,” J.Experimental Psychology, vol. 18, pp. 643-661, 1935.
[55] J. Sykes and S. Brown , “Affective Gaming: Measuring Emotion through the Gamepad,” Proc. Conf. Human Factors, pp. 732-733, Apr. 2003.
[56] A. Tan and R. Muhlberger , “From Operator to Interaction: Designing an Affective Computing System for Interactive Lighting,” Proc. Computer/Human Interaction Conf., Apr. 2009.
[57] J. Tao and T. Tan , “Affective Computing: A Review,” Affective Computing and Intelligent Interaction, pp. 981-995, Springer, 2005.
[58] S. Taylor , S. Kornblum , E.J. Lauber , S. Minoshima , and R.A. Koeppe , “Isolation of Specific Interference Processing in the Stroop Task: PET Activation Studies,” Neuroimage, vol. 6, no. 2, pp. 81-92, 1997.
[59] E. Vesterinen , “Affective Computing,” Proc. Digital Media Research Seminar, 2001.
[60] L.-X. Wang , Adaptive Fuzzy Systems and Control: Design and Stability Analysis. Prentice Hall, 1994.
[61] P.J. Whalen , G. Bush , R.J. McNally , S. Wilhelm , S.C. McInerney , M.A. Jenike , and S.L. Rauch , “The Emotional Counting Stroop Paradigm: A Functional Magnetic Resonance Imaging Probe of the Anterior Cingulate Affective Division,” Biological Psychiatry, vol. 44, no. 12, pp. 1219-1228, 1998.
[62] D. Wu , “Intelligent Systems for Decision Support,” PhD dissertation, Univ. of Southern California, May 2009.
[63] D. Wu and J.M. Mendel , “Linguistic Summarization Using IF-THEN Rules and Interval Type-2 Fuzzy Sets,” IEEE Trans. Fuzzy Systems, to appear, 2010.
[64] D. Wu and J.M. Mendel , “Social Judgment Advisor: An Application of the Perceptual Computer,” Proc. IEEE World Congress on Computational Intelligence, July 2010.
[65] D. Wu , T.D. Parsons , E. Mower , and S.S. Narayanan , “Speech Emotion Estimation in 3D Space,” Proc. IEEE Int'l Conf. Multimedia and Expo, pp. 737-742, July 2010.
[66] D. Wu and W.W. Tan , “Type-2 FLS Modeling Capability Analysis,” Proc. IEEE Int'l Conf. Fuzzy Systems, pp. 242-247, May 2005.
[67] D. Wu and W.W. Tan , “Genetic Learning and Performance Evaluation of Type-2 Fuzzy Logic Controllers,” Eng. Applications of Artificial Intelligence, vol. 19, no. 8, pp. 829-841, 2006.
[68] D. Wu and W.W. Tan , “Interval Type-2 Fuzzy PI Controllers: Why They Are More Robust,” Proc. IEEE Int'l. Conf. Granular Computing, pp. 802-807, Aug. 2010.
[69] R. Yerkes and J. Dodson , “The Relation of Strength of Stimulus to Rapidity of Habit-Formation,” J. Comparative Neurology and Psychology, vol. 18, pp. 459-482, 1908.
[70] L.A. Zadeh , “Fuzzy Sets,” Information and Control, vol. 8, no. 3, pp.338-353, 1965.
[71] L.A. Zadeh , “The Concept of a Linguistic Variable and Its Application to Approximate Reasoning-1,” Information Sciences, vol. 8, no. 3, pp. 199-249, 1975.
[72] Z. Zeng , M. Pantic , G.I. Roisman , and T.S. Huang , “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
19 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool