This Article 
 Bibliographic References 
 Add to: 
An Ensemble Method for Classifying Startle Eyeblink Modulation from High-Speed Video Records
January-March 2011 (vol. 2 no. 1)
pp. 50-63
Reza R. Derakhshani, University of Missouri-Kansas City, Kansas City
Christopher T. Lovelace, University of Missouri-Kansas City, Kansas City
Psychophysiological measurements of startle eyeblink can provide information about the state of an individual regarding sensory, attentional, cognitive, and affective processing, and thus reveal valences of interest for affective computing. However, eyeblink is usually measured using intrusive contact electromyographic (EMG) electrodes, accompanied by a laborious manual process of feature extraction. We introduce a new noninvasive automatic system using high-speed video recording of startle blinks in conjunction with data-driven feature selection and support vector machine (SVM) ensembles to classify startle eyeblinks. Using a prestimulus (prepulse) to produce robust modulation of acoustically elicited startle eyeblinks, we tracked the blinks using 250 frames per second video, and extracted different features from eyelid displacement and velocity signals. The SVMs were able to determine whether a trial had contained startle or prepulse+startle stimuli with an accuracy of up to 73 percent (five-fold cross validation). By fusing the decisions made on different feature sets, an ensemble of seven SVMs increased this rate to almost 79 percent. Since startle eyeblinks are robustly modulated by not only sensory events (such as the prepulse used in this study) but also affective and cognitive states, eyelid tracking using high-speed video, in conjunction with the introduced classification method, is an effective and user-friendly alternative to EMG for classification of startle blinks to infer users' affective-cognitive states.

[1] R.W. Picard , Affective Computing. MIT Press, 1997.
[2] R.W. Picard et al., “Toward Machine Emotional Intelligence: Analysis of Affective Physiological State,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 10, pp. 1175-1191, Oct. 2001.
[3] A. Jaimes and N. Sebe , “Multimodal Human-Computer Interaction: A Survey,” Computer Vision and Image Understanding, vol. 108, pp. 116-134, 2007.
[4] Z. Zhihong , “A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39-58, Jan. 2009.
[5] C.H. Chen and P.S.P. Wang , Handbook of Pattern Recognition and Computer Vision, third ed. World Scientific, 2005.
[6] D.L. Filion et al., “The Psychological Significance of Human Startle Eyeblink Modification: A Review,” Biological Psychology, vol. 47, pp. 1-43, 1998.
[7] P.J. Lang et al., “Emotion, Attention, and the Startle Reflex,” Psychological Rev., vol. 97, pp. 377-95, July 1990.
[8] S.R. Vrana et al., “The Startle Probe Response: A New Measure of Emotion?” J. Abnormal Psychology, vol. 97, pp. 487-91, Nov. 1988.
[9] T.D. Blumenthal et al., “Committee Report: Guidelines for Human Startle Eyeblink Electromyographic Studies,” Psychophysiology, vol. 42, pp. 1-15, 2005.
[10] C.T. Lovelace et al., “Classification of Startle Eyeblink Metrics Using Neural Networks,” Proc. Int'l Joint Conf. Neural Networks, pp. 1908-1914, 2009.
[11] C.T. Lovelace et al., “Whence the Eye Blinketh? Correspondence between Multiple Measures of Startle Eyeblink,” Proc. 48th Ann. Meeting of the Soc. for Psychophysiological Research, pp. S1-S132, 2008.
[12] F.K. Graham , “Control of Reflex Blink Excitability,” Neural Mechanisms of Goal-Directed Behavior and Learning, R.F. Thompson, et al., eds., pp. 511-519, Academic Press, 1980.
[13] J.A. Stern et al., “The Endogenous Eyeblink,” Psychophysiology, vol. 21, pp. 22-33, Jan. 1984.
[14] M.M. Bradley et al., “Startle Reflex Modification: Emotion or Attention?” Psychophysiology, vol. 27, pp. 513-522, Sept. 1990.
[15] C. Grillon and J. Baas , “A Review of the Modulation of the Startle Reflex by Affective States and Its Application in Psychiatry,” Clinical Neurophysiology, vol. 114, pp. 1557-1579, Sept. 2003.
[16] B. Verschuere et al., “Startling Secrets: Startle Eye Blink Modulation by Concealed Crime Information,” Biological Psychology, vol. 76, pp. 52-60, 2007.
[17] R. Derakhshani and A. Ross , “A Texture-Based Neural Network Classifier for Biometric Identification Using Ocular Surface Vasculature,” Proc. Int'l Joint Conf. Neural Networks, pp. 2982-2987, 2007.
[18] B. Faisal , “Video Surveillance for Biometrics: Long-Range Multi-Biometric System,” Proc. IEEE Fifth Int'l Conf. Advanced Video and Signal Based Surveillance, pp. 175-182, 2008.
[19] P. Cobos et al., “Revisiting the James versus Cannon Debate on Emotion: Startle and Autonomic Modulation in Patients with Spinal Cord Injuries,” Biological Psychology, vol. 61, pp. 251-269, 2002.
[20] M.A. Oskoei and H. Hu , “Support Vector Machine-Based Classification Scheme for Myoelectric Control Applied to Upper Limb,” IEEE Trans. Biomedical Eng., vol. 55, no. 8, pp. 1956-1965, Aug. 2008.
[21] R.L. Mandryk and M.S. Atkins , “A Fuzzy Physiological Approach for Continuously Modeling Emotion during Interaction with Play Technologies,” Int'l J. Human-Computer Studies, vol. 65, pp. 329-347, 2007.
[22] R. Albert et al., “Development of a VR Therapy Application for Iraq War Military Personnel with PTSD,” Medicine Meets Virtual Reality 13: The Magical Next Becomes the Medical Now, pp. 407-413, IOS Press, 2005.
[23] M.D. Marsico and S. Levialdi , “Evaluating Web Sites: Exploiting User's Expectations,” Int'l J. Human-Computer Studies, vol. 60, pp. 381-416, 2004.
[24] R.L. Hazlett and J. Benedek , “Measuring Emotional Valence to Understand the User's Experience of Software,” Int'l J. Human-Computer Studies, vol. 65, pp. 306-314, 2007.
[25] K. Dautenhahn and I. Werry , “Towards Interactive Robots in Autism Therapy: Background, Motivation and Challenges,” Pragmatics & Cognition, vol. 12, pp. 1-35, 2004.
[26] R.E. Kaliouby et al., “Affective Computing and Autism,” Annals of the New York Academy of Sciences, vol. 1093, pp. 228-248, 2006.
[27] P. Rani et al., “Anxiety Detecting Robotic System; Towards Implicit Human-Robot Collaboration,” Robotica, vol. 22, pp. 85-95, 2004.
[28] C. Karatekin et al., “Oculomotor and Pupillometric Indices of Pro- and Antisaccade Performance in Youth-Onset Psychosis and Attention Deficit/Hyperactivity Disorder,” Schizophrenia Bull., vol. 36, pp. 1167-1186, May 2009.
[29] Y. Nestoriuc et al., “Meta-Analysis of Biofeedback for Tension-Type Headache: Efficacy, Specificity, and Treatment Moderators,” J. Consulting and Clinical Psychology, vol. 76, pp. 379-396, June 2008.
[30] A. Kapoor and R.W. Picard , “Multimodal Affect Recognition in Learning Environments,” Proc. 13th Ann. ACM Int'l Conf. Multimedia, 2005.
[31] J. Rasmussen , Information Processing and Human-Machine Interaction: An Approach to Cognitive Engineering. Elsevier Science, 1986.
[32] A.K. Jain et al., Biometrics: Personal Identification in Networked Society. Kluwer, 1999.
[33] A.K. Jain , Handbook of Biometrics. Springer, 2007.
[34] A. Jain and D. Zongker , “Feature Selection: Evaluation, Application, and Small Sample Performance,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 153-158, Feb. 1997.
[35] I. Guyon and A. Elisseeff , “An Introduction to Variable and Feature Selection,” J. Machine Learning Research, vol. 3, pp. 1157-1182, 2003.
[36] A.K. Jain et al., “Statistical Pattern Recognition: A Review,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 1, pp. 4-37, Jan. 2000.
[37] L. Li et al., “Application of the GA/KNN Method to SELDI Proteomics Data,” Bioinformatics, vol. 20, pp. 1638-1640, July 2004.
[38] R. Ruiz et al., “Incremental Wrapper-Based Gene Selection from Microarray Data for Cancer Classification,” Pattern Recognition, vol. 39, pp. 2383-2392, 2006.
[39] P. Pudil et al., “Floating Search Methods in Feature Selection,” Pattern Recognition Letters, vol. 15, pp. 1119-1125, 1994.
[40] S. Theodoridis and K. Koutroumbas , Pattern Recognition, third ed., Academic Press, 2006.
[41] J. Príncipe et al., Neural and Adaptive Systems: Fundamentals through Simulations. Wiley, 1999.
[42] T. Fawcett , “An Introduction to ROC Analysis,” Pattern Recognition Letters, vol. 27, pp. 861-874, 2006.
[43] E. Alpaydin , Introduction to Machine Learning. MIT Press, 2004.
[44] P. Baldi et al., “Assessing the Accuracy of Prediction Algorithms for Classification: An Overview,” Bioinformatics, vol. 16, pp. 412-424, 2000.
[45] R. Duda et al., Pattern Classification, second ed., Wiley, 2001.
[46] C.M. Bishop , Pattern Recognition and Machine Learning. Springer, 2006.
[47] V.N. Vapnik , The Nature of Statistical Learning Theory, second ed., Springer, 2000.
[48] P. Rani et al., “An Empirical Study of Machine Learning Techniques for Affect Recognition in Human-Robot Interaction,” Pattern Analysis & Applications, vol. 9, pp. 58-69, 2006.
[49] K. Kim et al., “Emotion Recognition System Using Short-Term Monitoring of Physiological Signals,” Medical and Biological Eng. and Computing, vol. 42, pp. 419-427, 2004.
[50] C.J. Burges , “A Tutorial on Support Vector Machines for Pattern Recognition,” Data Mining and Knowledge Discovery, vol. 2, pp. 121-167, 1998.
[51] S. Haykin , Neural Networks and Learning Machines, third ed., Prentice Hall, 2009.
[52] T. Dietterich , “Machine-Learning Research: Four Current Directions,” AI Magazine, vol. 18, pp. 97-136, 1997.
[53] R. Ranawana and V. Palade , “Multi-Classifier Systems: Review and a Roadmap for Developers,” Int'l J. Hybrid Intelligent Systems, vol. 3, pp. 35-61, 2006.
[54] L.I. Kuncheva , Combining Pattern Classifiers: Methods and Algorithms. John Wiley and Sons, 2004.
[55] Y. Lu , “Knowledge Integration in a Multiple Classifier System,” Applied Intelligence, vol. 6, pp. 75-86, 1996.
[56] R. Polikar , “Ensemble Based Systems in Decision Making,” IEEE Circuits and Systems Magazine, vol. 6, no. 3, pp. 21-45, 2006.
[57] J. Daugman , “How Iris Recognition Works,” IEEE Trans. Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 21-30, Jan. 2004.
[58] G. Hung et al., “Dynamics of the Human Eyeblink,” Am. J. Optometry and Physiological Optics, vol. 54, pp. 678-90, Oct. 1977.
[59] R.W. Picard , “Affective Computing: Challenges,” Int'l J. Human-Computer Studies, vol. 59, pp. 55-64, 2003.
[60] M. Kass et al., “Snakes: Active Contour Models,” Int'l J. Computer Vision, vol. 1, pp. 321-331, 1988.
[61] T. Cootes et al., “Active Appearance Models,” Proc. Fifth European Conf. Computer Vision, pp. 484-498, 1998.
[62] A. Tsymbal et al., “Diversity in Search Strategies for Ensemble Feature Selection,” Information Fusion, vol. 6, pp. 83-98, 2005.
[63] J. Kittler , “Combining Classifiers: A Theoretical Framework,” Pattern Analysis & Applications, vol. 1, pp. 18-27, 1998.
[64] M.I. Jordan and R.A. Jacobs , “Hierarchical Mixtures of Experts and the EM Algorithm,” Neural Computation, vol. 6, pp. 181-214, 2008.
[65] Y. Freund and R.E. Schapire , “A Short Introduction to Boosting,” J. Japanese Soc. for Artificial Intelligence, vol. 14, pp. 771-780, 1999.
[66] T. Denoeux , “A Neural Network Classifier Based on Dempster-Shafer Theory,” IEEE Trans. Systems, Man and Cybernetics, Part A: Systems and Humans, vol. 30, no. 2, pp. 131-150, Mar. 2000.
[67] J.A. Benediktsson et al., “Hybrid Consensus Theoretic Classification,” IEEE Trans. Geoscience and Remote Sensing, vol. 35, no. 4, pp. 833-843, July 1997.
[68] H. Drucker et al., “Boosting and Other Ensemble Methods,” Neural Computation, vol. 6, pp. 1289-1301, 1994.
[69] K. Woods et al., “Combination of Multiple Classifiers Using Local Accuracy Estimates,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 391-396, 1996.
[70] R. Hu and R.I. Damper , “A ‘No Panacea Theorem’ for Classifier Combination,” Pattern Recognition, vol. 41, pp. 2665-2673, 2008.
[71] D.H. Wolpert and W.G. Macready , “No Free Lunch Theorems for Optimization,” IEEE Trans. Evolutionary Computation, vol. 1, no. 1, pp. 67-82, Apr. 1997.
[72] G.J. Siegle et al., “Blink Before and After You Think: Blinks Occur Prior to and Following Cognitive Load Indexed by Pupillary Responses,” Psychophysiology, vol. 45, pp. 679-687, 2008.
[73] B.C. Wangelin et al., “Aversive Picture Processing: Effects of a Concurrent Task on Sustained Defensive System Engagement,” Psychophysiology, 2010.
[74] D. Braff et al., “Human Studies of Prepulse Inhibition of Startle: Normal Subjects, Patient Groups, and Pharmacological Studies,” Psychopharmacology, vol. 156, pp. 234-258, 2001.
[75] P.J. Lang and L.M. McTeague , “The Anxiety Disorder Spectrum: Fear Imagery, Physiological Reactivity, and Differential Diagnosis,” Anxiety Stress Coping, vol. 22, pp. 5-25, Jan. 2009.
[76] M.B. Bradley et al., “Fear of Pain and Defensive Activation,” Pain, vol. 137, pp. 156-163, 2008.
[77] A.O. Hamm et al., “Emotional Learning, Hedonic Change, and the Startle Probe,” J. Abnormal Psychology, vol. 102, pp. 453-465, 1993.
[78] R. Johnson Jr. et al., “The Self in Conflict: The Role of Executive Processes during Truthful and Deceptive Responses about Attitudes,” NeuroImage, vol. 39, pp. 469-482, 2008.
[79] S. Leal et al., “The Time of the Crime: Cognitively Induced Tonic Arousal Suppression When Lying in a Free Recall Context,” Acta Psychologica, vol. 129, pp. 1-7, 2008.

Index Terms:
Affective computing, image processing, pattern recognition, user interfaces.
Reza R. Derakhshani, Christopher T. Lovelace, "An Ensemble Method for Classifying Startle Eyeblink Modulation from High-Speed Video Records," IEEE Transactions on Affective Computing, vol. 2, no. 1, pp. 50-63, Jan.-March 2011, doi:10.1109/T-AFFC.2010.15
Usage of this product signifies your acceptance of the Terms of Use.