The Community for Technology Leaders
RSS Icon
Issue No.01 - January-June (2010 vol.1)
pp: 60-71
Timothy W. Bickmore , Northeastern University, Boston
Rukmal Fernando , Northeastern University, Boston
Lazlo Ring , Northeastern University, Boston
Daniel Schulman , Northeastern University, Boston
We describe a series of experiments with an agent designed to model human conversational touch—capable of physically touching users in synchrony with speech and other nonverbal communicative behavior—and its use in expressing empathy to users in distress. The agent is composed of an animated human face that is displayed on a monitor affixed to the top of a human mannequin, with touch conveyed by an air bladder that squeezes a user's hand. We demonstrate that when touch is used alone, hand squeeze pressure and number of squeezes are associated with user perceptions of affect arousal conveyed by an agent, while number of squeezes and squeeze duration are associated with affect valence. We also show that, when affect-relevant cues are present in facial display, speech prosody, and touch used simultaneously by the agent, facial display dominates user perceptions of affect valence, and facial display and prosody are associated with affect arousal, while touch had little effect. Finally, we show that when touch is used in the context of an empathic, comforting interaction (but without the manipulation of affect cues in other modalities), it can lead to better perceptions of relationship with the agent, but only for users who are comfortable being touched by other people.
Animations, evaluation, haptic I/O, natural language, user interfaces.
Timothy W. Bickmore, Rukmal Fernando, Lazlo Ring, Daniel Schulman, "Empathic Touch by Relational Agents", IEEE Transactions on Affective Computing, vol.1, no. 1, pp. 60-71, January-June 2010, doi:10.1109/T-AFFC.2010.4
[1] A. Horvath and D. Symonds, "Relation between Working Alliance and Outcome in Psychotherapy a Meta-Analysis," J. Counseling Psychology, vol. 38, pp. 139-149, 1991.
[2] R. Frankel, "Emotion and the Physician-Patient Relationship," Motivation and Emotion, vol. 19, pp. 163-173, 1995.
[3] W. Reynolds, The Measurement and Development of Empathy in Nursing. Ashgate Publishing, 2000.
[4] T. Bickmore and R. Picard, "Establishing and Maintaining Long-Term Human-Computer Relationships," ACM Trans. Computer Human Interaction, vol. 12, pp. 293-327, 2005.
[5] S. Brave, C. Nass, and K. Hutchinson, "Computers That Care: Investigating the Effects of Orientation of Emotion Exhibited by an Embodied Computer Agent," Int'l J. Human-Computer Studies, vol. 62, pp. 161-178, 2005.
[6] J. Klein, Y. Moon, and R. Picard, "This Computer Responds to User Frustration: Theory, Design, Results, and Implications," Interacting with Computers, vol. 14, pp. 119-140, 2002.
[7] T. Bickmore and R. Picard, "Towards Caring Machines," Proc. SIGCHI '04, 2004.
[8] W. Weiss, "Measurement of the Sensory Qualities in Tactile Interaction," Nursing Research, vol. 41, pp. 82-85, 1992.
[9] S. Jones and A. Yarbrough, "A Naturalistic Study of the Meanings of Touch," Comm. Monographs, vol. 52, pp. 19-58, 1985.
[10] T. Nguyen, R. Heslin, and M. Nguyen, "The Meanings of Touch: Sex Differences," J. Comm., vol. 25, pp. 92-103, 1975.
[11] M. Herstenstein, D. Keltner, B. App, B. Bulleit, and A. Jaskolka, "Touch Communicates Distinct Emotions," Emotion, vol. 6, pp. 528-533, 2006.
[12] D. Vaidis and S.G. Halimi-Falkowicz, "Increasing Compliance with a Request: Two Touches Are More Effective," Psychological Reports, vol. 103, pp. 88-92, 2008.
[13] L.K. Guerrero and P.A. Andersen, "The Waxing and Waning of Relational Intimacy: Touch as a Function of Relational Stage, Gender and Touch Avoidance," J. Social and Personal Relationships, vol. 8, pp. 147-166, 1991.
[14] F. Willis and L. Briggs, "Relationship and Touch in Public Settings," J. Nonverbal Behavior, vol. 16, pp. 55-63, 1992.
[15] N. Henley, "Status and Sex: Some Touching Observations," Bull. Psychonomic Soc., vol. 2, pp. 91-93, 1973.
[16] J. Bottorff, "The Use and Meaning of Touch in Caring for Patients with Cancer," Oncology Nursing Forum, vol. 20, pp. 1531-1538, 1993.
[17] S. Chang, "The Conceptual Structure of Physical Touch in Caring," J. Advanced Nursing, vol. 33, pp. 820-287, 2008.
[18] J. Fisher and S. Gallant, "Effect of Touch on Hospitalized Patients," Advances in Touch, N. Gunzenhauser, T. Brazelton, and T. Field, eds., pp. 141-147, Johnson & Johnson, 1990.
[19] E. Bush, "The Use of Human Touch to Improve the Well-Being of Older Adults," J. Holistic Nursing, vol. 19, pp. 256-270, 2001.
[20] J. Kutner, M. Smith, L. Corbin, L. Hemphill, K. Benton, K. Mellis, B. Beaty, S. Felton, T. Yamashita, L. Bryant, and D. Fairclough, "Massage Therapy versus Simple Touch to Improve Pain and Mood in Patients with Advanced Cancer," Annals of Internal Medicine, vol. 149, pp. 369-379, 2008.
[21] S. Schoenhofer, "Affectional Touch in Critical Care Nursing: A Descriptive Study," Heart Lung, vol. 18, pp. 146-154, 1989.
[22] R. Lewis, V. Derlega, B. Nichols, A. Shankar, K. Drury, and L. Hawkins, "Sex Differences in Observers' Reactions to a Nurse's Use of Touch," J. Nonverbal Behavior, vol. 19, pp. 101-113, 1995.
[23] T. Field, Touch Therapy. Churchill Livingstone, 2000.
[24] F. Meuller, F. Vetere, M. Gibbs, J. Kjeldskov, S. Pedell, and S. Howard, "Hug over a Distance," Proc. SIGCHI '05, 2005.
[25] E. Eichhorn, R. Wettach, and E. Hornecker, "A Stroking Device for Spatially Separated Couples," Proc. Int'l Conf. Human-Computer Interaction with Mobile Devices and Services, 2008.
[26] K. Chung, C. Chiu, X. Xiao, and P. Chi, "Stress OutSourced: A Haptic Social Network via Crowdsourcing," Proc. SIGCHI '09, pp. 2439-2448, 2009.
[27] S. Brave, A. Dahley, P. Frei, V. Su, and H. Ishii, "inTouch," Proc. SIGGRAPH '98, 1998.
[28] B. Fogg, L. Cutler, P. Arnold, and C. Eisbach, "HandJive: A Device for Interpersonal Haptic Entertainment," Proc. Conf. Human Factors in Computing Systems, pp. 57-64, 1998.
[29] J. Smith and K. MacLean, "Communicating Emotion through a Haptic Link: Design Space and Methodology," Int'l J. Human Computer Studies, special issue on affective evaluation-innovative approaches, vol. 65, pp. 376-387, 2007.
[30] A. Haans and W.A. IJsselsteijn, "The Virtual Midas Touch: Helping Behavior after a Mediated Social Touch," IEEE Trans. Haptics, vol. 2, no. 3, pp. 136-140, July/Sept. 2009.
[31] J. Bailenson and N. Yee, "Virtual Interpersonal Touch and Digital Chameleons," J. Nonverbal Behavior, vol. 31, pp. 225-242, 2007.
[32] M. Baumann, K. MacLean, T. Hazelton, and A. McKay, "Emulating Human Attention-Getting Practices with Wearable Haptics," Proc. IEEE Haptic Symp., 2010.
[33] C. Vaucelle and Y. Abbas, "Touch-Sensitive Apparel," Proc. SIGCHI '07, 2007.
[34] S. Yohanan and K. MacLean, "A Tool to Study Affective Touch: Goals & Design of the Haptic Creature," Proc. SIGCHI '09, pp. 4153-4158, 2009.
[35] H. Cramer, N. Kemper, A. Amin, and B. Wielinga, "'Give Me a Hug': The Effects of Touch and Autonomy on People's Responses to Embodied Social Agents," Computer Animation and Virtual Worlds, CAS 2009 special issue, vol. 20, pp. 437-445, 2009.
[36] K. Salminen, V. Surakka, J. Lylykangas, J. Raisamo, R. Saarinen, R. Raisamo, J. Rantala, and G. Evreinov, "Emotional and Behavioral Responses to Haptic Stimulation," Proc. SIGCHICHI '08, pp. 1555-1562, 2008.
[37] T. Bickmore, A. Gruber, and R. Picard, "Establishing the Computer-Patient Working Alliance in Automated Health Behavior Change Interventions," Patient Education and Counseling, vol. 59, pp. 21-30, 2005.
[38] N. Dahlback, A. Jonsson, and L. Ahrenberg, "Wizard of Oz Studies: Why and How," Proc. Int'l Conf. Intelligent User Interfaces, pp. 193-199, 1993.
[39] J. Russel, "A Circumplex Model of Affect," J. Personality and Social Psychology, vol. 39, pp. 1161-1178, 1980.
[40] J. Bland and D. Altman, "Calculating Correlation Coefficients with Repeated Observations: Part 1—Correlation within Subjects," British Medical J., vol. 18, pp. 310-446, 1995.
[41] E. Bedrick and C. Tsai, "Model Selection for Multivariate Regression in Small Samples," Biometrics, vol. 50, pp. 226-231, 1994.
[42] D. Bates and M. Maechler, "lme4: Linear Mixed-Effects Models Using S4 Classes. R Package Version 0.999375-32," http://CRAN.R-project.orgpackage=lme4, accessed 6/9/2010, 2009.
[43] R Core Development Team "R: A Language and Environment for Statistical Computing," R Foundation for Statistical Computing, http:/, 2009.
[44] R. Baayen, "languageR: Data Sets and Functions with "Analyzing Linguistic Data: A Practical Introduction to Statistics," http://CRAN.R-project.orgpackage=languageR , 2009.
[45] A. Mehrabian and S. Ferris, "Inference of Attitudes from Nonverbal Communication in Two Channels," J. Consulting Psychology, vol. 31, pp. 248-252, 1967.
[46] J. Cahn, "The Generation of Affect in Synthesized Speech," J. Am. Voice I/O Soc., vol. 8, pp. 1-19, 1990.
[47] R. Bakeman, "Recommended Effect Size Statistics for Repeated Measures Designs," Behavior Research Methods, vol. 37, pp. 379-384, 2005.
[48] S. Olejnik and J. Algina, "Generalized eta and Omega Squared Statistics: Measures of Effect Size for Some Common Research Designs," Psychological Methods, vol. 8, pp. 434-447, 2003.
[49] T. Bickmore and R. Fernando, "Towards Empathic Touch by Relational Agent," Proc. Autonomous Agents and Multiagent Systems Workshop on Empathic Agents, 2009.
[50] P. Anderse and K. Leibowitz, "The Development and Nature of the Construct Touch Avoidance," Environmental Psychology and Nonverbal Behavior, vol. 3, pp. 89-106, 1978.
[51] J. Wiggins, "A Psychological Taxonomy of Trait-Descriptive Terms," J. Personality and Social Psychology, vol. 37, pp. 395-412, 1979.
[52] A. Horvath and L. Greenberg, "Development and Validation of the Working Alliance Inventory," J. Counseling Psychology, vol. 36, pp. 223-233, 1989.
[53] M. Mori, "Bukimi No Tani' the Uncanny Valley," Energy, vol. 7, pp. 33-35, 1970.
[54] R. Wang and F. Queck, "Touch & Talk: Contextualizing Remote Touch for Affective Interaction," Proc. Fourth Int'l Conf. Tangible, Embedded, and Embodied Interaction, pp. 13-20, 2010.
[55] F. Rothling, R. Haschke, J. Steil, and H. Ritter, "Platform Portable Anthropomorphic Grasping with the Bielefeld 20-DOF Shadow and 9-DOF TUM Hand," Proc. IEEE/RSJ Int'l Conf. Intelligent Robots and Systems, 2007.
[56] A. Billard, B. Robins, J. Nadel, and K. Dautenhahn, "Building Robota, a Mini-Humanoid Robot for the Rehabilitation of Children with Autism," RESNA Assistive Technology J., vol. 19, pp. 37-49, 2006.
14 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool