The Community for Technology Leaders
RSS Icon
Issue No.02 - Second (2012 vol.5)
pp: 172-183
Francisco CMB Oliveira , Ceará State University, Brazil
Francis Quek , Virgina Tech, Blacksburg
Heidi Cowan , Wright State University, Dayton
Bing Fang , Virgina Tech, Blacksburg
Mathematics instruction and discourse typically involve two modes of communication: speech and graphical presentation. For the communication to remain situated, dynamic synchrony must be maintained between the speech and dynamic focus in the graphics. In sighted people, vision is used for two purposes: access to graphical material and awareness of embodied behavior. This embodiment awareness keeps communication situated with visual material and speech. Our goal is to assist students who are blind or visually impaired (SBVI) in the access to such instruction/communication. We employ the typical approach of sensory replacement for the missing visual sense. Haptic fingertip reading can replace visual material. We want to make the SBVI aware of the deictic gestures performed by the teacher over the graphic in conjunction with speech. We employ a haptic glove interface to facilitate this embodiment awareness. We address issues from the conception through the design and implementation to the effective and successful use of our Haptic Deictic System (HDS) in inclusive classrooms.
Haptic glove, multimodal interaction, discourse, gaming, embodied skill, assistive technology.
Francisco CMB Oliveira, Francis Quek, Heidi Cowan, Bing Fang, "The Haptic Deictic System—HDS: Bringing Blind Students to Mainstream Classrooms", IEEE Transactions on Haptics, vol.5, no. 2, pp. 172-183, Second 2012, doi:10.1109/TOH.2011.35
[1] F. Oliveira, H. Cowan, B. Fang, and F. Quek, "Fun to Develop Embodied Skill: How Games Help the Blind to Understand Pointing," Proc. Third Int'l Conf. Pervasive Technologies Related to Assistive Environments (Petra '10), 2010.
[2] D. McNeill, Hand and Mind: What Gestures Reveal about Thought. Univ. of Chicago Press, 1992.
[3] H.H. Clark and C.R. Marshall, "Definite Reference and Mutual Knowledge," Psycholinguistics: Critical Concepts in Psychology, p. 414, 2002.
[4] H.H. Clark, Using Language. Cambridge Univ. Press, 1996.
[5] R. Penrose, The Emperor's New Mind. Oxford Univ. Press, 1989.
[6] D. McNeill, Gesture and Thought. Univ. of Chicago Press, 2005.
[7] J.M. Iverson and S. Goldin-Meadow, "Why People Gesture as They Speak," Nature, vol. 396, p. 228, 1998.
[8] S. Goldin-Meadow, "The Role of Gesture in Communication and Thinking," Trends in Cognitive Sciences, vol. 3, no. 11, pp. 419-429, 1999.
[9] J.M. Kennedy, Drawing and the Blind. Yale Press, 1993.
[10] J.J. Gibson, "Observations on Active Touch," Psychological Rev., vol. 69, no. 6, pp. 477-491, 1962.
[11] P. Bach-y-Rita and Brain, Mechanisms in Sensory Substitution. Academic Press, Inc., 1972.
[12] A. D'Angiulli, J. Kennedy, and M. Helle, "Blind Children Recognizing Tactile Pictures Respond like Sighted Children Given Guidance in Exploration," Scandinavian J. Psychology, vol. 39, no. 3, pp. 187-190, 1998.
[13] M. Jeannerod, "The Representing Brain: Neural Correlates of Motor Intention and Imagery," Behavioral and Brain Sciences, vol. 17, no. 2, pp. 187-245, 1994.
[14] N. Kerr, "The Role of Vision in 'Visual Imagery' Experiments: Evidence from the Congenitally Blind," J. Experimental Psychology General, vol. 112, no. 2, pp. 265-77, 1983.
[15] T. Dick and E. Kubiak, "Issues and Aids for Teaching Mathematics to the Blind," Math. Teacher, vol. 90, no. 5, pp. 344-349, 1997.
[16] Y. Hatwell, A. Streri, and E. Gentaz, Touching for Knowing, pp. 67-83. John Benjamins Publishing Co., 2000.
[17] H. Segond, D. Weiss, and E. Sampaio, "Human Spatial Navigation via a Visuo-Tactile Sensory Substitution System," Perception, vol. 34, pp. 1231-1249, 2005.
[18] D. McGookin and S. Brewster, "MultiVis: Improving Access to Visualizations for Visually Impaired People," CHI: Proc. Extended Abstracts on Human Factors in Computing Systems, 2006.
[19] Sensable, "Products and Services," http:/, Last Checked: 3/19/2009.
[20] L. Wells and S. Landau, "Merging of Tactile Sensory Input and Audio Data by Means of the Talking Tactile Tablet," Proc. Eurohaptics, 2003.
[21] VTPlayer. http:/, Last Checked: 01/04/2010.
[22] S.A. Wall and S. Brewster, "Sensory Substitution Using Tactile Pin Arrays: Human Factors, Technology and Applications," Signal Processing, vol. 86, no. 12, pp. 3674-3695, 2006.
[23] S.A. Wall and S.A. Brewster, "Tac-Tiles: Multimodal Pie Charts for Visually Impaired Users," Proc. the Fourth Nordic Conf. Human-Computer Interaction: Changing Roles, 2006.
[24] G. Jansson and P. Pedersen, "Obtaining Geographical Information from a Virtual Map with a Haptic Mouse," Proc. 22nd Int'l Cartographic Conf. (ICC '05), 2005.
[25] M.S. Manshad and A.S. Manshad, "Multimodal Vision Glove for Touchscreens," Proc. 10th Int'l ACM SIGACCESS Conf. Computers and Accessibility, 2008.
[26] F. Winberg and J. Bowers, "Assembling the Senses: Towards the Design of Cooperative Interfaces for Visually Impaired Users," Proc. ACM Conf. Computer Supported Cooperative Work, pp. 332-341, 2004.
[27] G.L. Lohse, "Models of Graphical Perception," Handbook of Human-Computer Interaction, Elsevier 1997.
[28] E. Mynatt and G. Weber, "Nonvisual Presentation of Graphical User Interfaces," CHI: Proc. ACM SIGCHI Conf. Human Factors in Computing, 1994.
[29] R. Verrillo and G. Gescheider, "Perception via the Sense of Touch," Tactile Aids for the Hearing Impaired, pp. 1-36, John Wiley and Sons 1992.
[30] R.J. Christman, Sensory Experience. Harper and Row Publishers, 1979.
[31] K. KACZMAREK and J. WEBSTER, "Electrotactile and Vibrotactile Displays for Sensory Substitution Systems," IEEE Trans. Biomedical Eng., vol. 38, no. 1, pp. 1-16, Jan. 1991.
[32] L. Dipietro, A.M. Sabatini, and P. Dario, "A Survey of Glove-Based Systems and Their Applications," IEEE Trans. Systems, Man, and Cybernetics, Part C: Applications and Rev., vol. 38, no. 4, pp. 461-482, July 2008.
[33] A. Mulder and S. Fels, "Sound Sculpting: Manipulating Sound through Virtual Sculpting," Proc. Western Computer Graphics Symp., 1998.
[34] J.S. Zelek, "A Haptic Glove as a Tactile-Vision Sensory Substitution for Wayfinding," J. Visual Impairment & Blindness, vol. 97, no. 10, pp. 1-24, 2003.
[35] H. Tan, A. Lim, and R. Traylor, "A Psychophysical Study of Sensory Saltation with an Open Response Paradigm," Proc. Ninth Int'l Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2000.
[36] J.B.F. Van Erp et al., "Waypoint Navigation with a Vibrotactile Waist Belt," ACM Trans. Applied Perception, vol. 2, no. 2, pp. 106-117, 2005.
[37] J.B.F. van Erp, "Guidelines for the Use of Vibro-Tactile Displays in Human Computer Interaction," Proc. Eurohaptics, 2002.
[38] J. Craig, "Difference Threshold for Intensity of Tactile Stimuli," Perception & Psychophysics, vol. 11, no. 2, pp. 150-152, 1972.
[39] G. Goff, "Differential Discrimination of Frequency of Cutaneous Mechanical Vibration," J. Experimental Psychology, vol. 74, no. 2, pp. 294-299, 1967.
[40] C. Jay, M. Glencross, and R. Hubbold, "Modeling the Effects of Delayed Haptic and Visual Feedback in a Collaborative Virtual Environment," ACM Trans. Computer-Human Interaction, vol. 14, p. 8, 2007.
[41] G. Gescheider, S. Bolanowski, and R. Verrillo, "Temporal Relations in Cutaneous Stimulation," Proc. Conf. Cutaneous Comm. Systems and Devices, pp. 33-37, 1974.
[42] L. Petrosino and D. Fucci, "Temporal Resolution of the Aging Tactile Sensory System," Perceptual and Motor Skills, vol. 68, no. 1, pp. 288-290, 1989.
[43] R. Verrillo and G. Gescheider, "Enhancement and Summation in the Perception of two Successive Vibrotactile Stimuli," Percept Psychophysics, vol. 18, no. 2, pp. 128-136, 1975.
[44] R. Verrillo, "Temporal Summation in Vibrotactile Sensitivity," The J. the Acoustical Soc. Am., vol. 37, no. 5, pp. 843-846, 1965.
[45] F.A. Geldard, Sensory Saltation: Metastability in the Perceptual World. Lawrence Erlbaum Assoc., 1975.
[46] J. Kirman, "Tactile Apparent Movement—The Effects of Interstimulus Onset Interval and Stimulus Duration," Perception and Psychophysics, vol. 15, no. 1, pp. 1-6, 1974.
[47] C.D. Wickens and J.G. Hollands, Engineering Psychology and Human Performance. Prentice Hall, 2001.
[48] F. Oliveira and F. Quek, "A Multimodal Communication with a Haptic Glove: On the Fusion of Speech and Deictic over a Raised Line Drawing," Proc. Petra—The First Int'l Conf. PErvasive Technologies Related to Assistive Environments, 2008.
[49] K. Sathian, "Practice Makes Perfect: Sharper Tactile Perception in the Blind," Neurology, vol. 54, no. 12, pp. 2203-2204, 2000.
[50] C. Spence, M. Nicholls, and J. Driver, "The Cost of Expecting Events in the Wrong Sensory Modality," Perception & Psychophysics, vol. 63, no. 2, pp. 330-336, 2001.
[51] H.H. Clark, Arenas of Language Use. Univ. of Chicago Press, 1992.
[52] M. Wilson, "Six Views of Embodied Cognition," Psychnomic Bull. and Rev., vol. 9, no. 4, pp. 625-636, 2002.
[53] R.T. Rose, F. Quek, and Y. Shi, "MacVisSTA: A System for Multimodal Analysis," Proc. Sixth Int'l Conf. Multimodal Interfaces, 2004.
[54] D. McNeill, "Gesture, Gaze, and Ground," Proc. Second Int'l Workshop Machine Learning for Multimodal Interaction, 2006.
[55] A. Monk, Common Ground in Electronically Mediated Communication, pp. 265-286. Morgan & Claypool Publishers, 2003.
[56] D.G. Tatar, G. Foster, and D.G. Bobrow, "Designing for Conversation: Lessons From Cognoter," Int'l J. Man-Machine Studies, vol. 34, no. 2, pp. 185-209, 1991.
[57] P. Dourish, Where the Action Is: The Foundations of Embodied Interaction. MIT Press, 2001.
[58] A.D. Fisk and A. Schneider, "Controlled and Automatic Processing During Tasks Requiring Sustained Attention," Human Factors, vol. 23, no. 6, pp. 737-750, 1981.
[59] A. Hodzic et al, "Improvement and Decline in Tactile Discrimination Behavior After Cortical Plasticity Induced by Passive Tactile Coactivation," J. Neuroscience, vol. 24, no. 2, pp. 442-446, 2004.
[60] M. Csikszentmihalyi, Flow: The Psychology of Optimal Experience. Harper & Row, 1990.
[61] J. Chen, "Flow in Games (and Everything Else)," Comm. ACM, vol. 50, no. 4, pp. 31-34, 2007.
[62] D. Staub and C. Peck, "What Are the Outcomes for Nondisabled Students?," Educational Leadership, vol. 52, no. 4, pp. 36-40, 1994.
[63] E. Baker, M. Wang, and H. Walberg, "The Effects of Inclusion on Learning," Educational Leadership, vol. 52, no. 4, pp. 32-35, 1994.
[64] R. Baear, R.W. Flexer, and R.K. McMahan, "Transition Models and Promising Practices," Transition Planning for Secondary Students with Disabilities, Pearson/Merrill Prentice Hall, 2005.
[65] C. Smith, A. Walton, A. Loveland, G. Umberger, R. Kryscio, and D. Gash, "Memories that Last in Old Age: Motor Skill Learning and Memory Preservation," Neurobiology of Aging, vol. 26, no. 6, pp. 883-890, 2005.
22 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool