The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - April-June (2013 vol.6)
pp: 193-205
F. Danieau , Technicolor Res. & Innovation, INRIA Rennes, Cesson-Sevigne, France
A. Lecuyer , INRIA, Rennes, France
P. Guillotel , Technicolor Res. & Innovation, Cesson-Sevigne, France
J. Fleureau , Technicolor Res. & Innovation, Cesson-Sevigne, France
N. Mollet , Technicolor Res. & Innovation, Cesson-Sevigne, France
M. Christie , IRISA, Rennes, France
ABSTRACT
Haptic technology has been widely employed in applications ranging from teleoperation and medical simulation to art and design, including entertainment, flight simulation, and virtual reality. Today there is a growing interest among researchers in integrating haptic feedback into audiovisual systems. A new medium emerges from this effort: haptic-audiovisual (HAV) content. This paper presents the techniques, formalisms, and key results pertinent to this medium. We first review the three main stages of the HAV workflow: the production, distribution, and rendering of haptic effects. We then highlight the pressing necessity for evaluation techniques in this context and discuss the key challenges in the field. By building on existing technologies and tackling the specific challenges of the enhancement of audiovisual experience with haptics, we believe the field presents exciting research perspectives whose financial and societal stakes are significant.
INDEX TERMS
Haptic interfaces, Vibrations, Rendering (computer graphics), Virtual reality, Visualization, Production, Multimedia communication,user experience, Haptic interfaces, multimedia, audiovisual
CITATION
F. Danieau, A. Lecuyer, P. Guillotel, J. Fleureau, N. Mollet, M. Christie, "Enhancing Audiovisual Experience with Haptic Feedback: A Survey on HAV", IEEE Transactions on Haptics, vol.6, no. 2, pp. 193-205, April-June 2013, doi:10.1109/TOH.2012.70
REFERENCES
[1] M. Heilig, "Sensorama Simulator," US Patent 3,050,870, Washington, D.C.: Patent and Trademark Office, Aug. 1962.
[2] G.C. Burdea and P. Coiffet, Virtual Reality Technology. John Wiley & Sons, 2003.
[3] S.J. Biggs and M. Srinivasan, "Haptic Interfaces," Handbook of Virtual Environment Technology, K.M. Stanney, ed., pp. 93-116. Lawrence Erlbaum Assoc., 2002.
[4] V. Hayward, O.R. Astley, M. Cruz-Hernandez, D. Grant, and G. Robles-De-La-Torre, "Haptic Interfaces and Devices," Sensor Rev., vol. 24, no. 1, pp. 16-29, 2004.
[5] M. Reiner, "The Role of Haptics in Immersive Telecommunication Environments," IEEE Trans. Circuits and Systems for Video Technology, vol. 14, no. 3, pp. 392-401, Mar. 2004.
[6] S. O'Modhrain and I. Oakley, "Touch TV: Adding Feeling to Broadcast Media," Proc. European Conf. Interactive Television: From Viewers to Actors, 2003.
[7] N. Magnenat-Thalmann and U. Bonanni, "Haptics in Virtual Reality and Multimedia," IEEE Multimedia, vol. 13, no. 3, pp. 6-11, July-Sept. 2006.
[8] A.E. Saddik, M. Orozco, M. Eid, and J. Cha, Haptic Technologies---Bringing Touch to Multimedia. Springer, 2011.
[9] T. Yamaguchi, A. Akabane, J. Murayama, and M. Sato, "Automatic Generation of Haptic Effect into Published 2D Graphics," Proc. Eurohaptics Conf., 2006.
[10] J. Cha, M. Eid, and A.E. Saddik, "Touchable 3D Video System," ACM Trans. Multimedia Computing, Comm., and Applications, vol. 5, pp. 1-25, Oct. 2009.
[11] J. Cha, Y.-S. Ho, Y. Kim, J. Ryu, and I. Oakley, "A Framework for Haptic Broadcasting," IEEE Multimedia, vol. 16, no. 3, pp. 16-27, July 2009.
[12] K. Kilkki, "Quality of Experience in Communications Ecosystem," J. Universal Computer Science, vol. 14, no. 5, pp. 615-624, 2008.
[13] J. Dionisio, "Projects in VR Virtual Hell: A Trip through the Flames," IEEE Computer Graphics and Applications, vol. 7, no. 3, pp. 11-14, May/June 1997.
[14] S. Palan, R. Wang, N. Naukam, L. Edward, and K.J. Kuchenbecker, "Tactile Gaming Vest (TGV)," http://iroboticist.com/2010/03/26tgv/, Sept. 2012.
[15] B. Lee, J. Lee, J. Cha, C. Seo, and J. Ryu, "Immersive Live Sports Experience with Vibrotactile Sensation," Proc. IFIP TC13 Int'l Conf. Human-Computer Interaction, pp. 1042-1045, 2005.
[16] S.U Rehman, "Turn Your Mobile into the Ball: Rendering Live Football Game Using Vibration," IEEE Trans. Multimedia, vol. 10, no. 6, pp. 1022-1033, Oct. 2008.
[17] P. Lemmens, F. Crompvoets, D. Brokken, J.V.D. Eerenbeemd, and G.-j. D. Vries, "A body - Conforming Tactile Jacket to Enrich Movie Viewing," Proc. Eurohaptics Conf. and Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 7-12, 2009.
[18] M. Waltl, Enriching Multimedia with Sensory Effects: Annotation and Simulation Tools for the Representation of Sensory Effects. VDM Verlag Saarbrücken, 2010.
[19] M.A. Rahman, A. Alkhaldi, and J. Cha, "Adding Haptic Feature to YouTube," Proc. Int'l Conf. Multimedia, pp. 1643-1646, 2010.
[20] Y. Kim, J. Cha, J. Ryu, and I. Oakley, "A Tactile Glove Design and Authoring System for Immersive Multimedia," IEEE Multimedia, vol. 17, no. 3, pp. 34-45, July-Sept. 2010.
[21] Y. Suzuki, M. Kobayashi, and A.J. Interface, "Air Jet Driven Force Feedback in Virtual Reality," IEEE Computer Graphics and Applications, vol. 25, no. 1, pp. 44-7, Jan./Feb. 2005.
[22] T. Hoshi, M. Takahashi, T. Iwamoto, and H. Shinoda, "Noncontact Tactile Display Based on Radiation Pressure of Airborne Ultrasound," IEEE Trans. Haptics, vol. 3, no. 3, pp. 155-165, July-Sept. 2010.
[23] J. Alexander, M.T. Marshall, and S. Subramanian, "Adding Haptic Feedback to Mobile TV," Proc. ACM Extended Abstracts of the SIGCHI Conf. Human Factors in Computing Systems, pp. 1975-1980, 2011.
[24] D. Gaw, D. Morris, and K. Salisbury, "Haptically Annotated Movies: Reaching Out and Touching the Silver Screen," Proc. Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 287-288, 2006.
[25] Y. Kim, S. Park, H. Kim, H. Jeong, and J. Ryu, "Effects of Different Haptic Modalities on Students' Understanding of Physical Phenomena," Proc. IEEE World Haptics Conf., pp. 379-384, 2011.
[26] S. O'Modhrain and I. Oakley, "Adding Interactivity: Active Touch in Broadcast Media," Proc. Int'l Conf. Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 293-294, 2004.
[27] F. Danieau, J. Fleureau, A. Cabec, P. Kerbiriou, P. Guillotel, N. Mollet, M. Christie, and A. Lécuyer, "A Framework for Enhancing Video Viewing Experience with Haptic Effects of Motion," Proc. IEEE Haptics Symp., pp. 541-546, 2012.
[28] K. Shimoga, "A Survey of Perceptual Feedback Issues in Dexterous Telemanipulation. II. Finger Touch Feedback," Proc. IEEE Virtual Reality Ann. Int'l Symp., pp. 271-279, 1993.
[29] L.A. Jones and M. Berris, "The Psychophysics of Temperature Perception and Thermal-Interface Design," Proc. IEEE Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 137-142, 2002.
[30] L.A. Jones, "Kinesthetic Sensing," Human and Machine Haptics. MIT Press, 2000.
[31] A. Berthoz, The Brain's Sense of Movement. Harvard Univ. Press, 2000.
[32] J.B.V. Erp, "Guidelines for the Use of Vibro-Tactile Displays in Human Computer Interaction," Proc. EuroHaptics Conf., pp. 18-22, 2002.
[33] K. Hale and K. Stanney, "Deriving Haptic Design Guidelines from Human Physiological, Psychophysical, and Neurological Foundations," IEEE Computer Graphics and Applications, vol. 24, no. 2, pp. 33-39, Mar./Apr. 2004.
[34] J. Smith and K. MacLean, "Communicating Emotion through a Haptic Link: Design Space and Methodology," Int'l J. Human-Computer Studies, vol. 65, no. 4, pp. 376-387, 2007.
[35] D. Tsetserukou, A. Neviarouskaya, H. Prendinger, N. Kawakami, and S. Tachi, "Affective Haptics in Emotional Communication," Proc. IEEE Int'l Conf. Affective Computing and Intelligent Interaction and Workshops, pp. 1-6, 2009.
[36] A. Brady, B. MacDonald, I. Oakley, S. Hughes, and S. O'Modhrain, "Relay: A Futuristic Interface for Remote Driving," Proc. EuroHaptics Conf., 2002.
[37] K. Kuchenbecker, J. Fiene, and G. Niemeyer, "Event-Based Haptics and Acceleration Matching: Portraying and Assessing the Realism of Contact," Proc. EuroHaptics Conf. and Symp. Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 381-387, 2005.
[38] T.L. McDaniel, K. Kahol, P. Tripathi, D.P. Smith, L. Bratton, R. Atreya, and S. Panchanathan, "A Methodology to Establish Ground Truth for Computer Vision Algorithms to Estimate Haptic Features from Visual Images," Proc. IEEE Int'l Workshop Haptic Audio Visual Environments and Their Applications, pp. 94-99, 2005.
[39] T. Tuytelaars and K. Mikolajczyk, "Local Invariant Feature Detectors: A Survey," Foundations and Trends in Computer Graphics and Vision, vol. 3, no. 3, pp. 177-280, 2007.
[40] W. Zhao, R. Chellappa, P. Phillips, and A. Rosenfeld, "Face Recognition: A Literature Survey," ACM Computing Surveys, vol. 35, no. 4, pp. 399-458, 2003.
[41] W. Hu, T. Tan, L. Wang, and S. Maybank, "A Survey on Visual Surveillance of Object Motion and Behaviors," IEEE Trans. Systems, Man and Cybernetics, Part C (Applications and Rev.), vol. 34, no. 3, pp. 334-352, Aug. 2004.
[42] B.T. Truong and S. Venkatesh, "Video Abstraction: A Systematic Review and Classification," ACM Trans. Multimedia Computing, Comm., and Applications, vol. 3, no. 1,article 3, 2007.
[43] X. Zhu, X. Wu, A.K. Elmagarmid, Z. Feng, and L. Wu, "Video Data Mining: Semantic Indexing and Event Detection from the Association Perspective," IEEE Trans Knowledge and Data Eng., vol. 17, no. 5, pp. 665-677, May 2005.
[44] P. Bach-y Rita and S.W. Kercel, "Sensory Substitution and the Human-Machine Interface," Trends in Cognitive Sciences, vol. 7, no. 12, pp. 541-546, 2003.
[45] A. Chang and C. O'Sullivan, "Audio-Haptic Feedback in Mobile Phones," Proc. Extended Abstracts of the Computer Human Interaction Conf. Human Factors in Computing Systems, pp. 1264-1267, 2005.
[46] S. Nanayakkara, E. Taylor, L. Wyse, and S. Ong, "An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair," Proc. ACM Int'l Conf. Human Factors in Computing Systems, pp. 337-346, 2009.
[47] T. Zhang and C.-C. Kuo, "Audio Content Analysis for Online AudioVisual Data Segmentation and Classification," IEEE Trans. Speech and Audio Processing, vol. 9, no. 4, pp. 441-457, May 2001.
[48] S. Chang, T. Sikora, and A. Purl, "Overview of the MPEG-7 Standard," IEEE Trans. Circuits and Systems for Video Technology, vol. 11, no. 6, pp. 688-695, June 2001.
[49] M. Enriquez and K. MacLean, "The Hapticon Editor: A Tool in Support of Haptic Communication Research," Haptic Interfaces for Virtual Environment and Teleoperator Systems, pp. 356-362, 2003.
[50] J. Ryu and S. Choi, "posVibEditor: Graphical Authoring Tool of Vibrotactile Patterns," Proc. IEEE Int'l Workshop Haptic Audio Visual Environments and Games, pp. 120-125, Oct. 2008.
[51] Y. Seo, B.-C. Lee, Y. Kim, J.-P. Kim, and J. Ryu, "K-HapticModeler: A Haptic Modeling Scope And Basic Framework," Proc. Int'l Workshop Haptic Audio Visual Environments and Their Applications, pp. 136-141, 2007.
[52] M. Eid, S. Andrews, A. Alamri, and A. El Saddik, "HAMLAT: A HAML-Based Authoring Tool for Haptic Application Development," Proc. EuroHaptics Conf., pp. 857-866, 2008.
[53] K. Yoon, B. Choi, E. Lee, and T. Lim, "4-D Broadcasting with MPEG_V," Proc. IEEE Int'l Conf. Multimedia Signal Processing, vol. 2, pp. 257-262, 2010.
[54] F. El-far, M. Eid, M. Orozco, and A. Saddik, "Haptic Applications Meta-Language," Proc. IEEE Int'l Symp. Distributed Simulation and Real-Time Applications, pp. 261-264, 2006.
[55] A. Sourin and L. Wei, "Visual Immersive Haptic Rendering on the Web," Proc. ACM Siggraph Int'l Conf. Virtual-Reality Continuum and Its Applications in Industry, p. 1, 2008.
[56] Y. Kim, H. Shin, and M. Hahn, "A Bidirectional Haptic Communication Framework and an Authoring Tool for an Instant Messenger," Proc. IEEE Int'l Advanced Comm. Technology, vol. 3, pp. 2050-2053, 2009.
[57] "ISO/IEC FDIS 23005-3 Information Technology—Media Context and Control—Part 3: Sensory Information," MPEG Group, 2011.
[58] "Information Technology---Coding of Audio-Visual Objects---Part 11: Scene Description and Application Engine," ISO/IEC 14496-11, 2005.
[59] D. Wu, Y.T. Hou, W. Zhu, Y.-Q. Zhang, and J.M. Peha, "Streaming Video over the Internet: Approaches and Directions," IEEE Trans. Circuits and Systems for Video Technology, vol. 11, no. 3, pp. 282-300, Mar. 2001.
[60] P.F. Hokayem and M.W. Spong, "Bilateral Teleoperation: An Historical Survey," Automatica, vol. 42, no. 12, pp. 2035-2057, 2006.
[61] E. Dijk, A. Weffers-Albu, and T. De Zeeuw, "A Tactile Actuation Blanket to Intensify Movie Experiences with Personalised Tactile Effects," Proc. Int'l Conf. Intelligent Technologies for Interactive Entertainment, pp. 2-3, 2010.
[62] A. Israr and I. Poupyrev, "Tactile Brush: Drawing on Skin with a Tactile Grid Display," Proc. SIGCHI Conf. Human Factors in Computing Systems, pp. 2019-2028, 2011.
[63] R. Lindeman, Y. Yanagida, H. Noma, and K. Hosaka, "Wearable Vibrotactile Systems for Virtual Contact and Information Display," Virtual Reality, vol. 9, no. 2, pp. 203-213, 2006.
[64] R. Jain, "Quality of Experience," IEEE Multimedia, vol. 11, no. 1, pp. 96-97, Jan.-Mar. 2004.
[65] A. Hamam, M. Eid, A.E. Saddik, and N.D. Georganas, "A Quality of Experience Model for Haptic User Interfaces," Proc. Ambi-Sys Workshop Haptic User Interfaces in Ambient Media Systems, pp. 1-6, 2008.
[66] M. Waltl and C. Timmerer, "Increasing the User Experience of Multimedia Presentations with Sensory Effects," Proc. 11th Int'l Workshop Image Analysis for Multimedia Interactive Services (WIAMIS), pp. 1-4, 2010.
[67] A. Hamam, N. Georganas, and A. El Saddik, "Effect of Haptics on the Quality of Experience," Proc. IEEE Int'l Symp. Haptic Audio-Visual Environments and Games (HAVE), pp. 1-6, 2010.
[68] M. Meehan, B. Insko, M. Whitton, and F.P.J. Brooks, "Physiological Measures of Presence in Stressful Virtual Environments," ACM Trans. Graphics, vol. 21, no. 3, pp. 645-652, 2002.
[69] M. Schuemie, P. van der Straaten, M. Krijn, and C. van der Mast, "Research on Presence in Virtual Reality: A Survey," CyberPsychology and Behavior, vol. 4, no. 2, pp. 183-201, 2001.
[70] R.L. Mandryk, K.M. Inkpen, and T.W. Calvert, "Using Psychophysiological Techniques to Measure User Experience with Entertainment technologies," Behavior and Information Technology, vol. 25, no. 2, pp. 141-158, 2006.
[71] J. Fleureau, P. Guillotel, and Q. Huynh-Thu, "Physiological-Based Affect Event Detector for Entertainment Video Applications," IEEE Trans. Affective Computing, vol. 3, no. 3, pp. 379-385, July-Sept. 2012.
15 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool