1070-986X/04/$31.00 © 2004 IEEE
Published by the IEEE Computer Society
Multisensory Communication and Experience through Multimedia
Successful communication involves more than communicating information—it involves communicating experience. Transferring multimodal data without concern for whether this information can transcend into a consistent multisensory experience for the receiver doesn't address the full spectrum of communication. How data transcends into experience and how experience is an important component of knowledge are crucial aspects that this special issue intends to address.
With this special issue, we focus the multimedia community's attention on future forms of mediated communication that will involve all or most of our senses. Our goal is to investigate the role that these multisensory experiences and expressive communications will play in the development of multimedia technologies and content. We give special attention to mediated experiential communication, especially exploring how we can enrich and exploit expressive gesture communication (for example, using multimedia systems with music signals and full-body movement to enhance user engagement).
Research on multifaceted aspects of multisensory communication and experience through multimedia is attracting a growing interest from both the academic and industrial communities. On the one hand, recent scientific developments on cognition, affect, emotion, expressive gesture, multimodal interfaces, and displays are opening new possibilities for integrating more sophisticated models of human behavior in computer systems. On the other hand, the emergence of "disappearing" computers and multimodal interfaces embedded in the natural environments of users are enabling the tackling of complex meaning and subtleties of human interaction in multimedia systems. This emergence thereby enables a deeper, user-centered, enhanced physical participation and experience in the human–machine interaction process.
Several examples bear witness to the importance of this topic. Research programs by the European Commission (EC; see http://www.cordis.lu/ist) support several initiatives, such as the Information Society Technologies' Future and Emerging Technologies (IST FET; see for example the Sounding Object project at http://www.soundingobject.org). Another such initiative is the IST Programme "Interfaces and Enhanced Audio–Visual Services" (see the Multisensory Expressive Gesture Applications project at http://www.megaproject.org).
Significant research results from these and other EC projects can be found elsewhere. 1
In the sixth framework program of the EC, the IST strategic objective for multimodal interfaces includes
• the Network of Excellence on Enactive Interfaces,
• the STREP Project of Tangible Acoustic Interfaces for Computer–Human Interaction (TAI-CHI), and
• the Humaine Network of Excellence on computational models of emotions.
Numerous academic and industry research groups in the US are also investigating a wide scope of multimodal communication and mediated experiences issues. Examples include, but are not limited to, the
• tangible interfaces, media fabrics, and affective computing groups at the Massachusetts Institute of Technology's Media Lab;
• experiential computing group at Georgia Tech;
• physical modeling of sound experiences at Princeton;
• media presence research group at Microsoft;
• Multimedia Communications Research Laboratory at Bell Laboratories; and
• pervasive computing projects at IBM.
New hybrid, transdisciplinary academic programs are being created for the investigation of the mediation of experiences (the arts, media, and engineering program at Arizona State University), the relationships of arts and computing at UC Irvine, the future of entertainment technologies at Carnegie Mellon, and the future of communication technologies at the University of California, San Diego.
Research and industry projects are active and have been for more than 10 years in Japan on Kansei information processing, such as the Humanoid Project at Waseda University in Tokyo. 2
We as a society can reap many benefits from research on multisensory communication and experience through multimedia. The articles we've selected demonstrate some of the ways we can apply these advances, such as
• communications within the home (Chandrasiri et al.);
• distributed environments and avatars (Cavazza et al.);
• entertainment and the performing arts (in Gutierrez et al., users can conduct a virtual orchestra or play a virtual musical instrument in an interactive mixed reality audio–visual environment); and
• therapy and rehabilitation (Hunt et al.).
These articles also demonstrate that progress is being made in solving some important low-level issues, like the synchronization of digital sound and image gestures in real-time interactive environments. They show enormous potential from cross-fertilization of more traditional scientific theories and technology with theories and work from the humanities and arts. 3
In cases where this integration succeeds, the benefits are evident (see for example Hunt et al.'s article on music therapy).
Additionally, the articles demonstrate that we're only at the beginning of an exciting journey toward the development of multisensory communication and experience through media. Successful development of such mediation systems will necessitate even greater integration of the engineering, art, humanities, and science disciplines. A fully integrated, transdisciplinary approach will help us address some of our greatest challenges.
We will, for example, be able to investigate cohesively developing content-processing capabilities in the cognitive, as well as in the affective/emotive, and motoric domains. We'll be able to approach digital mediation not only at lower levels (such as the signal level) but also at higher structural levels (for example, gestures and phrases) 4
and higher communication levels (moving into context-dependent and user dependent meanings) 5
. We can overcome the artificial separations existing between development of media technology, content, and applications and present media development as one coherent process. Finally, the new media emerging from such integrated work will not only concentrate on mediating experiences of the past (such as conducting a standard orchestra) but will mediate new experiences arising out of our digitally enabled reality, where we see, hear, remember, compute, express, and communicate in ways previously not possible.
is an associate professor at the University of Genova (Faculty of Engineering). He is also the founder and scientific director of the InfoMus Lab ( http://www.infomus.dist.unige.it) at the Department of Informatics, Systems, and Telematics at the-University of Genova, Italy. His research interests include sound and music computing, multimodal intelligent interfaces, computational models of nonverbal expressive gesture, and interactive multimodal-multimedia systems for museum, theater, music, entertainment, therapy, and rehabilitation. He is currently an associate editor for the Journal of New Music Research
is a professor and the director of the Arts Media and Engineering Program ( http://ame.asu.edu) at Arizona State University. His research work and publications are in the areas of computer-mediated arts, experiential systems, pitch perception, auditory feedback for rehabilitation, computer music tools for arts education, multimedia composition algorithms, and interdisciplinary graduate education.