, University of Genova
, Arizona State University
Pages: pp. 17-19
Successful communication involves more than communicating information—it involves communicating experience. Transferring multimodal data without concern for whether this information can transcend into a consistent multisensory experience for the receiver doesn't address the full spectrum of communication. How data transcends into experience and how experience is an important component of knowledge are crucial aspects that this special issue intends to address.
With this special issue, we focus the multimedia community's attention on future forms of mediated communication that will involve all or most of our senses. Our goal is to investigate the role that these multisensory experiences and expressive communications will play in the development of multimedia technologies and content. We give special attention to mediated experiential communication, especially exploring how we can enrich and exploit expressive gesture communication (for example, using multimedia systems with music signals and full-body movement to enhance user engagement).
Research on multifaceted aspects of multisensory communication and experience through multimedia is attracting a growing interest from both the academic and industrial communities. On the one hand, recent scientific developments on cognition, affect, emotion, expressive gesture, multimodal interfaces, and displays are opening new possibilities for integrating more sophisticated models of human behavior in computer systems. On the other hand, the emergence of "disappearing" computers and multimodal interfaces embedded in the natural environments of users are enabling the tackling of complex meaning and subtleties of human interaction in multimedia systems. This emergence thereby enables a deeper, user-centered, enhanced physical participation and experience in the human–machine interaction process.
Several examples bear witness to the importance of this topic. Research programs by the European Commission (EC; see http://www.cordis.lu/ist) support several initiatives, such as the Information Society Technologies' Future and Emerging Technologies (IST FET; see for example the Sounding Object project at http://www.soundingobject.org). Another such initiative is the IST Programme "Interfaces and Enhanced Audio–Visual Services" (see the Multisensory Expressive Gesture Applications project at http://www.megaproject.org).
Significant research results from these and other EC projects can be found elsewhere. 1 In the sixth framework program of the EC, the IST strategic objective for multimodal interfaces includes
Numerous academic and industry research groups in the US are also investigating a wide scope of multimodal communication and mediated experiences issues. Examples include, but are not limited to, the
New hybrid, transdisciplinary academic programs are being created for the investigation of the mediation of experiences (the arts, media, and engineering program at Arizona State University), the relationships of arts and computing at UC Irvine, the future of entertainment technologies at Carnegie Mellon, and the future of communication technologies at the University of California, San Diego.
Research and industry projects are active and have been for more than 10 years in Japan on Kansei information processing, such as the Humanoid Project at Waseda University in Tokyo. 2
We as a society can reap many benefits from research on multisensory communication and experience through multimedia. The articles we've selected demonstrate some of the ways we can apply these advances, such as
These articles also demonstrate that progress is being made in solving some important low-level issues, like the synchronization of digital sound and image gestures in real-time interactive environments. They show enormous potential from cross-fertilization of more traditional scientific theories and technology with theories and work from the humanities and arts. 3 In cases where this integration succeeds, the benefits are evident (see for example Hunt et al.'s article on music therapy).
Additionally, the articles demonstrate that we're only at the beginning of an exciting journey toward the development of multisensory communication and experience through media. Successful development of such mediation systems will necessitate even greater integration of the engineering, art, humanities, and science disciplines. A fully integrated, transdisciplinary approach will help us address some of our greatest challenges.
We will, for example, be able to investigate cohesively developing content-processing capabilities in the cognitive, as well as in the affective/emotive, and motoric domains. We'll be able to approach digital mediation not only at lower levels (such as the signal level) but also at higher structural levels (for example, gestures and phrases) 4 and higher communication levels (moving into context-dependent and user dependent meanings) 5. We can overcome the artificial separations existing between development of media technology, content, and applications and present media development as one coherent process. Finally, the new media emerging from such integrated work will not only concentrate on mediating experiences of the past (such as conducting a standard orchestra) but will mediate new experiences arising out of our digitally enabled reality, where we see, hear, remember, compute, express, and communicate in ways previously not possible.