New Signals in Multimedia Systems and Applications

Pablo Cesar, Centrum Wiskunde and Informatica
Vivek Singh, Rutgers University
Ramesh Jain, University of California, Irvine
Nicu Sebe, University of Trento, Italy
Nuria Oliver, Data-Pop Alliance

Pages: 12–13

Abstract—The objective of this special issue is to revisit how sensor technology is transforming the way context and human behavior is understood, enabling personal and enriched media experiences.

Keywords—multimedia; sensors; Guest Editor Introduction

Recent developments in mobile, sensor, and wearable technology are making available a plethora of new signals, which have the potential to enable truly personalized and enhanced media-based experiences. For example, today, a person’s every glance, heartbeat, emotion, movement, financial activity, and social activity can be digitally captured and shared with the community if the person chooses to allow it. This is rapidly reshaping the way we infer context, reason about it, and make decisions.

Consequently, multimedia computing—once the realm of audiovisual data—is embracing its “multi” nature, promoting research on the impact of other modalities and media. Sensors, in the broadest sense, will have a profound impact, shaking up the foundations of the field. This special issue explores how new sensor technology (from social sensors to biosensors) will affect multimedia systems and applications. In particular, it focuses on novel and groundbreaking research from the multimedia community on sensing, understanding, and reacting to the user experience. Such advancements are expected to have profound impacts on multiple aspects of human life, including health, culture, education, mobility, economics, and politics.

The objective of this special issue is to revisit how sensor technology is transforming the way context and human behavior is understood, enabling personal and enriched media experiences. We are especially interested in the different stages of the multimedia-processing lifecycle, from the moment multimodal signals are captured and processed, to when meaning is inferred and the environment reacts. The number and types of sensors are constantly increasing. Some examples include social sensors, accessories such as watches, and biosensors embedded in textiles. They provide a plethora of multimodal raw data that need to be processed and analyzed before we can gain actionable insights. Sensors provide immense amounts of data in relatively small amounts of time. Making sense of such information streams is extremely complex, and new algorithms are required. These algorithms should effectively assimilate data coming from different sensors for a better understanding of the user. We know, based on previous attempts, that it is extremely complicated to build such algorithms to robustly run “in the wild.” The final stage after sensing and understanding is reacting, which refers to the reactivity of the environment (the delivered multimedia content, the user’s surroundings, or the network used for communication). This piece of the puzzle provides the adaptation techniques that will improve the user experience.

This special issue includes three research papers and two Spotlight department articles, covering a variety of use scenarios and introducing novel solutions and architectures.

“Behavior Analysis through Multimodal Sensing for Care of Parkinson’s and Alzheimer’s Patients” (by Federico Alvarez et al.) describes a system that involves data capturing and multimodal fusion to extract relevant features and perform data analysis for providing useful recommendations. The proposed system gathers signals from diverse sources in health monitoring environments, understands the user behavior and context, and triggers proper actions for improving the patient’s quality of life. This article’s application area is independent living.

“Rhythm: A Unified Measurement Platform for Human Organizations” (by Oren Lederman et al.) introduces a platform that combines wearable electronic badges and online applications to capture team- and network-level interaction patterns in organizations and workspaces.

“Generalized Multi-Instance Control Mapping for Interactive Media Systems” (by Stephen Sinclair, Joseph Malloch, and Marcelo M. Wanderley) proposes a unified concept of virtual signal objects using an extension to control mapping, which is shown to be a useful metaphor in signal representation and control for multimedia. In the framework, signal instances remain independent between devices but are coordinated using an additional shared global identity. The article focuses on the application area of creativity and cognition.

“Health Media: From Multimedia Signals to Personal Health Insights” (by Susanne Boll, Jochen Meyer, and Noel E. O’Connor) is a Spotlight department article that explores the contribution of new signals in an emerging (broader) interpretation of multimedia for personal health. The authors review the new multimedia signals in health, the research challenges they pose, and the trends in multimedia they are driving. The article also discusses how core multimedia research is becoming an important enabler for applications with the potential for significant societal impact; the authors illustrate those applications with recent systems from their own and related work.

“Sensing Technologies for Monitoring Serious Mental Illnesses” (by Saeed Abdullah and Tanzeem Choudhury) is another Spotlight article that discusses the role that emerging sensors can play in effective symptom monitoring and personalized interventions that can significantly improve mental health care across different populations. Specifically, it discusses how sensing technologies can enable granular tracking of behavioral, physiological, and social signals relevant to mental health. These signals can provide valuable insights into personalized early-warning signs, leading to effective intervention strategies and better preemptive care.

Pablo Cesar is the head of the Distributed and Interactive Systems group at Centrum Wiskunde & Informatica (CWI) in Amsterdam and is an associate professor at Delft University of Technology. Contact him at or .
Vivek Singh is an assistant professor at Rutgers University and a visiting professor at the Massachusetts Institute of Technology Media Lab. Contact him at or .
Ramesh Jain is a professor in the Donald Bren School of Information and Computer Sciences at the University of California, Irvine. Contact him at or .
Nicu Sebe is a professor and the dean of the Department of Information Engineering and Computer Science at the University of Trento, Italy. Contact him at or .
Nuria Oliver is the chief data scientist at Data-Pop Alliance and the director of Research in Data Science at Vodafone Research. Contact her at or .
66 ms
(Ver 3.x)