Computing is leading a revolution—everything we do is changing at a pace never experienced before in human history. Whether we talk about the pervasive, ubiquitous, mobile, grid, or even the social computing revolution, we can be sure that computing is impacting the way we interact with each other, the way we design and build our homes and cities, the way we learn, the way we communicate, the way we play, the way we work. Simply put, computing technologies are increasingly affecting and transforming almost every aspect of our daily lives.
Unfortunately, the changes are not always positive, and much of the technology we use is clunky, unfriendly, unnatural, culturally biased, and difficult to use. As a result, several aspects of daily life are becoming increasingly complex and demanding.
We have access to huge amounts of information, much of which is irrelevant to our own local sociocultural context and needs or is inaccessible because it is not available in our native language, we cannot fully utilize the existing tools to find it, or such tools are inadequate or nonexistent. Thanks to computing technologies, our options for communicating with others have increased, but that does not necessarily mean that our communications have become more efficient. Furthermore, our interactions with computers remain far from ideal, and too often only literate, educated individuals who invest significant amounts of time in using computers can take direct advantage of what computing technologies have to offer.
We could argue that knowledge and communications are two of the main pillars of any society. As information of all kinds—on products, services, people, maps, and so on—increasingly forms part of the digital ecosystem, the computing technologies we develop become, paradoxically, both the gateways to all kinds of resources and the barriers to access them. Therefore, in addition to having become critical in improving lives, computing is now essential to the livelihood not only of the relatively few of us who have the necessary skills and access to resources, but potentially for everyone. 1,2
For the most part, however, the computing community designs and implements computing algorithms and technologies without fully taking into account our cognitive abilities, the ways we perceive and handle information, go about our work and life, create and maintain our social relations, or use our cultural context. That is, researchers and engineers often develop computing technologies in relative isolation.
Most current methodologies start with an idea that builds on or improves existing technologies or that solves problems in a particular technological domain, largely ignoring human issues. The obvious outcomes are more powerful, less expensive computers that are more difficult to use, in some cases even effectively slower and less accessible to most of the world's population. Given the current rate of penetration of computing technologies (directly or indirectly) in almost every imaginable human activity, it is clear that the existing computing research and development models are no longer adequate. People who adapt more quickly to technology (whether the technology is good or bad) have greater opportunities to benefit at many levels.
Developing technologies in relative human isolation does not contribute to alleviating the problems of wealth distribution, sustainability, and access to healthcare and education. Technologies that are difficult to use not only waste our time and make our lives worse, they make access to important resources more difficult, particularly for people who do not have the education or language skills of the minority who develop those technologies.
From this perspective, the current path of development of computer technologies is not only detrimental but also dangerous because it contributes to increasing the gap between the educated and uneducated, and between the rich and poor. The problem is even greater if we consider that the difficulties in using much of the technology available today are leading to a rapidly increasing content gap—most digital information is produced in developed countries, from particular cultural perspectives, and in only a few languages. To make things worse, access to the information itself is through technology developed without considering the local sociocultural context of the majority of the world's population.
Given the developments in recent years in decreased computing costs, increased wireless communications, and the spread of the Internet, the time is ripe for a major shift in the computing revolution. In our opinion, the goal of human-centered computing is to focus the computing revolution on addressing human abilities and needs.
HCC is a set of methodologies that apply to any field that uses computers in applications in which people directly interact with devices or systems that use computer technologies. The field is emerging from the convergence of multiple disciplines that are concerned both with understanding human beings and with the design of computational artifacts. HCC researchers and designers have a range of backgrounds and interests, from computer science, sociology, psychology, and cognitive science to engineering, graphic design, industrial design, and so on.
HCC studies the design, development, and deployment of mixed-initiative human-computer systems ( http://is.arc.nasa.gov/HCC/intro.html). It embodies a systems view that includes computational tools, cognitive and social systems, and physical facilities and environments. HCC "inherits the complexity of software engineering and systems integration, plus modeling of human-machine and human-human interaction. Advances in theory and modeling require systematic data on such interactions in realistically complex environments." HCC facilitates the design of effective computer systems that take into account personal, social, and cultural aspects and addresses issues such as information design, human-information interaction, human-computer interaction, human-human interaction, and the relationships between computing technology and art, social, and cultural issues.
HCC research has multiple goals and has been carried out under this name at several institutions since at least the early 1990s ( www.byte.com/art/9404/sec6/art4.htm). Some researchers have focused on understanding people, both as individuals and in social groups, by studying the ways they adopt and adapt computational technologies in their lives, while others have focused on developing new design strategies for computational artifacts ( www.human-centered.org).
Human-centered design of computational tools focuses on problems that traditional human-computer interaction (HCI) does not generally address. Traditional design approaches often include heuristic evaluations and measurements of productivity and efficiency. HCC researchers also bring a diverse array of conceptual and research tools to traditional computing areas.
HCC is not just about the interaction, the interface, or the design process. It is about knowledge, people, technology, and everything that ties them together. This includes how the technology is actually used and in what context. Furthermore, HCC involves both the creation of theoretical frameworks and the design and implementation of technical approaches and systems in many areas ( www.nsf.gov/cise/iis/about.jsp).
Both HCC's scope and its possibilities are wide, but in our view, three factors form the core of HCC system and algorithm design processes:
• augmenting or taking into account individual human abilities and limitations,
• social and cultural awareness, and
• adaptability across individuals and specific situations.
If we consider these factors in the design of systems and algorithms, HCC applications should exhibit the following qualities:
• integrate input from different types of sensors and communicate through a combination of media as output,
• act according to the social and cultural context in which they are deployed, and
• be useful to diverse individuals in their daily life.
In considering the scope, application areas, and qualities of HCC systems, much can be learned from fields and disciplines related to HCC.
HCC, HCI, and Human Computation
Many important contributions to HCC have been made in fields such as HCI, computer-supported cooperative work (CSCW), user-centered design, 3
cognitive psychology, sociology, anthropology, and others.
User-centered design can be characterized as a multistage problem-solving process that requires designers not only to analyze and foresee how users are likely to use an interface but also to test the validity of their assumptions with regard to user behavior in the real world. 4
CSCW combines the understanding of the way people work in groups with the enabling technologies of computer networking and associated hardware, software, services, and techniques. 5
In contrast, HCC covers more than the traditional areas of usability engineering, HCI, and human factors, which are primarily concerned with user interfaces or user interaction. 6
Compared to HCI, HCC has a twofold perspective ( www.cs.berkeley.edu/~jfc/hcc):
• HCC is "conceived as a theme that is important for all computer-related research, not as a field that overlaps or is a subdiscipline of computer science"; and
• HCC acknowledges that "computing connotes both concrete technologies (that facilitate various tasks) and a major social and economic force."
Interactive evolutionary computation, an interesting technique first developed in the early 1990s and more recently known as human computation
also puts the human at the center, but in a different way. In traditional computation, a human provides a formalized problem description to a computer and receives a solution to interpret. In human computation, the roles are reversed: The computer asks a person or a large number of people to solve a problem, then collects, interprets, and integrates their solutions. In other words, the computer "asks" the human to do the work it cannot do. This is done as a way to overcome technical difficulties: Instead of trying to get computers to solve problems that are too complex, use human beings.
In human computation, the human is helping the computer solve difficult problems, but in HCC, the computer helps humans maximize their abilities regardless of the situation—the human need not be in front of a computer performing a computing task when dealing with the computer.
Although HCC and human-based computation approach computing from two different perspectives, they both try to maximize the synergy between human abilities and computing resources. Work in human computation can therefore be of significant importance to HCC. On one hand, data collected through human computation systems can be valuable for developing machine-learning models. On the other hand, it can help us better understand human behavior and abilities, again of direct use in HCC algorithm development and system design.
Research Agenda and Key HCC Challenges
HCC systems should be multimodal—processing inputs and outputs in a naturally rich communication channel; they must be proactive—understanding cultural and social contexts and responding accordingly; and they must be easily accessible outside the desktop to a wide range of users. 8
A human-centered approach to computing thus needs to consider how human beings understand and interpret multimedia signals at the perceptual, cognitive, and affective levels, and how we interact naturally—embedding the cultural and social contexts as well as personal factors such as emotion, attitude, and attention. This requires considering work in fields such as neuroscience, psychology, cognitive science, and others. A key challenge is incorporating what is known in those fields within computational frameworks that integrate different media.
Research on machine learning integrated with domain knowledge, data mining, sensor fusion, and multimodal interaction will play a key role. 9
Further research into quantifying human-related knowledge is necessary, which means developing new theories and mathematical models of multimedia integration at multiple levels. We believe that an HCC research agenda will involve a nonexhaustive list of goals including the following: 6
• new human-centered methodologies for the design of models and algorithms and the development of diverse HCC systems;
• focused research on the integration of multiple sensors, media, and human sciences that have people as the central point;
• new interdisciplinary academic and industrial programs, initiatives, and meeting opportunities;
• discussions on the impact of computing technology that include the social, economic, and cultural contexts in which such technology is or might be deployed;
• research data that reflects human-centered approaches—for example, rich data collected from real multisensorial and culturally diverse social situations;
• common computing resources—for example, software tools and platforms;
• evaluation metrics for theories, design processes, implementations, and systems from a human-centered perspective; and
• methodologies for privacy protection and the consideration of ethical and cultural issues.
Clearly, an HCC research agenda should include a broad understanding and a multidisciplinary approach, as Eric Brewer and colleagues proposed in the specific context of developing regions. 1
The call for papers for this special issue prompted an extraordinary response from the computing community. We received 130 submissions, which in our opinion reflects the increasing interest in HCC. The number of submissions, as well as the scope of the work submitted, is also indicative of the challenges in HCC. Although research is being carried out in many areas, much work needs to be done to define HCC's theoretical foundations and to gain a clearer understanding of what constitutes an HCC methodology or application. One goal of this special issue and of the newly created IEEE Computer Society Task Force on Human-Centered Computing is to restart the discussions on the foundational aspects of HCC ( www.human-centered.org).
For this special issue, we selected four articles that cover application areas of interest to HCC researchers and practitioners. The articles exemplify some of the important principles in human-centered computing and represent some current trends and developments in the field.
In "A Communication Support System for Older People with Dementia," Norman Alm and colleagues present a computer-based communication support system that demonstrates how simple technology can be used effectively to assist older people with dementia in carrying out conversations. The work presented in this article is truly a multidisciplinary HCC effort developed by participants from software engineering, psychology, and design. The system provides cognitive communication support via a rich hypermedia display of material from the past that facilitates communication between patients, caretakers, and family members.
"A Language for Human Action" by Gutemberg Guerra-Filho and Yiannis Aloimonos advocates the idea that the space of individual human actions can be seen as having a structure similar to that of language. The authors introduce a human activity language (HAL) that represents visual and motor information via symbols. They address three issues: phonology, where they define "atomic" segments used to compose human activities; morphology, which determines the structure of each action and their organization in an action lexicon; and syntax, which incorporates basic constraints to formulate action sentences. A language for human action provides an intriguing link between human language and human motion, and the authors have used the principles behind it to build a computational framework.
In "An Interactive Multimedia Diary for the Home," Gamhewage C. de Silva, Toshihiko Yamasaki, and Kiyoharu Aizawa present a video retrieval and summarization system for a home environment equipped with a large number of sensors. In this application, the computer is actually part of the environment and the focus is on recording daily human activity, not interaction with any particular computing device. In addition to analyzing the signals obtained from pressure-based sensors mounted in the floor, the system uses audio and video data streams to detect simple events. The ultimate goal is to incorporate multiple channels of information to allow detection and recognition of high-level actions and events such as conversations and to facilitate the creation of interactive multimedia diaries for life-log applications.
Finally, "Holistic Sensing and Active Displays for Intelligent Driver Support Systems" by Mohan M. Trivedi and Shinko Y. Cheng presents an overview of multidisciplinary research on the design and evaluation of computational systems for improving driving safety, focusing on a novel dynamic active display. The authors also discuss the design of novel instrumented vehicles used for conducting realistic driving experiments in which researchers record information about the environment, vehicle, and driver state, both for ethnographic studies and for signal analysis geared toward improving driving safety.
We hope the readers of this special issue find these articles exciting, as we believe they could lead to restarting discussions on human-centered computing.
We thank all of the authors who submitted articles for this special issue. Given the number of submissions, the review process involved two stages. We first preselected 40 submissions based on their overall quality and relevance to the call for papers. Each of these papers was then reviewed by at least two independent reviewers. The acceptance decision involved multiple discussions between the special issue editors and Scott Hamilton, Computer's senior acquisitions editor. We also thank Nahum Gershon for fruitful discussions on this topic.
is scientific manager and a senior researcher at IDIAP Research Institute, Martigny, Switzerland. His research focuses on developing computer vision techniques that use machine learning, involve humans directly, and are rooted in principles, theories, or techniques from cognitive psychology, the arts, and information sciences. Jaimes received a PhD in electrical engineering from Columbia University. Contact him at email@example.com.
is a senior researcher at IDIAP Research Institute. His research focuses on automatic analysis of human activities and social interaction from sensor data, integrating concepts from multimedia signal processing, machine learning, and social psychology. Gatica-Perez received a PhD in electrical engineering from the University of Washington. Contact him at firstname.lastname@example.org.
is a professor in the Department of Computer Science at the University of Amsterdam, The Netherlands. His research interests include human-computer interaction from a computer vision perspective and multimedia information retrieval. Sebe received a PhD in computer science from Leiden University, The Netherlands. Contact him at email@example.com.
Thomas S. Huang
is the William L. Everitt Distinguished Professor of Electrical and Computer Engineering, a research professor at the Coordinated Science Laboratory, and head of the Image Formation and Processing Group at the Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign. His research interests focus on the broad area of information technology, especially the transmission and processing of multidimensional signals. Huang received a PhD in electrical engineering from MIT. He is a member of the National Academy of Engineering, a Foreign Member of the Chinese Academies of Engineering and Sciences, and a Fellow of the International Association of Pattern Recognition, the Optical Society of America, and the IEEE. Contact him at firstname.lastname@example.org.