The Community for Technology Leaders

Smart Depiction for Visual Communication

Maneesh Agrawala, Stanford University
Frédo Durand, Massachusetts Institute of Technology, Computer Science and Artificial Intelligence Laboratory

Pages: pp. 20-21

As computers become faster, smaller, ubiquitous, and interconnected, the rate at which we generate new data is growing exponentially. Sophisticated computational simulation techniques have made it fast and easy to create large volumes of data by modeling all kinds of physical phenomena. Meanwhile, digital sensors and sensor networks have simplified the process of collecting data from the real world. Moreover, the Internet has connected us to one another in a way that further facilitates creation and dissemination of information. In fact, Lyman and Varian estimate that new stored information grew at a rate of 30 percent a year from 1999 to 2002 and that we produced 5 exabytes (equivalent to about 37,000 new libraries the size of the Library of Congress) of completely new information in the year 2002 alone. 1

While computers are helping us produce this vast amount of data, they also play an integral role in helping us make sense of it. Visual displays such as images, diagrams, sketches, video, film, and animations are increasingly generated, manipulated, and transmitted by computers. However, even with the aid of a computer, producing effective visual content can take hours or days and consume considerable human effort and skill. The challenge is to develop new systems and user interfaces that facilitate visual communication by making it fast and easy to generate compelling visual content.

The most effective graphics combine principles from graphic design, visual art, perceptual psychology, and cognitive science. Effective depictions take a particular perspective: They omit irrelevant information, and they emphasize important information, often distorting it in the service of communication. When well designed, such displays capitalize on humans' facility for processing visual information and thereby improve comprehension, memory, inference, and decision making.

A design principle explains how a visual technique can be used to emphasize (or de-emphasize) a particular type of information conveyed by the display. For example, in the domain of route maps, mapmakers exaggerate the lengths of short roads so that all the turning points in the routes will be visible. 2 In this case, lengthening short roads is the visual technique and the turning points are the information conveyed by the technique. Smart depiction systems are computer algorithms and interfaces that embody these kinds of design principles. Such systems hold the potential for significantly reducing the time and effort required to generate rich and effective visual content.

This special issue of IEEE Computer Graphics and Applications highlights recent advances in smart depiction systems. It brings together four articles that present design principles and depiction systems for a diverse set of domains including color reproduction, medical illustration, text information display, and visual storytelling.

The first article, "Detail Preserving Reproduction of Color Images for Monochromats and Dichromats" by Rasche, Geist, and Westall, describes the problem of reducing color images into a gray-scale representation for printing on a gray-scale printer. They explain that standard conversion techniques for color to gray scale will map pixels that have similar luminance but widely different chrominance to similar shades of gray and often end up obscuring or de-emphasizing important information. The authors present a new approach for gray-scale conversion based on the principle that perceived differences between pairs of colors should be proportional to their perceived gray differences. They show how this principle preserves important detail in the gray-scale images. They also describe how the technique can be used to reduce three-channel color images into two-channel dichromatic images that preserve important detail for visually impaired individuals.

The next article, "Illustration Motifs for Effective Medical Volume Illustration" by Svakhine, Ebert, and Stredney, examines the problem of creating medical illustrations from volume data. Their system provides an interface that lets users quickly specify which features of the volume data they are most interested in. The complexity of the interface is designed to adapt to the user so that novice users are given a simpler interface than experts. The authors then show how to highlight the important information by applying a variety of illustration styles or motifs.

"Depicting Dynamics Using Principles of Visual Art and Narrations" by Nienhaus and Döllner considers the use of symbolic glyphs to illustrate dynamic motions in both static images. They draw on principles from sequential art or comic book storytelling, storyboarding techniques, and short design for film. Given a scene and a behavior description, their system analyzes the scene dynamics and automatically augments the scene representation with glyphs that emphasize the motions. Nienhaus and Döllner show how the glyphs can be used to symbolize past, ongoing, and future activities in a single image. Moreover, the glyphs can communicate nonvisual information such as tension, danger, and other emotions.

The final article, "Visualizing Live Text Streams Using Motion and Temporal Pooling" by Albrecht-Buehler, Watson, and Shamma, presents a list of design principles and guidelines specifically targeted at using motion to convey information. Their principles are based on human perception of motion. They then address the problem of visualizing streams of textual information (as available from newswires, blogs, closed-captioning services, and so on). In particular, their TextPool system uses motion to convey when data has changed, which elements of the text are most important, as well as correlations and similarity in the data.

Smart depiction and visual communication raise a wealth of exciting issues for future work, and interdisciplinary approaches are a key to success. Cognitive and perceptual sciences are essential for understanding how our visual system and brain process visual information. Such understanding of mental processes is crucial in designing depiction algorithms that facilitate comprehension. The visual arts and design have developed a plethora of depiction techniques and principles for effective communication, and analyzing them through the insights of cognitive and perceptual sciences will allow for better formalization and translation in a form that leads naturally to implementation in computer graphics techniques. Computer graphics has a successful history of fruitful synergies with scientific and artistic fields, and we are excited to see that smart depiction research perpetuates this tradition in the service of effective visual communication.


About the Authors

Bio Graphic
Maneesh Agrawala is a researcher in the Document Processing and Understanding Group at Microsoft Research. His primary research interests are in visualization, human–computer interaction, and computer graphics. His current focus is on investigating how cognitive design principles can be used to improve the effectiveness of visualizations. Agrawala has a BS in mathematics and a PhD in computer science, both from Stanford University. At Stanford, he developed the LineDrive system for automatically designing route maps. (LineDrive remains publicly accessible at Contact him at
Bio Graphic
Frédo Durand is an assistant professor in the Electrical Engineering and Computer Science Department at the Massachusetts Institute of Technology, and a member of the Computer Science and Artificial Intelligence Laboratory. His research interests span most aspects of picture generation and creation. This includes realistic graphics, real-time rendering, nonphotorealistic rendering, as well as computational photography. Durand has a PhD from Grenoble University, France, where he worked with Claude Puech and George Drettakis on both theoretical and practical aspects of 3D visibility. He received a Eurographics Young Researcher Award in 2004 and a National Science Foundation CAREER award in 2005. Contact him at
68 ms
(Ver 3.x)