Andy Witkin: From Computer Vision to Computer Graphics

Tony De , Pixar

Pages: pp. 102-105



On 12 September 2010, the computer graphics community suffered a tremendous loss when Andy Witkin died suddenly while scuba diving off the coast of Monterey, California, at the age of 58. Teacher, mentor, friend, and colleague to many, his impact on the field has been both wide and deep. He will be sorely missed.

Andy began his career in computer vision, then became interested in problems on the boundary between computer vision and computer graphics, and later switched to working on problems firmly in the computer graphics mainstream. In 2001, ACM Siggraph presented Andy with the Computer Graphics Achievement Award for "his pioneering work in bringing a physics based approach to computer graphics." The award committee wrote,

His papers on active contours (snakes) and deformable models, variational modeling, scale-space filtering, space time constraints, and dynamic simulation are considered landmarks that have been inspirational to others and have shaped the field in such different areas as image analysis, surface modeling, and animation.

In 2006, the Academy of Motion Picture Arts and Sciences honored him, David Baraff, and Michael Kass with an Academy Award for "pioneering work in physically based computer-generated techniques used to simulate realistic cloth in motion pictures." As a senior scientist at Pixar, he contributed to all the Pixar feature films from Monsters, Inc. to the as-yet unreleased Brave. Unfortunately, the available space is insufficient to do justice to Andy's contributions here. For a more thorough treatment, see

Early Career

Andy was born to a family of academic heavyweights. His mother, Evelyn Witkin, won the 2002 National Medal of Science for her work on DNA mutagenesis and repair. His father, Herman A. Witkin, was a prominent research psychologist known for his work on perceptual styles. To some degree, Andy began his academic career by following in his father's footsteps. After receiving his BA from Columbia University, Andy decided to apply for graduate school in the MIT Psychology Department. The deadline had just passed, so Andy called to see whether MIT would still consider his application. "Are you related to Herman A. Witkin?" the voice from MIT asked. "Yes, he's my father," Andy replied. There was an awkward pause. "We'll try not to hold that against you." Professional rivalries notwithstanding, Andy was accepted to the PhD program and worked on problems in visual perception under advisor Whitman Richards.

Andy's thesis concerned the perception of surface orientation due to foreshortening effects. If a surface contains markings whose orientations are approximately uniformly distributed on the surface, the distribution in the image plane generally won't be uniform. Andy showed how to use this nonuniformity to estimate surface orientation.


After graduating from MIT, Andy took a research job at SRI but soon left to help start the Fairchild Laboratory for Artificial Intelligence Research. The lab, which was under Marty Tenenbaum's direction, was initially part of Fairchild, a Schlumberger subsidiary. Later, it took on a broader research role at Schlumberger and became called Schlumberger Palo Alto Research. At Schlumberger, Andy managed a computer vision and graphics research group that eventually included Kurt Fleischer, Demetri Terzopoulos, and Michael Kass.

Andy's first broadly influential research contribution was his seminal paper on scale space. 1 Much previous research showed the value of analyzing features of different spatial scales for a variety of perceptual tasks. Andy's insight was to consider scale as a continuous quantity and analyze the way features come and go as the scale changes. Andy believed that features that were persistent across scale would constitute a robust description for tasks such as matching or classification. His insight lives on with SIFT (scale-invariant feature transform) 2 and related methods, which have become popular for object recognition and feature matching.

At Schlumberger, Andy became interested in computer graphics. Today, it's widely recognized that computer vision and computer graphics are closely related. In computer graphics, we describe a 3D world to a computer and ask it to make a 2D picture. In computer vision, we start with a 2D image and try to infer a 3D world. In the late 1980's, however, the two fields were almost entirely distinct. Few collaborations spanned the boundary, and practitioners of the two fields tended to view each other with suspicion if not disdain. Andy was one of the first researchers to try to combine the two disciplines and bridge the cultural divide.

Broadly speaking, computer vision aims to look at the world and create useful representations of what's there. Exactly what those representations should be, however, is difficult to describe. It's hard to evaluate a visual representation without reference to an application, but back in the late '80s, the obvious applications were either far too difficult or of extremely limited interest.

To make progress in this difficult situation, Andy developed a research agenda he called seize and torture. The idea was to first analyze and extract (seize) some model from real-world images, using computer-vision techniques. To test whether the analysis was successful, he sought to modify (torture) the model and synthesize imagery to illustrate what things would look like under the modified circumstances.

The seize-and-torture approach by itself did little to illuminate the nature of the models you might want to extract from real-world imagery. One of Andy's early influences in this regard was D'Arcy Thompson's work exploring the relationship between biological forms and the physical processes that give rise to them. Andy believed that a similar approach to computer vision could be fruitful. His idea was to try to describe aspects of the visual world in terms of the physical processes that could have created them.

In 1987, Andy and Michael Kass applied this point of view to visual patterns with a dominant local orientation. 3 By measuring visual-orientation fields and integrating them into coordinate systems, they could describe patterns in terms of deformation and flow. They explored these possibilities in Knot Reel, 4 an animation they created with Kurt Fleischer. The whimsical exploration of knots being hammered into wood and wood turning into fluid, flowing around knots, earned the 1987 Parigraph Grand Prix for Animation.

From the traditional computer vision viewpoint, any human intervention in image analysis was anathema. How could you claim to invent true machine vision if it required a human? But from the seize-and-torture viewpoint, it was legitimate. Whereas most of the computer vision field was wedded to the idea of image analysis for autonomous vision systems, Andy became interested in computer vision as an aid to a human doing some sort of synthesis. In effect, computer graphics had become the customer of the computer vision analysis. And the computer graphics community isn't shy about introducing whatever human-guided intervention is required to achieve a desired effect. From this perspective, it made a lot of sense to use computer vision techniques as "power steering" to help human operators extract information from images.

The first example of the power-steering approach was snakes, an active-contour model that Andy developed with Kass and Demetri Terzopoulos. 5 Snakes are contours that interactively minimize an energy function that tries to make them as smooth as possible while also attracting them to edges or other features in images. The human operator only needs to put the snake near a desired edge; the snake's energy minimization will do the fine-tuning. If the snake becomes attracted to an edge not intended by the operator, the operator can pull it to the intended edge through an interface. The snakes paper was runner-up for the Marr Prize at the 1987 International Conference on Computer Vision. According to CiteSeer, as of 14 September 2010, it was the 11th-most-cited article in all of computer science. A further example of the power-steering approach was Andy's use of symmetry-seeking virtual materials to create 3D models from silhouettes. 6 The paper describing this work won a Best Paper Award at the Sixth National Conference on Artificial Intelligence in 1987.

By late 1987, Andy's research interests had moved almost entirely from computer vision to computer graphics, and he began collaborating with Al Barr and his students at Caltech. Andy felt that traditional models used in computer graphics were impoverished, consisting of disembodied geometry unaware of weight, contact, and other key physical constraints. He hoped to change the experience of creating computer graphics by enriching these models.

In 1987, John Lasseter wrote a Siggraph paper describing how a skilled animator could apply traditional cartoon techniques such as anticipation, follow-through, and squash and stretch to computer animation. 7 Andy and Michael Kass noted that many aspects of these techniques have a firm basis in the physics of movement. They wondered whether these techniques could emerge naturally from a suitably rich physical model of movement. In 1988, they developed spacetime constraints8 to create motion that's as physically realistic as possible while following an animator's directions.

As the name implies, the technique begins with a set of constraints. For example, in Figure 1, the Luxo lamp is constrained to begin and end at specified points. Subject to these constraints, the method minimizes deviations from Newtonian physics and tries to minimize the average muscle power that the lamp exerts. The result is the jump in Figure 1a. The lamp anticipates the jump by squashing, stretches during the jump, and finally squashes again as it follows through during its landing. All this behavior emerged from optimization of the physical model and wasn't specified by an animator.


Figure 1   Spacetime constraints. (a) The method generates the jumping lamp by minimizing deviations from Newtonian physics while trying to minimize the average muscle power that the lamp exerts. (b) Increasing the lamp base's mass makes it look heavier, without additional animator intervention. (Source: Michael Kass; used with permission.)

John Lasseter pointed out in his paper that an animator can make parts of a character look heavy by making them lag behind, and provided an example of a Luxo lamp with a heavy base. Andy and Michael increased the base's mass in their optimization to get the modified jump in Figure 1b. The base lags behind as expected, but without any special intervention by an animator.

Carnegie Mellon

In 1988, the Schlumberger Laboratory changed its focus and moved to Austin, Texas. Andy then took a tenured faculty position at Carnegie Mellon University, where he taught for 10 years. While at Carnegie Mellon, Andy wrote a series of influential papers with his students on a variety of topics in physically based modeling and animation. Several of Andy's students have gone on to notable academic careers in graphics. In 2004, Andy's PhD student Zoran Popovic received Siggraph's Significant New Researcher Award.

At Carnegie Mellon, Andy continued collaborating with Michael Kass. In 1991, they developed a method of creating organic-looking textures by simulating chemical reactions, 9 which Alan Turing had proposed as a mechanism to explain animal colorations such as zebra stripes. Their work (see Figure 2) received the Golden Nica for Computer Graphics at the Prix Ars Electronica 1991.


Figure 2   Reaction diffusion. This method creates organic-looking textures by simulating chemical reactions. (Source: Michael Kass; used with permission.)

At Carnegie Mellon, Andy became interested in the problem of animating cloth and began a collaboration with David Baraff. 10 Their work has been extremely influential in the field of cloth simulation. It formed the basis of the commercial product Maya Cloth, which has been used in a variety of feature films, including Stuart Little, Lord of the Rings: The Return of the King, and Spiderman 2.


In 1998, Andy and David left Carnegie Mellon to join Pixar. Michael Kass had developed a different cloth simulator used for the Pixar short film Geri's Game. With Andy and David's arrival, all three combined their best ideas to create a new clothing simulator, used in Monsters, Inc. and subsequent films.

A key challenge in making a clothing simulator robust enough to be practical in a production environment was dealing with interpenetrations. Figure 3a shows a character wearing clothing that looks reasonable from the camera's viewpoint. However, the cut-out view in Figure 3b reveals a collection of self-intersections of the character, particularly in the calf and thigh area and in the hip. This situation is physically impossible, and the equations of traditional physics behave extremely poorly under these circumstances.


Figure 3   Self intersections. (a) The clothing on this character looks reasonable from the camera's viewpoint. (b) However, the cut-out reveals a collection of self-intersections of the character, which are physically impossible. (Source: Pixar; used with permission.)

You might think that the animation could be modified to prevent this sort of occurrence, but adjusting the animation is prohibitively costly and excessively restricting from an artistic viewpoint. Animators are often rightly proud of character poses that look good from the camera's viewpoint but aren't physically possible. Andy, David, and Michael developed several techniques 11 for dealing with physically impossible character motion that were vital to making clothing simulation practical, leading to their 2006 academy award.

In the following years at Pixar, Andy led the Studio Tools character articulation group, where he developed new articulation methods for The Incredibles and Ratatouille. Most recently, Andy was engaged in the development of advanced hair simulation technology for the upcoming Pixar/Disney feature film Brave and pursuing basic research in using data-mining techniques to define animation style.

Andy made enormous contributions to two fields, computer vision and computer graphics, and in doing so was a major force in cementing the two together. But that's only part of his professional legacy. He was also a mentor and friend to several generations of researchers, many of whom have gone on to have a major impact in their own right. Additionally, he inspired many other practitioners and researchers who never had the opportunity to work closely with him but were nonetheless drawn to his remarkable ability to consistently attack hard and important problems, solve them in new and surprising ways, and communicate those solutions with terrific clarity and simplicity. Finally, his contributions at Pixar in modeling, rigging, and simulation have left their mark on every Pixar film since Monsters, Inc., and audiences will continue to enjoy them long into the future.

In the words of Kurt Fleischer, one of Andy's long-time collaborators, "Andy had a unique way of looking at the world, which he exhibited both in his quirky sense of humor and his brilliant solutions to technical problems. He was one of a kind." He was indeed.


67 ms
(Ver 3.x)