The Community for Technology Leaders

Music to the Eyes


Pages: pp. 4-5

In college, Aaron Hertzmann double majored in art and computer science, which he always viewed as being distinct fields. "In art classes, I primarily did painting and drawing," he explained. "When I told people my double major, they would ask whether I was planning to go into computer graphics in the future. I'd shake my head and say that programming and art are two completely different things. How wrong I was! Both are, in fact, very cognitive and very intuitive. And one can express aesthetic ideas through code."

He came to nonphotorealistic rendering (NPR) almost by accident. After starting graduate school to work on image-based rendering, he attended Siggraph 1997. He felt that the current work of the time in painterly rendering was a far cry from real painting, so he started experimenting. Working toward an unarticulated aesthetic goal, he implemented some existing algorithms, added extensions, and began to play around. "At some point, I liked the images I was getting, so I wrote it up and sent it in (it was published in the Siggraph 1998 proceedings). After that, I began to think more about how one describes elements of artistic style mathematically and algorithmically."


Hertzmann captures images and video and now uses his own algorithm to process video, creating a result that looks like a hand-painted image. "The algorithm is based on my experience with painting," he explained. "It first places large brush strokes to roughly sketch out the appearance of the images, and then places smaller strokes to refine other parts of the image. The strokes are long and curved, roughly following image contours." Figure 1 is an example of a painted output from his system.


Figure 1   Hallway at the Media Research Lab at New York University.

Originally, Hertzmann worked with painterly rendering of still image captures. Then—along with Ken Perlin—Hertzmann created an animated version of the system, which can process animation and live video by continually redrawing the output painting, adding new brush strokes wherever the video changes. Users can interact with the system, the output of which is projected onto a large canvas. The result is similar to paint-on-glass animation. "I feel like the painting algorithm produces images as a result of my personal aesthetic for loose and expressive painting styles. The interactive version of the system seems to make great, original paintings very regularly. On the other hand, when other people have implemented my algorithm from my papers, I can always recognize it, even thought the results look slightly different in each implementation."


One of Hertzmann's own implementations of his live painterly rendering algorithms took place in 1999. The now defunct Siam records hired him to apply his painterly rendering system to live music. Erik Friendlander's avant-jazz-funk quartet Topaz—consisting of sax, drums, electric bass, and cello—performed live in a studio and Hertzmann's software captured the video and processed and rendered it. Using several methods such as crossfades and dissolves, he chose painting styles for visual appeal to enhance the constantly changing moods of the music. This issue's cover image is a still from this project. Here we imagistically see Friendlander playing cello with his drummer behind him on the right. "I was quite surprised by how well things came out," Hertzmann said. "The software turned out what was, in my opinion, quite lovely effects at various points in the film."

Using a dissolve effect, at the beginning of one particular song, Hertzmann animated a fade from complete black into a close up of the cello bow playing the strings as if a painter were carefully applying just enough strokes over time to gradually animate the cello. Figures 2, 3, and 4 are stills from this sequence.


Figure 2   Main horizontal line is a cello's bow and the thin vertical lines are the strings, as the whole image gradually fades from black to abstraction to a more recognizable image a few seconds later.


Figure 3   Drummer Satoshi Takeishi from Topaz.


Figure 4   Expressionistic painting style of cellist Erik Friedlander viewed from the side, during an energetic passage.

All of Hertzmann's stills are in video resolution, not print resolution, so for the cover image, he took a still from the original video and reprocessed it at a higher resolution. In other words, he made a painting of a painting. "Perhaps not the most elegant super-resolution algorithm," he said, "but it seems to work well enough."


These days, in NPR, Hertzmann is working on extensions to the painterly rendering algorithms to make them more general and more accessible to artists. He's also excited about working in 3D shape modeling and video motion capture of 3D nonrigid models from video sequences. "I'm pretty excited about recent results we've gotten for 3D scanning of objects with arbitrary reflectances." It doesn't stop there. Hertzmann is also applying machine-learning techniques to certain problems in computer graphics, especially animation. "My goal is to convince graphics researchers to make use of learning techniques, especially probabilistic modeling techniques, in graphic applications."

Regarding the immediate future, he plans to stay a professor of computer science (at the University of Toronto), although he really doesn't consider himself a scientist. "I guess I consider myself someone who builds algorithms, and also an artist with a lowercase 'a.' I think everyone can be an artist, although not everyone is paid for it."

This issue includes a tutorial by Hertzmann on stroke-based rendering.

60 ms
(Ver 3.x)