This Article 
 Bibliographic References 
 Add to: 
AniPaint: Interactive Painterly Animation from Video
March 2012 (vol. 18 no. 3)
pp. 475-487
Peter O'Donovan, University of Toronto, Toronto
Aaron Hertzmann, University of Toronto, Toronto
This paper presents an interactive system for creating painterly animation from video sequences. Previous approaches to painterly animation typically emphasize either purely automatic stroke synthesis or purely manual stroke keyframing. Our system supports a spectrum of interaction between these two approaches which allows the user more direct control over stroke synthesis. We introduce an approach for controlling the results of painterly animation: keyframed Control Strokes can affect automatic stroke's placement, orientation, movement, and color. Furthermore, we introduce a new automatic synthesis algorithm that traces strokes through a video sequence in a greedy manner, but, instead of a vector field, uses an objective function to guide placement. This allows the method to capture fine details, respect region boundaries, and achieve greater temporal coherence than previous methods. All editing is performed with a WYSIWYG interface where the user can directly refine the animation. We demonstrate a variety of examples using both automatic and user-guided results, with a variety of styles and source videos.

[1] P. Haeberli, “Paint by Numbers: Abstract Image Representations,” Proc. ACM SIGGRAPH, pp. 207-214, 1990.
[2] B.J. Meier, “Painterly Rendering for Animation,” Proc. ACM SIGGRAPH, pp. 477-484, 1996.
[3] P. Litwinowicz, “Processing Images and Video for an Impressionist Effect,” Proc. ACM SIGGRAPH, pp. 407-414, 1997.
[4] A. Hertzmann, “Painterly Rendering with Curved Brush Strokes of Multiple Sizes,” Proc. ACM SIGGRAPH, pp. 453-460, 1998.
[5] A. Hertzmann and K. Perlin, “Painterly Rendering for Video and Interaction,” Proc. First Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR), pp. 7-12, 2000.
[6] J. Hays and I. Essa, “Image and Video Based Painterly Animation,” Proc. Third Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR), pp. 113-120, 2004.
[7] A. Hertzmann, “Paint by Relaxation,” Proc. Int'l Conf. Computer Graphics (CGI), pp. 47-54, 2001.
[8] J. Collomosse and P. Hall, “Genetic Paint: A Search for Salient Paintings,” Proc. EvoMUSART, pp. 437-447, 2005.
[9] B. Sabiston, “Waking Life: Making of,” Dvd Featurette, 2001.
[10] A. Agarwala, A. Hertzmann, D.H. Salesin, and S.M. Seitz, “Keyframe-Based Tracking for Rotoscoping and Animation,” Proc. ACM SIGGRAPH, pp. 584-591, 2004.
[11] C. Liu, W. Freeman, E. Adelson, and Y. Weiss, “Human-Assisted Motion Annotation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008.
[12] R.D. Kalnins, L. Markosian, B.J. Meier, M.A. Kowalski, J.C. Lee, P.L. Davidson, M. Webb, J.F. Hughes, and A. Finkelstein, “WYSIWYG NPR: Drawing Strokes Directly on 3D Models,” Proc. ACM SIGGRAPH, pp. 755-762, 2002.
[13] K. Odermatt and C. Springfield, “Creating 3D Painterly Environments for Disney's ‘Treasure Planet’,” Proc. ACM SIGGRAPH, pp. 160-160, 2002.
[14] A. Bousseau, M. Kaplan, J. Thollot, and F.X. Sillion, “Interactive Watercolor Rendering with Temporal Coherence and Abstraction,” Proc. Fourth Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR), pp. 141-149, 2006.
[15] A. Bousseau, F. Neyret, J. Thollot, and D. Salesin, “Video Watercolorization Using Bidirectional Texture Advection,” ACM Trans. Graphics, vol. 26, no. 3, pp. 104:1-104:7, 2007.
[16] J.P. Collomosse, D. Rowntree, and P.M. Hall, “Stroke Surfaces: Temporally Coherent Artistic Animations from Video,” IEEE Trans. Visualization and Computer Graphics, vol. 11, no. 5, pp. 540-549, Sept./Oct. 2005.
[17] D. DeCarlo and A. Santella, “Stylization and Abstraction of Photographs,” Proc. ACM SIGGRAPH, pp. 769-776, 2002.
[18] B. Gooch, G. Coombe, and P. Shirley, “Artistic Vision: Painterly Rendering Using Computer Vision Techniques,” Proc. Second Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR), pp. 83-90, 2002.
[19] A. Kolliopoulos, J.M. Wang, and A. Hertzmann, “Segmentation-Based 3D Artistic Rendering,” Proc. Eurographics Symp. Rendering (EGSR), pp. 361-370, 2006.
[20] J. Wang, Y. Xu, H.-Y. Shum, and M.F. Cohen, “Video Tooning,” Proc. ACM SIGGRAPH, pp. 574-583, 2004.
[21] K. Zeng, M. Zhao, C. Xiong, and S. Zhu, “From Image Parsing to Painterly Rendering,” ACM Trans. Graphics, vol. 29, no. 1, pp. 1-11, 2009.
[22] L. Lin, K. Zeng, H. Lv, Y. Wang, Y. Xu, and S.-C. Zhu, “Painterly Animation with Video Content Extraction,” Proc. Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR), 2010.
[23] M. Kagaya, W. Brendel, Q. Deng, T. Kesterson, S. Todorovic, P. Neill, and E. Zhang, “Video Painting with Space-Time-Varying Style Parameters,” IEEE Trans.Visualization and Computer Graphics, vol. 17, no. 1, pp. 74-87, Jan. 2011.
[24] X. Bai, J. Wang, D. Simons, and G. Sapiro, “Video SnapCut: Robust Video Object Cutout Using Localized Classifiers,” Proc. ACM SIGGRAPH, vol. 28, no. 3, 2009.
[25] W. Brendel and S. Todorovic, “Video Object Segmentation by Tracking Regions,” Proc. IEEE 12th Int'l Conf. Computer Vision (ICCV), 2009.
[26] T. Fleisch, F. Rechel, P. Santos, and A. Stork, “Constraint Stroke-Based Oversketching for 3D Curves,” Proc. Workshop Sketch-Based Interfaces and Modeling (SBIM), 2004.
[27] A. Hertzmann, “Fast Paint Texture,” Proc. Second Int'l Symp Non-Photorealistic Animation and Rendering (NPAR), pp. 91-ff, 2002.
[28] J. Canny, “A Computational Approach to Edge Detection,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 679-698, Nov. 1986.
[29] H. Kang, S. Lee, and C.K. Chui, “Coherent Line Drawing,” Proc. Fifth Int'l Symp. Non-Photorealistic Animation and Rendering (NPAR), pp. 43-50, 2007.
[30] M.P. Salisbury, M.T. Wong, J.F. Hughes, and D.H. Salesin, “Orientable Textures for Image-Based Pen-and-Ink Illustration,” Proc. ACM SIGGRAPH, pp. 401-406, 1997.
[31] A. Hausner, “Simulating Decorative Mosaics,” Proc. ACM SIGGRAPH, pp. 573-580, 2001.
[32] M. Kass and A. Witkin, “Analyzing Oriented Patterns,” J. Computer Vision, Graphics, and Image Processing, vol. 37, no. 3, pp. 362-385, 1987.

Index Terms:
Nonphotorealistic rendering, painterly animation, interactive video processing.
Peter O'Donovan, Aaron Hertzmann, "AniPaint: Interactive Painterly Animation from Video," IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 3, pp. 475-487, March 2012, doi:10.1109/TVCG.2011.51
Usage of this product signifies your acceptance of the Terms of Use.