This Article 
 Bibliographic References 
 Add to: 
Geometry-Driven Photorealistic Facial Expression Synthesis
January/February 2006 (vol. 12 no. 1)
pp. 48-60

Abstract—Expression mapping (also called performance driven animation) has been a popular method for generating facial animations. A shortcoming of this method is that it does not generate expression details such as the wrinkles due to skin deformations. In this paper, we provide a solution to this problem. We have developed a geometry-driven facial expression synthesis system. Given feature point positions (the geometry) of a facial expression, our system automatically synthesizes a corresponding expression image that includes photorealistic and natural looking expression details. Due to the difficulty of point tracking, the number of feature points required by the synthesis system is, in general, more than what is directly available from a performance sequence. We have developed a technique to infer the missing feature point motions from the tracked subset by using an example-based approach. Another application of our system is expression editing where the user drags feature points while the system interactively generates facial expressions with skin deformation details.

[1] D. Ballard and C. Brown, Computer Vision. Englewood Cliffs, N.J.: Prentice-Hall, 1982.
[2] T. Beier and S. Neely, “Feature-Based Image Metamorphosis,” Computer Graphics, pp. 35-42, July 1992.
[3] V. Blanz, C. Basso, T. Poggio, and T. Vetter, “Reanimating Faces in Images and Video,” Proc. Eurographics Conf., 2003.
[4] M. Brand, “Voice Puppetry,” Computer Graphics, Proc. Ann. Conf. Series, pp. 22-28, Aug. 1999.
[5] C. Bregler, M. Covell, and M. Slaney, “Video Rewrite: Driving Visual Speech with Audio,” Computer Graphics, pp. 353-360, Aug. 1997.
[6] S.E. Brennan, “Caricature Generator,” MS Visual Studies, Dept. of Architecture, Massachusetts Inst. of Technology, Cambridge, Mass., Sept. 1982.
[7] T.F. Cootes and C.J. Taylor, “Statistical Models of Appearance for Computer Vision,” , 2001.
[8] Y. Du and X. Lin, “Realistic Mouth Synthesis Based on Shape Appearance Dependence Mapping,” Pattern Recognition Letters, vol. 23, no. 14, pp. 1875-1885, 2002.
[9] T. Ezzat, G. Geiger, and T. Poggio, “Trainable Videorealistic Speech Animation,” Computer Graphics, Proc. Ann. Conf. Series, pp. 388-398, Aug. 2002.
[10] O. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint. MIT Press, 1993.
[11] B. Guenter, C. Grimm, D. Wood, H. Malvar, and F. Pighin, “Making Faces,” Computer Graphics, Proc. Ann. Conf. Series, pp. 55-66, July 1998.
[12] P. Joshi, W.C. Tien, M. Desbrun, and F. Pighin, “Learning Controls for Blend Shape Based Realistic Facial Animation,” Proc. Symp. Computer Animation (SCA '03), pp. 187-192, July 2003.
[13] Y. Lee, D. Terzopoulos, and K. Waters, “Realistic Modeling for Facial Animation,” Computer Graphics, pp. 55-62, Aug. 1995.
[14] S.Z. Li and L. Gu, “Real-Time Multi-View Face Detection, Tracking, Pose Estimation, Alignment, and Recognition,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, Demo Summary, 2001.
[15] P. Litwinowicz and L. Williams, “Animating Images with Drawings,” Computer Graphics, pp. 235-242, Aug. 1990.
[16] Z. Liu, Y. Shan, and Z. Zhang, “Expressive Expression Mapping with Ratio Images,” Computer Graphics, Proc. Ann. Conf. Series, pp. 271-276, Aug. 2001.
[17] D.G. Luenberger, Linear and Nonlinear Programming. Addison-Wesley, 1984.
[18] N. Magneneat-Thalmann, N.E. Primeau, and D. Thalmann, “Abstract Muscle Actions Procedures for Human Face Animation,” The Visual Computer, vol. 3, no. 5, pp. 290-297, 1988.
[19] J.-Y. Noh and U. Neumann, “Expression Cloning,” Computer Graphics, Proc. Ann. Conf. Series, pp. 277-288, Aug. 2001.
[20] F.I. Parke, “Computer Generated Animation of Faces,” Proc. ACM Ann. Conf., Aug. 1972.
[21] F.I. Parke and K. Waters, Computer Facial Animation. Wellesley, Mass.: AK Peters, 1996.
[22] K. Perlin and A. Goldberg, “Improv: A System for Scripting Interactive Actors in Virtual Worlds,” Computer Graphics, Proc. Ann. Conf. Series, pp. 205-216, Aug. 1996.
[23] F. Pighin, J. Hecker, D. Lischinski, R. Szeliski, and D.H. Salesin, “Synthesizing Realistic Facial Expressions from Photographs,” Computer Graphics, Proc. Ann. Conf. Series, pp. 75-84, July 1998.
[24] F. Pighin, R. Szeliski, and D.H. Salesin, “Resynthesizing Facial Animation through 3D Model-Based Tracking,” Proc. Int'l Conf. Computer Vision (ICCV '99), 1999.
[25] S. Platt and N. Badler, “Animating Facial Expression,” Computer Graphics, vol. 15, no. 3, pp. 245-252, 1981.
[26] H. Pyun, Y. Kim, W. Chae, H.W. Kang, and S.Y. Shin, “An Example-Based Approach for Facial Expression Cloning,” Proc. Symp. Computer Animation (SCA '03), pp. 167-176, July 2003.
[27] S.M. Seitz and C.R. Dyer, “View Morphing,” Computer Graphics, pp. 21-30, Aug. 1996.
[28] D. Terzopoulos and K. Waters, “Analysis and Synthesis of Facial Image Sequences Using Physical and Anatomical Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 569-579, June 1993.
[29] S. Toelg and T. Poggio, “Towards an Example-Based Image Compression Architecture for Video-Conferencing,” Technical Report 1494, MIT, 1994.
[30] P.-H. Tu, I.-C. Lin, J.-S. Yeh, R.-H. Liang, and M. Ouhyoung, “Expression Detail Mapping for Realistic Facial Animation,” Proc. Eighth Int'l Conf. CAD/Graphics, pp. 20-25, Oct. 2003.
[31] K. Waters, “A Muscle Model for Animating Three-Dimensional Facial Expression,” Computer Graphics, vol. 22, no. 4, pp. 17-24, 1987.
[32] L. Williams, “Performance-Driven Facial Animation,” Computer Graphics, pp. 235-242, Aug. 1990.
[33] Y. Ye, Interior Point Algorithms: Theory and Analysis. John Wiley, 1997.
[34] Q. Zhang, Z. Liu, B. Guo, and H. Shum, “Geometry-Driven Photorealistic Facial Expression Synthesis,” Proc. Symp. Computer Animation (SCA '03), pp. 177-186, July 2003.
[35] Z. Zhang, “Flexible Camera Calibration by Viewing a Plane from Unknown Orientations,” Proc. Int'l Conf. Computer Vision (ICCV '99), pp. 666-673, 1999.

Index Terms:
Facial animation, expression mapping, expression details, facial expressions, performance-driven animation.
"Geometry-Driven Photorealistic Facial Expression Synthesis," IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 1, pp. 48-60, Jan.-Feb. 2006, doi:10.1109/TVCG.2006.9
Usage of this product signifies your acceptance of the Terms of Use.