This Article 
 Bibliographic References 
 Add to: 
Eye Gaze Correction with Stereovision for Video-Teleconferencing
July 2004 (vol. 26 no. 7)
pp. 956-960

Abstract—The lack of eye contact in desktop video teleconferencing substantially reduces the effectiveness of video contents. While expensive and bulky hardware is available on the market to correct eye gaze, researchers have been trying to provide a practical software-based solution to bring video-teleconferencing one step closer to the mass market. This paper presents a novel approach: Based on stereo analysis combined with rich domain knowledge (a personalized face model), we synthesize, using graphics hardware, a virtual video that maintains eye contact. A 3D stereo head tracker with a personalized face model is used to compute initial correspondences across two views. More correspondences are then added through template and feature matching. Finally, all the correspondence information is fused together for view synthesis using view morphing techniques. The combined methods greatly enhance the accuracy and robustness of the synthesized views. Our current system is able to generate an eye-gaze corrected video stream at five frames per second on a commodity 1 GHz PC.

[1] T.J. Cham, S. Krishnamoorthy, and M. Jones, Analogous View Transfer for Gaze Correction in Video Sequences Proc. Int'l Conf. Automation, Robotics, Control and Vision, 2002.
[2] D.H. Douglas and T.K. Peucker, Algorithms for the Reduction of the Number of Points Required to Represent a Digitized Line or Its Caricature Canadian Cartographer, vol. 10, no. 2, pp. 112-122, 1973.
[3] J. Gemmell, C.L. Zitnick, T. Kang, K. Toyama, and S. Seitz, Gaze-Awareness for Videoconferencing: A Software Approach IEEE Multimedia, vol. 7, no. 4, pp. 26-35, Oct. 2000.
[4] M. Jones, Multidimensional Morphable Models: A Framework for Representing and Matching Object Classes Int'l J. Computer Vision, vol. 29, no. 2, pp. 107-131, Aug. 1998.
[5] M. Kass, A. Witkin, and D. Terzopoulos, Snake: Active Contour Models Int'l J. Computer Vision, vol. 1, no. 4, pp. 321-331, 1987.
[6] R. Kollarits, C. Woodworth, J. Ribera, and R. Gitlin, An Eye-Contact Camera/Display System for Videophone Applications Using a Conventional Direct-View LCD SID Digest, 1995.
[7] J. Liu, I. Beldie, and M. Wopking, A Computational Approach to Establish Eye-Contact in Videocommunication Proc. Int'l Workshop Stereoscopic and Three Dimensional Imaging (IWS3DI), pp. 229-234, 1995.
[8] Z. Liu, Z. Zhang, C. Jacobs, and M. Cohen, Rapid Modeling of Animated Faces From Video J. Visualization and Computer Animation, vol. 12, no. 4, pp. 227-240, 2001.
[9] C. Loop and Z. Zhang, “Computing Rectifying Homographies for Stereo Vision,” Proc. Computer Vision and Pattern Recognition '99, vol. I, pp. 125-131, 1999.
[10] L. Mhlbach, B. Kellner, A. Prussog, and G. Romahn, The Importance of Eye Contact in a Videotelephone Service Proc. 11th Int'l Symp. Human Factors in Telecomm., 1985.
[11] M. Ott, J. Lewis, and I. Cox, Teleconferencing Eye Contact Using a Virtual Camera Proc. Conf. Human Factors in Computing Systems, pp. 119-110, 1993.
[12] S. Pollard, J. Porrill, J. Mayhew, and J. Frisby, Disparity Gradient, Lipschitz Continuity, and Computing Binocular Correspondance Robotics Research: Proc. Third Int'l Symp., O.D. Faugeras and G. Giralt, eds., vol. 30, pp. 19-26, 1986.
[13] J. Shi and C. Tomasi, Good Features to Track Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 593-600, 1994.
[14] R. Stokes, "Human Factors and Appearance Design Considerations of the Mod II PicturePhone Station Set," IEEE Trans. Communication Technology, Vol. COM-17, No. 2, April 1969, pp. 318-323.
[15] R. Yang and Z. Zhang, Eye Gaze Correction with Stereovision for Video-Teleconferencing Proc. European Conf. Computer Vision, pp. 479-494, 2002.
[16] R. Yang and Z. Zhang, Model-Based Head Pose Tracking With Stereovision Proc. Fifth IEEE Int'l Conf. Automatic Face and Gesture Recognition, May 2002.
[17] Z. Zhang, A Flexible New Technique for Camera Calibration IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, Nov. 2000.

Index Terms:
Stereoscopic vision, eye-gaze correction, model-based tracking, head tracking, pose determination.
Ruigang Yang, Zhengyou Zhang, "Eye Gaze Correction with Stereovision for Video-Teleconferencing," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 26, no. 7, pp. 956-960, July 2004, doi:10.1109/TPAMI.2004.27
Usage of this product signifies your acceptance of the Terms of Use.