2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Boston, MA, USA
June 7, 2015 to June 12, 2015
Pei-Lun Hsieh , University of Southern California, Los Angeles, United States
Chongyang Ma , University of Southern California, Los Angeles, United States
Jihun Yu , Industrial Light & Magic, 1110 Gorgas Ave, San Francisco, CA 94129, United States
Hao Li , University of Southern California, Los Angeles, United States
We introduce a realtime facial tracking system specifically designed for performance capture in unconstrained settings using a consumer-level RGB-D sensor. Our framework provides uninterrupted 3D facial tracking, even in the presence of extreme occlusions such as those caused by hair, hand-to-face gestures, and wearable accessories. Anyone's face can be instantly tracked and the users can be switched without an extra calibration step. During tracking, we explicitly segment face regions from any occluding parts by detecting outliers in the shape and appearance input using an exponentially smoothed and user-adaptive tracking model as prior. Our face segmentation combines depth and RGB input data and is also robust against illumination changes. To enable continuous and reliable facial feature tracking in the color channels, we synthesize plausible face textures in the occluded regions. Our tracking model is personalized on-the-fly by progressively refining the user's identity, expressions, and texture with reliable samples and temporal filtering. We demonstrate robust and high-fidelity facial tracking on a wide range of subjects with highly incomplete and largely occluded data. Our system works in everyday environments and is fully unobtrusive to the user, impacting consumer AR applications and surveillance.
P. Hsieh, C. Ma, J. Yu and H. Li, "Unconstrained realtime facial performance capture," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 1675-1683.