An appearance-based face authentication system integrating 2D color or intensity images and 3D data is presented in this paper. The proposed system is based on a low-cost 3D and color sensor, capable of synchronous real-time acquisition of 3D images and associated color images. Novel algorithms are proposed that exploit depth information and prior knowledge of face geometry and symmetry to achieve robust face detection, localization and authentication under conditions of background clutter, occlusion, face pose alteration and harsh illumination. A method for the enrichment of face databases with synthetically generated views depicting variations in pose and illumination is proposed to cope with head rotations and illumination variations, avoiding a cumbersome enrolment process. The performance of the proposed authentication scheme is tested thoroughly on a face database of 1500 images, recorded with the 3D acquisition system. Experimental results demonstrate significant gains resulting from the combined use of depth and color or intensity information, in comparison to the use of 2D images alone.