This paper presents a novel neural network based technique for face detection that eliminates limitations pertaining to the skin color variations among people. We propose to model the skin color in the three dimensional RGB space which is a color cube consisting of all the possible color combinations. Skin samples in images with varying lighting conditions, from the Old Dominion University skin database, are used for obtaining a skin color distribution. The primary color components of each plane of the color cube are fed to a three-layered network, trained using the backpropagation algorithm with the skin samples, to extract the skin regions from the planes and interpolate them so as to provide an optimum decision boundary and hence the positive skin samples for the skin classifier. The use of the color cube eliminates the difficulties of finding the non-skin part of training samples since the interpolated data is consider skin and rest of the color cube is consider non-skin. Subsequent face detection is aided by the color, geometry and motion information analyses of each frame in a video sequence. The performance of the new face detection technique has been tested with real-time data of size 320?240 frames from video sequences captured by a surveillance camera. It is observed that the network can differentiate skin and non-skin effectively while minimizing false detections to a large extent when compared with the existing techniques. In addition, it is seen that the network is capable of performing face detection in complex lighting and background environments.