A method for detecting and tracking human faces in color videos is presented. The method first uses a chroma chart with information about skin colors of various races to determine regions of skin color in the first frame of a video. A new chroma chart is computed for each region, which more precisely represents the color contents of that region. Chroma charts for different regions that are similar are combined, while those that are considerably different are kept separate. Model facial patterns are then used to detect faces within the skin regions. Once a face is detected, the particular pattern and color of the face are used to track the face. Regions where facial patterns are not detected are expected to correspond to exposed parts of the body or of the background and are ignored. The proposed method can track faces with a high degree of accuracy once they are identified.