A robust method for tracking faces of multiple people moving in a scene using Kalman filter is proposed in this paper. To distinguish faces of people during partial occlusion the proposed method uses the non-parametric cloth distribution. To overcome the problem of total occlusion, faces are tracked using the values generated by Kalman prediction algorithm. The size, top-left coordinate and velocity of motion of the detected face being the parameters of the Kalman vector; the predicted values are used to locate faces in the next frame. The faces are redetected and the templates are updated at discrete time intervals when the similarity measures, between the faces detected and respective face templates, are less than a preset threshold. Skin segmentation based face detection makes the algorithm computationally simple, and updating the face template makes it invariant to pose changes. The proposed method is experimented to be invariant to lightning conditions, change of pose, and works well in the case of partial and total occlusion for a short period.