In this paper, we propose a novel framework for face alignment based on the Active Appearance Model (AAM) in surveillance systems with Pan-Tilt-Zoom (PTZ) cameras. The AAM converges poorly in face images which are affected by illumination factors, cluttered backgrounds and status of the camera. To search for robust face model parameters, we propose a robust AAM fitting method based on segmenting faces and combining Person-specific and Generic models to achieve accurate face alignment. We segment faces using histogram back-projection and a skin color histogram, which is updated using a skin mask extracted by the AAM. For robust face recognition, we combined Generic and Person-specific models with a slight reduction in processing time. The extracted AAM parameters are as accurate as those when using the Person-specific model and can be used as features for face recognition. Empirical experiments show that our proposed method extracts very accurate face parameters and is not sensitive to initial shapes.