Visual tracking is one of the fundamental problems in computer vision. The pose of an object extracted through consecutive frames has a variety of applications ranging from robot navigation to camera based man-machine interfaces. In this paper we examine the use of Active Appearance Models (AAM) for the task of visual tracking. The original Active Appearance Model is limited to have all points of the model visible in all frames. We introduce a notion of visibility uncertainty for the points in the AAM, removing the above limitation and therefore allowing the object to contain self-occlusions. The visibility uncertainty is easily integrated into the existing AAM framework, keeping model initialization time to a minimum.We have experiments illustrating that the extension allow AAMs to track through self occlusions at near real-time.