This paper describes an unified probabilistic framework for appearance based tracking of rigid and non-rigid objects. A spatio-temporal dependent shape/texture Eigenspace and mixture of diagonal gaussians are learned in a Hidden Markov Model(HMM) like structure to better constrain the model and for recognition purposes. Particle filtering is used to track the object while switching between different shape/texture models. This framework allows recognition and temporal segmentation of activities. Additionally an automatic stochastic initialization is proposed, the number of states in the HMM are selected based on the Akaike Information Criterion and comparison with deterministic tracking for 2D models is discussed. Preliminary results of eye-tracking, lip-tracking and temporal segmentation of mouth events are presented.