Reliable facial expression recognition by machine is still a challenging task. We propose a framework to recognise various expressions by tracking facial features. Our method uses localized active shape models to track feature points in the subspace obtained from localized Non-negative Matrix Factorization. The tracked feature points are used to train conditional model for recognising prototypic expressions like Anger, Disgust, Fear, Joy, Surprise and Sadness. We formulate the task as a sequence labelling problem and use Conditional Random Fields(CRF) to probabilistically predict expressions. In CRF, the distribution is conditioned on the entire sequence rather than a single observation. For the joint probability defined for the entire sequence, CRF does global normalization of the exponential model, as opposed to MEMM, for which the per state exponential distribution is locally normalized. Unlike generative models(HMM), no prior dependencies between the features are assumed. We adopt a simplistic approach to classify expressions without explicitly monitoring the change in shapes of the individual facial features. Instead, we allow CRF to learn the complex dependencies between the features and recognize the expressions directly. Experimental results demonstrate that accurately tracked feature shapes provide reliable discriminative cues to robustly recognize facial expressions for an image sequence.