In this paper, we propose a novel technique for model based recognition of complex object motion trajectories using Gaussian Mixture Models (GMM). We build our models on Principal Component Analysis (PCA)-based representation of trajectories after segmenting them into small units of perceptually similar pieces of motions. These subtrajectories are then fitted with automatically learnt mixture of Gaussians to estimate the underlying class probability distribution. Experiments are performed on two data sets; the ASL data set (from UCI’s KDD archives) consists of 207 trajectories depicting signs for three words, from Australian Sign Language (ASL); the HJSL data set contains 108 trajectories from sports videos. Our experiments yield an accuracy of 85+% performing much better than existing approaches.