In this paper an adaptive and fully automatic video object tracking scheme is developed on the basis of motion segmentation of the image sequences using a novel probabilistic framework. The basic idea is to track only the moving objects in the current frame and update the frame using a robust Bayesian estimation so that it provides an accurate estimation of the next frame, even when the next frame might be missing. The proposed model uses homogeneity of image regions based upon probabilistic motion parameters to segment out Video Object Regions (VOR). Each VOR is modeled as a 4-clique Markov field. Experimental results on the Claire Video sequence are provided which clearly elucidate that the proposed algorithm is computationally efficient as well as being accurate and almost real time.