In this paper, we propose a new approach to simultaneously estimate the motion parameters and shape parameters of a rigid object over the image sequence while the object is in motion. Our approach applies the Kalman filter multiple times to sequentially reduce the error of feature point correspondences and to generate an accurate motion parameter by propagating the object motion uncertainty over the image frame as well as optimally establishing the feature correspondences on the basis of template-based optical flows. Some preliminarily experimental results are shown to verify our proposed approach.