This paper addresses robust recovery of structure and motion for rigid bodies in video sequences. For small interframe motion, feature appearance is commonly estimated according to the previous frame. However, in the case of occlusion or corrupt frames, the small inter-frame motion model fails. In this paper, we propose to use robust identification techniques to estimate the motion dynamics based on a set of previous frames. These dynamics are then recursively used to robustly estimate object structure and motion and to predict the object appearance in future frames. Results are tested for 3d reconstructions of rigid bodies in real and synthetic image sequences.