A recursive two-step method to recover structure and motion from image sequences based on Kalman filtering is described in this paper. The algorithm consists of two major steps. The first step is an extended Kalman filter for the estimation of the object's pose. The second step is a set of extended Kalman filters, one for each model point, for refining the positions of the model features in the 3D space. The initial guess is a planar model formed under the assumption of orthographic projection on the first image. These two steps alternate from frames to frames. The planar model converges to the final structure as the image sequence is scanned sequentially. The performance of the algorithm is demonstrated with both synthetic data and real world objects. Comparisons with different approaches have been performed and show that our method is more efficient.