Peter Sturm,Radu Horaud, INRIAThis paper is about the problem of structure and motion recovery from two views of a rigid scene. Especially, we deal with the case of scenes containing planes, i.e. there are sets of coplanar points. Coplanarity is a strong constraint for both structure recovery and motion estimation. Most existing works do only exploit one of the two aspects, or, if both, then in a sub-optimal manner. A typical example is to estimate motion (epipolar geometry) using raw point correspondences, to perform a 3D reconstruction and then to fit planes and maybe correct 3D point positions to make them coplanar. In this paper, we present an approach to estimate camera motion and piecewise planar structure simultaneously and optimally: the result is the estimation of camera motion and 3D structure, that minimizes reprojection error, while satisfying the piecewise planarity. The estimation problem is minimally parameterized using 2D entities? epipoles, epipolar transformation, plane homographies and image points?subsequently deriving the corresponding 3D entities is trivial. Experimental results show that the reconstruction is of clearly superior quality compared to traditional methods based only on points, even if the scene is not perfectly piecewise planar.