Unbiased and consistent estimates of structure and motion can be obtained by least squares minimization of the differential epipolar constraint. Previous work on this subject does not make use of geometrical constraints that often are present in natural and man built scenes. This paper shows how linear constraints among feature points (collinearity and coplanarity) and bilinear relations among such entities (parallelism and incidence) can be incorporated in the minimization process to improve the structure and motion estimates. There are 2 main contributions: (i) the formulation of a constrained minimization problem for structure and motion estimation from optical flow and (ii) the solution of the optimization problem by Levenberg-Marquardt and direct projection. We show that the proposed approach is stable, fast and efficient.