This paper presents a framework for the synthesis of stereoscopic video using as input only a monoscopic image sequence. Initially, bi-directional 2D motion estimation is performed, which is followed by an efficient method for the reliable tracking of object contours. Rigid 3D motion and structure is recovered utilizing extended Kalman filtering. Finally, occlusions are dealt with a novel Bayesian framework, which exploits future information to correctly reconstruct occluded areas. Experimental evaluation shows that the layered object scene representation, combined with the proposed methods for object tracking throughout the sequence and occlusion handling, yields very accurate results.