To acquire seamless visualization of environments fromdifferent viewing positions and orientations, it is desirableto generate virtual images for an arbitrary position givena set of reference views. In this paper, a simple interpolationmethod based on ray-tracing is proposed for viewpointsynthesis from panoramas taken with multi-sensor cameras.Instead of attempting to recover a dense 3D reconstructionof the scene, the method estimates the pose between eachpanorama and then backward projects the point along theray that exhibits the best colour consistency. We show thatby limiting the search space using sparse depth information,both the speed and the accuracy of the interpolationare improved.