In this paper, we propose a novel method for panoramic 3D scene recovery using rotational stereo cameras with simple epipolar constraints. By rotating two parallel stereo cameras about a vertical axis with a constant velocity, we acquire two sampled spacio-temporal volumes which are made of the sequential images captured by a uniform angular interval. The two spacio-temporal volumes can be resampled into a set of multi-perspective panoramas. We analyze the epipolar geometry among images (panoramas and original images) of two spacio-temporal volumes. The result shows that only three types of simple epipolar constraints (epipolar line is row or column of image) exist in the two spacio-temporal volumes. Then we compute a depth map from four image pairs using a multi-baseline algorithm with the three types of epipolar constraints; that is horizontal, vertical and combination of them. Experimental results using both synthetic and real images show that our approach produces high quality panoramic 3D reconstruction.