An immersive experience where one views imagery captured from a panoramic camera with a head mounted display (HMD) or "cave" display system is an example of image processing that would be appreciated by the masses. In such a system, a user can see what would be seen from several viewpoints in a natural way by simply moving their head around. Virtual perspective views would be generated from recorded imagery collected by a panoramic camera from a set of locations. With image based rendering techniques, the user could also see views from view-points different from where the panoramic camera was placed. This paper proposes a simple framework for designing such systems based on image cubes which has the benefits of fast low latency operation and an efficient way to create intermediate images. The image cube method de-couples the image creation from the output image generation for the low latency required for realistic HMD immersive viewing. A fast algorithm for generating intermediate views along linear paths between capture sites based on pre-calculated disparity maps is also presented. A prototype system is shown.