The goal of computer vision is to extract information about the world from collections of images. This information might be used to recognize or manipulate objects, to control movement through the environment, to measure or determine the condition of objects, and for many other purposes. The goal of this paper is to consider the representation of information derived from a collection of images and how it may support some of these tasks. By "collection of images" we mean any set of images relevant to a given scene. This includes video sequences, multiple images from a single still camera, or multiple images from different cameras. The central thesis of this paper is that the traditional approach to representation of information about scenes by relating each image to an abstract three dimensional coordinate system may not always be appropriate. An approach that more directly represents the relationships among the collection of images has a number of advantages. These relationships can also be computed using practical and efficient algorithms. This paper presents an hierarchical framework for scene representation. Each increasing level in the hierarchy supports additional types of tasks so that the overall structure grows in capability as more information about the scene is acquired. The proposed hierarchy of representations is as follows: (1) The images themselves (2) Two dimensional image mosaics. (3) Image mosaics with parallax and (4) Layers and tiles with parallax. We develop the algorithms used to build these representations and demonstrate results on real image sequences. Finally, the application of these representations to real world problems is discussed.