Content-based image retrieval (CBIR) systems often rely on global image characteristics such as color and texture histograms, but not the spatial relationships among component objects of an image. In this paper, we describe a new image retrieval scheme where objects of interest in an image are first identified and spatial relationships among them as well as their features including semantic information are annotated to represent the image. An important and unique aspect of the system is that users are allowed to formulate a query based on both objects of an image and their spatial relationships. This could give better precision and flexibility to retrieve images from image databases. In the prototype system, semantic information of interest for objects and the spatial relationships among them are captured and represented using XML