With the advent of many large image databases, both commercial and personal, content-based image retrieval has become an important research area. While most early efforts retrieved images based on appearance, it is now recognized that most users want to retrieve images based on the objects present in them. This paper addresses the challenging task of recognizing common objects in color photographic images. We represent images as sets of feature vectors of multiple types of abstract regions, which come from various segmentation processes. We model each abstract region as a mixture of Gaussian distributions over its feature space. We have developed a new semi-supervised version of the EM algorithm for learning the distributions of the object classes. We use supervisory information to tell the procedure the set of objects that exist in each training image, but we do not use any such supervisory information about where (ie. in which regions) the objects are located in the images. Instead, we rely on our EM-like algorithm to break the symmetry in an initial solution that is estimated with error. Experiments are conducted on a set of 860 images to show the efficacy of our approach.