The color set concept of Smith and Chang is a useful and efficient approach to image indexing and query in a way that combines color and spatial information. By indexing relatively important regions based on both their color content and their spatial locations, the color set method allows rapid retrieval of images matching a specified color-spatial query. To produce a color set representation for an image, it is necessary to determine the relevant color combinations that occur in the image. We present here a new technique for determining the relevant color combinations much more efficiently than the original algorithm. Using this refinement, the process of analyzing an image in order to produce a color set description becomes many times faster.