Efficient indexing and retrieval of digital video data is an important aspect of video database management. Video indexing involves content analysis of video sequences, which is usually a computationally intensive process. Since most video data are stored in compressed format, processing directly in the compressed domain offers the possibility of computationally more efficient algorithms. We present here an efficient algorithm for detecting skin color regions directly in MPEG video sequences. Most existing methods for skin color detection usually threshold some sort of measure of the likelihood of skin colors for each pixel and treat them independently. Our algorithm takes spatial neighborhood information of each block into account by performing region growing for those blocks with a high probability of belonging to the skin color class, since skin color blocks usually appear as contiguous regions in an image. A novel method for finding an adaptive threshold to control the region growing process is also presented, which incorporates the edge information into the segmentation. Experimental results on several video sequences show that the proposed algorithm is able to detect skin color regions in MPEG video sequences with various scene complexities.