The study proposes a novel scheme to extract various captions in sports videos. A caption detection process based on an iteratively temporal averaging technique is used to automatically detect and locate a caption region in a series of video frames. Moreover, a caption identification approach is used to classify some caption candidates into one of the caption types in order to extract accurately the contents of the caption rather than a projection-based segmentation process. Experimental results show that the proposed caption extraction approach is robust for generalizing the learning knowledge to identify various sports captions.