For more efficient organizing, browsing, and retrieving digital video content, it is important to extract video structure information at both scene and shot levels. This paper presents an effective approach to video scene segmentation based on a pseudo-object-based shot correlation analysis. A new measure of the semantic correlation of consecutive shots based on dominant color grouping and tracking is proposed. A new shot grouping method named expanding window is designed to cluster correlated consecutive shots into one scene. Evaluations based on real-world sports video programs validate the efficiency and effectiveness of our shot correlation measure and scene structure construction.