Scene extraction is the first step toward semantic understanding of the video. This paper presents an effective approach to video scene extraction based on the analysis of background images. Our approach exploits the fact that shots belonging to one particular scene often have similar background. Although part of the video frame is covered by foreground objects, background scene still can be reconstructed by mosaic technique. The proposed scene extraction algorithm consists of two main components: the determination of shot similarity measure and shot grouping process. In our approach, several low-level visual features are integrated to compute the similarity measure between two shots. On the other hand, highlevel knowledge in cinematography is used to guide the shot grouping process. Experimental results show that our approach is promising and outperforms some existing technique.