In this paper an efficient scheme for automatic organization of stereo-captured video sequences is presented, which exploits foreground VOP information of frames. More specifically after shot cut detection, for each frame of a shot, a fast, unsupervised foreground VOP extraction algorithm is applied, based on depth information and normalized Motion Geometric Spaces. Then for each frame, a feature vector is constructed, containing color and size characteristics of the foreground VOPs within the frame. Afterwards, for a given shot, key frames are extracted using an optimization method for locating minimally correlated feature vectors. Finally correlation links are generated between the extracted key frames to produce a graph-like structure of the video sequence. Experimental results on real life stereoscopic sequences indicate the promising performance of the proposed scheme.