This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Motion Layer Extraction in the Presence of Occlusion Using Graph Cuts
October 2005 (vol. 27 no. 10)
pp. 1644-1659
Extracting layers from video is very important for video representation, analysis, compression, and synthesis. Assuming that a scene can be approximately described by multiple planar regions, this paper describes a robust and novel approach to automatically extract a set of affine or projective transformations induced by these regions, detect the occlusion pixels over multiple consecutive frames, and segment the scene into several motion layers. First, after determining a number of seed regions using correspondences in two frames, we expand the seed regions and reject the outliers employing the graph cuts method integrated with level set representation. Next, these initial regions are merged into several initial layers according to the motion similarity. Third, an occlusion order constraint on multiple frames is explored, which enforces that the occlusion area increases with the temporal order in a short period and effectively maintains segmentation consistency over multiple consecutive frames. Then, the correct layer segmentation is obtained by using a graph cuts algorithm and the occlusions between the overlapping layers are explicitly determined. Several experimental results are demonstrated to show that our approach is effective and robust.

[1] S. Ayer and H. Sawhney, “Layered Representation of Motion Video Using Robust Maximum-Likelihood Estimation of Mixture Models and MDL Encoding,” Proc. Int'l Conf. Computer Vision, 1995.
[2] G. Adiv, “Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 7, no. 4, pp. 384-401, Apr. 1985.
[3] L. Bergen and F. Meyer, “A Novel Approach to Depth Ordering in Monocular Image Sequences,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000.
[4] S. Birchfield and C. Tomasi, “Multiway Cut for Stereo and Motion with Slanted Surfaces,” Proc. Int'l Conf. Computer Vision, 1999.
[5] Y. Boykov, O. Veksler, and R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[6] Y. Boykov and V. Kolmogorov, “Computing Geodesics and Minimal Surfaces via Graph Cuts,” Proc. Int'l Conf. Computer Vision, 2003.
[7] E. Dahlhaus, D. Johnson, C. Papadimitriou, P. Seymour, and M. Yannakakis, “The Complexity of Multiway Cuts,” Proc. ACM Symp. Theory of Computing, pp. 241-251, 1992.
[8] P. Giaccone and G. Jones, “Segmentation of Global Motion Using Temporal Probabilistic Classification,” Proc. British Machine Vision Conf., 1998.
[9] H. Ishikawa and D. Geiger, “Occlusions, Discontinuities, and Epipolar Lines in Stereo,” Proc. European Conf. Computer Vision, 1998.
[10] S. Kang, R. Szeliski, and J. Chai, ”Handing Occlusions in Dense Multi-View Stereo,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2001.
[11] Q. Ke and T. Kanade, “A Subspace Approach to Layer Extraction,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2001.
[12] Q. Ke and T. Kanade, “A Robust Subspace Approach to Layer Extraction,” Proc. IEEE Workshop Motion and Video Computing, 2002.
[13] S. Khan and M. Shah, “Object Based Segmentation of Video Using Color, Motion and Spatial,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2001.
[14] V. Kolmogorov and R. Zabih, “Visual Correspondence with Occlusions Using Graph Cuts,” Proc. Int'l Conf. Computer Vision, 2001.
[15] V. Kolmogorov and R. Zabih, “Multi-Camera Scene Reconstruction via Graph Cut,” Proc. Eourpean Conf. Computer Vision, 2002.
[16] V. Kwatra, I. Essa, A. Schodl, G. Turk, and A. Bobick, “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” Proc. ACM SIGGRAPH, 2003.
[17] S. Osher and R. Fedkiw, Level Set Methods and Dynamic Implicit Surfaces. Springer-Verlag, 2003.
[18] I. Patras, E. Hendirks, and R. Lagendijk, “Video Segmentation by MAP Labeling of Watershed Segments,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 326-332, Mar. 2001.
[19] J. Sethian, Level Set Methods and Fast Marching Methods. Cambridge Univ. Press, 1999.
[20] J. Shi and J. Malik, “Motion Segmentation and Tracking Using Normalized Cuts,” Proc. Int'l Conf. Computer Vision, 1998.
[21] J. Shi and C. Tomasi, “Good Features to Track,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1994.
[22] P. Smith, T. Drummond, and R. Cipolla, “Layered Motion Segmentation and Depth Ordering by Tracking Edges,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 4, pp. 479-494, Apr. 2004.
[23] R. Szeliski, “Video Mosaics for Virtual Environments,” IEEE Computer Graphics and Applications, vol. 16, no. 2, pp. 22-30, 1996.
[24] H. Tao, H. Sawhney, and R. Kumar, “Object Tracking with Bayesian Estimation of Dynamic Layer Representations,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 1, pp. 75-89, Jan. 2002.
[25] P. Torr and D. Murray, “Outlier Detection and Motion Segmentation,” Proc. SPIE Sensor Fusion Conf. VI, pp. 432-443, 1993.
[26] R. Vidal and Y. Ma, “A Unified Algebraic Approach to 2-D and 3-D Motion Segmentation,” Proc. European Conf. Computer Vision, 2004.
[27] J. Wang and E. Adelson, “Representing Moving Images with Layers,” IEEE Trans. Image Processing, vol. 3, no. 5, pp. 625-638, 1994.
[28] Y. Weiss, “Smoothness in Layers: Motion Segmentation Using Nonparametric on Homographics,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1997.
[29] J. Wills, S. Agarwal, and S. Belongie, “What Went Where,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2003.
[30] J. Xiao and M. Shah, “Two-Frame Wide Baseline Matching,” Proc. Int'l Conf. Computer Vision, 2003.
[31] J. Xiao and M. Shah, “Motion Layer Extraction in the Presence of Occlusion Using Graph Cut,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004.
[32] N. Xu, R. Bansal, and N. Ahuja, “Object Segmentation Using Graph Cuts Based Active Contours,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2003.
[33] L. Zelnik-Manor and M. Irani, “Multi View Subspace Constraints on Homographies,” Proc. Int'l Conf. Computer Vision, 1999.
[34] Y. Zhou and H. Tao, “A Background Layer Model for Object Tracking Through Occlusion,” Proc. Int'l Conf. Computer Vision, 2003.
[35] http://www.cs.ucf.edu/~vision/projectsmotion_layer_ extraction /, 2005.

Index Terms:
Index Terms- Layer-based motion segmentation, video analysis, graph cuts, level set representation, occlusion order constraint.
Citation:
Jiangjian Xiao, Mubarak Shah, "Motion Layer Extraction in the Presence of Occlusion Using Graph Cuts," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 10, pp. 1644-1659, Oct. 2005, doi:10.1109/TPAMI.2005.202
Usage of this product signifies your acceptance of the Terms of Use.