This Article 
 Bibliographic References 
 Add to: 
Spatiotemporal Segmentation Based on Region Merging
September 1998 (vol. 20 no. 9)
pp. 897-915

Abstract—This paper proposes a technique for spatiotemporal segmentation to identify the objects present in the scene represented in a video sequence. This technique processes two consecutive frames at a time. A region-merging approach is used to identify the objects in the scene. Starting from an oversegmentation of the current frame, the objects are formed by iteratively merging regions together. Regions are merged based on their mutual spatiotemporal similarity. The spatiotemporal similarity measure takes both temporal and spatial information into account, the emphasis being on the former. We propose a Modified Kolmogorov-Smirnov test for estimating the temporal similarity. This test efficiently uses temporal information in both the residual distribution and the motion parametric representation. The region-merging process is based on a weighted, directed graph. Two complementary graph-based clustering rules are proposed, namely, the strong rule and the weak rule. These rules take advantage of the natural structures present in the graph. Also, the rules take into account the possible errors and uncertainties reported in the graph. The weak rule is applied after the strong rule. Each rule is applied iteratively, and the graph is updated after each iteration. Experimental results on different types of scenes demonstrate the ability of the proposed technique to automatically partition the scene into its constituent objects.

[1] P. Bouthemy and E. Francois, "Motion Segmentation and Qualitative Dynamic Scene Analysis From an Image Sequence," Int'l J. Computer Vision, vol. 10, no. 2, pp. 157-182, 1993.
[2] G. Adiv, "Determining Three-Dimensional Motion and Structure From Optical Flow Generated by Several Moving Objects," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 7, no. 4, pp. 384-401, July 1985.
[3] F. Dufaux, F. Moscheni, and A. Lippman, “Spatiotemporal Segmentation Based on Motion and Static Ssegmentation,” Proc. IEEE Int'l Conf. Image Processing, vol. 1, pp. 306-309, Oct. 1995.
[4] M. Kunt, A. Ikonomopoulos, and M. Kocher, "Second Generation Image Coding Techniques," Proc. IEEE, vol. 73, no. 4, pp. 549-575, Apr. 1985.
[5] F. Dufaux and F. Moscheni, "Background Mosaicking for Low Bit Rate Video Coding," Proc. ICIP'96, vol. I, pp. 673-676,Lausanne, Switzerland, Sept. 1996.
[6] M. Allmen and C.R. Dyer, "Computing Spatiotemporal Relations for Dynamic Perceptual Organisation," CVGIP: Image Understanding, vol. 58, no. 3, pp. 338-351, Nov. 1993.
[7] S. Ayer, P. Schroeter, and J. Bigün, "Segmentation of Moving Objects by Robust Motion Parameter Estimation Over Multiple Frames," ECCV'94, vol. 2, pp. 316-327,Stockholm, Sweden, May 1994.
[8] J.L. Barron, D.J. Fleet, and S.S. Beauchemin, "Performance of Optical Flow Techniques. Int'l J. Computer Vision, vol. 12, no. 1, pp. 43-77, 1994.
[9] J.R. Bergen, P.J. Burt, R. Hingorani, and S. Peleg, "A Three-Frame Algorithm for Estimating Two-Component Image Motion," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, pp. 886-895, Sept. 1992.
[10] A.D. Jepson and M. Black, “Mixture Models for Optical Flow Computation,” Proc. Computer Vision and Pattern Recognition, pp. 760-761, June 1993.
[11] M. Irani, B. Rousso, and S. Peleg, "Computing Occluding and Transparent Motions," Int'l J. Computer Vision, vol. 12, no. 1, pp. 5-16, 1994.
[12] M.J. Black and P. Anandan,“A framework for the robust estimation of optical flow,” Proc. Int’l Conf. on Computer Vision, ICCV-93, Berlin, pp. 231-236, May 1993.
[13] H.G. Musmann, M. Hoetter, and J. Ostermann, "Object-Oriented Analysis-Synthesis Coding of Moving Images," Signal Processing: Image Communication, vol. 1, no. 2, pp. 117-138, Oct. 1989.
[14] S.F. Wu and J. Kittler, "A Gradient-Based Method for General Motion Estimation and Segmentation," J. Visual Comm. and Image Representation, vol. 4, no. 1, pp. 25-38, Mar. 1993.
[15] P.J. Burt, R. Hingorani, and R.J. Kolczynski, "Mechanisms for Isolating Component Patterns in the Sequential Analysis of Multiple Motion," IEEE Workshop Visual Motion, pp. 187-193,Princeton, N.J., Oct. 1991.
[16] P.J. Burt, J.R. Bergen, R. Hingorani, R. Kolczynski, W.A. Lee, A. Leung, J. Lubin, and H. Shvaytser, “Object Tracking with a Moving Camera: An Application of Dynamic Motion Analysis,” Proc. IEEE Workshop on Motion, Mar. 1989.
[17] M. Irani, B. Rousso, and S. Peleg, "Detecting and Tracking Multiple Moving Objects Using Temporal Integration," G. Sandini, ed., Second European Conf. Computer Vision, pp. 282-287,S.Margherita, Italy, 1992. Springer-Verlag.
[18] P.J. Rousseeuw and A.M. Leroy, Robust Regression and Outlier Detection.New York, NY: John Wiley&Sons, Inc, 1987.
[19] S. Ayer and P. Schroeter, "Hierarchical Robust Motion Estimation for Segmentation of Moving Objects," IEEE Workshop on Image and Multidimensional Signal Processing, pp. 122-123,Cannes, France, Sept. 1993.
[20] P. Schroeter and S. Ayer, "Multi-Frame Based Segmentation of Moving Objects by Combining Luminance and Motion," Proc. EUSIPCO 94,Edinburgh, U.K., Sept. 1994.
[21] S. Ayer, P. Schroeter, and J. Bigün, "Tracking Based on Hierachical Multiple Motion Estimation and Robust Regression," Time Varying Image Processing and Moving Objects Recognitions,Florence, Italy, June 1993.
[22] B. Duc, P. Schroeter, and J. Bigün, "Spatiotemporal Robust Motion Estimation and Segmentation," Sixth Int'l Conf. Computer Analysis of Images and Patterns, pp. 238-245,Prague,6-8 Sept. 1995.
[23] C. Gu, "Multivalued Morphology and Segmentation-Based Coding," PhD thesis, Swiss Fed. Inst. of Tech nology, Lausanne, 1996.
[24] S. Ayer and H. Sawhney, "Layered Representation of Motion Video Using Robust Maximum-Likelihood Estimation of Mixture Models and mdl Encoding," Int'l Conf. Computer Vision, pp. 777-784,Cambridge, Mass., June 1995.
[25] H. Zheng and S.D. Blostein, "Motion-Based Object Segmentation and Estimation Using the MDL Principle," IEEE Trans. Image Processing, vol. 4, no. 9, pp. 1,223-1,235, Sept. 1995.
[26] R. Szeliski and H.-Y. Shum, "Motion Estimation With Quadtree Splines," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 12, pp. 1,199-1,210, 1996
[27] P. Salembier, L. Torres, F. Meyer, and C. Gu, Region-Based Video Coding Using Mathematical Morpholgy," Proc. IEEE, vol. 83, no. 6, pp. 843-857, June 1995.
[28] P. Anandan, J.R. Bergen, K.J. Hanna, and R. Hingorani, "Hierarchical Model-Based Motion Estimation," M.I. Sezan and R.L. Lagendijk, eds., Motion Analysis and Image Sequence Processing, pp. 1-22. Kluwer Academic Publishers, 1993.
[29] J.Y.A. Wang and E.H. Adelson, "Spatiotemporal Segmentation of Video Data," SPIE Proc. Image and Video Processing II, vol. 2,182, San Jose, Calif., Feb. 1994.
[30] G. Adiv, “Inherent Ambiguities in Recovering 3-D Motion and Structure from a Noisy Flow Field,” Trans. Pattern Analysis and Machine Intelligence, vol. 11, pp. 477–489, 1989.
[31] D.W. Murray and B.F. Buxton, "Scene Segmentation From Visual Motion Using Global Optimization," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, no. 2, pp. 220-228, Mar. 1987.
[32] P. Bouthemy and J. Santillana Rivero, "A Hierarchical Likelihood Approach for Region Segmentation According to Motion-Based Criteria," ICCV'87, pp. 463-467,London, UK, 1987.
[33] F. Dufaux and F. Moscheni, "Segmentation-Based Motion Estimation for Second Generation Video Coding Techniques," L. Torres and M. Kunt, eds., Video Coding: The Second Generation Approach, pp. 219-263. Kluwer Academic Publishers, 1995.
[34] M.J. Black, "Combining Intensity and Motion for Incremental Segmentation and Tracking Over Long Image Sequences," G. Sandini, ed., Second European Conf. Computer Vision, pp. 485-493,Santa Margherita, Italy, May 1992.
[35] F. Heitz and P. Bouthemy, Multimodal Estimation of Discontinuous Optical Flow Using Markov Random Fields IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 12, pp. 1217-1232, Dec. 1993.
[36] H. Gu, Y. Shirai, and M. Asada, "MDL-Based Segmentation and Motion Modelling in a Long Image Sequence of Scene With Multiple Independently Moving Objects," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 1, pp. 58-64, Jan. 1996.
[37] W.B. Thompson, "Combining Motion and Contrast for Segmentation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 2, no. 6, pp. 543-549, Nov. 1980.
[38] B.A. Wandell, Foundations of Vision. Sinauer Associates, Inc., 1995.
[39] J.Y.A. Wang and E.H. Adelson, Representing Moving Images with Layers IEEE Trans. Image Processing, vol. 3, no. 5, pp. 625-638, Sept. 1994.
[40] F. Moscheni, "Spatiotemporal Segmentation and Object Tracking: An Application to Second Generation Video Coding," PhD thesis, Ecole Polytechnique Fédérale de Lausanne (EPFL),Lausanne, Switzerland, 1997.
[41] W.B. Thompson and T.-G. Pong, "Detecting Moving Objects," Int'l J. Computer Vision, vol. 4, pp. 39-57, 1990.
[42] J.W. Park and S.U Lee, "Joint Image Segmentation and Motion Estimation for Low Bit Rate Video Coding," Proc. ICIP'96, vol. 2, pp. 501-504,Lausanne, Switzerland, Sept. 1996.
[43] A. Papoulis, Probability, Random Variables, and Stochastic Processes.New York, NY: McGraw-Hill, 1991.
[44] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C.Cambridge, Mass.: Cambridge University Press, 1992.
[45] P. Meer, D. Mintz, A. Rosenfeld, and D.Y. Kim, "Robust Regression Methods for Computer Vision: A Review," Int'l J. Computer Vision, vol. 6, no. 1, pp. 59-70, 1991.
[46] A.K. Jain and C.D. Dubes, Algorithms for Clustering Data.Englewood Cliffs, NJ: Prentice Hall, 1988.
[47] M. Schütz and T. Ebrahimi, "Matching Error Based Criterion of Region Merging for Joint Motion Estimation and Segmentation Techniques," Proc. ICIP'96, vol. II, pp. 509-512,Lausanne, Switzerland, Sept. 1996.
[48] F. Moscheni, F. DuFaux, and M. Kunt, “A New Two-Stage Global/Local Motion Estimation Based on a Background/Foreground Segmentation,” Proc. Int'l Conf. Acoustics, Speech, and Signal Processing, May 1995.
[49] L. Kaufman and P.J. Rousseeuw, Finding Groups in Data.New York, NY: John Wiley&Sons, Inc., 1990.

Index Terms:
Automatic spatiotemporal segmentation, object segmentation, region merging, modified Kolmogorov-Smirnov test, weighted directed graph.
Fabrice Moscheni, Sushil Bhattacharjee, Murat Kunt, "Spatiotemporal Segmentation Based on Region Merging," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 9, pp. 897-915, Sept. 1998, doi:10.1109/34.713358
Usage of this product signifies your acceptance of the Terms of Use.