Subscribe
Issue No.06 - June (2010 vol.32)
pp: 1134-1141
Kemal Egemen Ozden , University of Leuven, Heverlee
Luc Van Gool , University of Leuven, Heverlee
ABSTRACT
Multibody structure from motion (SfM) is the extension of classical SfM to dynamic scenes with multiple rigidly moving objects. Recent research has unveiled some of the mathematical foundations of the problem, but a practical algorithm which can handle realistic sequences is still missing. In this paper, we discuss the requirements for such an algorithm, highlight theoretical issues and practical problems, and describe how a static structure-from-motion framework needs to be extended to handle real dynamic scenes. Theoretical issues include different situations in which the number of independently moving scene objects changes: Moving objects can enter or leave the field of view, merge into the static background (e.g., when a car is parked), or split off from the background and start moving independently. Practical issues arise due to small freely moving foreground objects with few and short feature tracks. We argue that all of these difficulties need to be handled online as structure-from-motion estimation progresses, and present an exemplary solution using the framework of probabilistic model-scoring.
INDEX TERMS
Structure-from-motion, motion segmentation, scale ambiguity, model selection, affine degeneracy.
CITATION
Kemal Egemen Ozden, Konrad Schindler, Luc Van Gool, "Multibody Structure-from-Motion in Practice", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.32, no. 6, pp. 1134-1141, June 2010, doi:10.1109/TPAMI.2010.23
REFERENCES
 [1] P. Beardsley, P. Torr, and A. Zisserman, "3D Model Acquisition from Extended Image Sequences," Proc. European Conf. Computer Vision, 1996. [2] J. Costeira and T. Kanade, "A Multi-Body Factorization Method for Motion Analysis," Proc. Int'l Conf. Computer Vision, 1995. [3] T. Darrel, A. Azerbayejani, and P. Pentland, "Segmentation of Rigidly Moving Objects Using Multiple Kalman Filters," MIT Media Lab, 1994. [4] O. Faugeras, Q.-T. Luong, and T. Papadopoulo, The Geometry of Multiple Images. MIT Press, 2001. [5] A.W. Fitzgibbon and A. Zisserman, "Multibody Structure and Motion: 3D Reconstruction of Independently Moving Objects," Proc. European Conf. Computer Vision, 2000. [6] M. Han and T. Kanade, "Reconstruction of a Scene with Multiple Linearly Moving Objects," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000. [7] M. Han and T. Kanade, "Multiple Motion Scene Reconstruction from Uncalibrated Views," Proc. Int'l Conf. Computer Vision, 2003. [8] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000. [9] K. Kanatani, "Motion Segmentation by Subspace Separation and Model Selection," Proc. Int'l Conf. Computer Vision, 2001. [10] A. Leonardis, A. Gupta, and A. Bajcsy, "Segmentation of Range Images as the Search for Geometric Parametric Models," Int'l J. Computer Vision, vol. 14, pp. 253-277, 1995. [11] T. Li, V. Kallem, D. Singaraju, and R. Vidal, "Projective Factorization of Multiple Rigid-Body Motions," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007. [12] D. Nister, O. Naroditsky, and J. Bergen, "Visual Odometry," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004. [13] K. Ozden, K. Cornelis, L. Van Eycken, and L. Van Gool, "Reconstructing 3D Trajectories of Independently Moving Objects Using Generic Constraints," Computer Vision and Image Understanding, vol. 96, no. 3, pp. 453-471, 2004. [14] K.E. Ozden, K. Cornelis, L. Van Eycken, and L. Van Gool, "Reconstructing 3D Independent Motions Using Non-Accidentalness," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004. [15] K.E. Ozden and L. Van Gool, "Background Recognition in Dynamic Scenes with Motion Constraints," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005. [16] M. Pollefeys, L. Van Gool, M. Vergauwen, F. Verbiest, K. Cornelis, J. Tops, and R. Koch, "Visual Modeling with a Hand-Held Camera," Int'l J. Computer Vision, vol. 59, no. 3, pp. 207-232, 2004. [17] G. Qian, R. Chellappa, and Q. Zheng, "Bayesian Algorithms for Simultaneous Structure from Motion Estimation of Multiple Independently Moving Objects," IEEE Trans. Image Processing, vol. 14, no. 1, pp. 94-109, Jan. 2005. [18] B.D. Ripley, Pattern Recognition and Neural Networks. Cambridge Univ. Press, 1996. [19] K. Schindler and D. Suter, "Two-View Multibody Structure-and-Motion with Outliers through Model Selection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 6, pp. 983-995, June 2006. [20] K. Schindler, D. Suter, and H. Wang, "A Model Selection Framework for Multibody Structure-and-Motion of Image Sequences," Int'l J. Computer Vision, vol. 79, pp. 159-177, 2008. [21] G. Schwarz, "Estimating the Dimension of a Model," Annals of Statistics, vol. 6, pp. 461-464, 1978. [22] P.H.S. Torr, "Bayesian Model Estimation and Selection for Epipolar Geometry and Generic Manifold Fitting," Int'l J. Computer Vision, vol. 50, no. 1, pp. 35-61, 2002. [23] R. Tron and R. Vidal, "A Benchmark for the Comparison of 3D Motion Segmentation Algorithms," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007. [24] R. Vidal and R. Hartley, "Motion Segmentation with Missing Data Using Power Factorization and GPCA," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004. [25] R. Vidal, Y. Ma, S. Soatto, and S. Sastry, "Two-View Multibody Structure from Motion," Int'l J. Computer Vision, vol. 68, no. 1, pp. 7-25, 2006. [26] H. Wang and D. Suter, "Robust Fitting by Adaptive-Scale Residual Consensus," Proc. European Conf. Computer Vision, 2008. [27] L. Wolf and A. Shashua, "On Projection Matrices ${P}^k\rightarrow {P}^2$ , $k=3,\ldots,6$ , and Their Applications in Computer Vision," Proc. Int'l Conf. Computer Vision, 2001. [28] J. Yan and M. Pollefeys, "A General Framework for Motion Segmentation: Independent, Articulated, Rigid, Non-Rigid, Degenerate and Non-Degenerate," Proc. European Conf. Computer Vision, 2006.