The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - Jan. (2013 vol.35)
pp: 5-17
F. Lafarge , Geometrica Res. Group, INRIA Sophia Antipolis, Sophia Antipolis, France
R. Keriven , Acute3D, Sophia Antipolis, France
M. Brédif , Matis Lab., Univ. Paris Est, St. Mande, France
Hoang-Hiep Vu , Imagine Group, Univ. Paris Est, Marne la Vallee, France
ABSTRACT
We present an original multiview stereo reconstruction algorithm which allows the 3D-modeling of urban scenes as a combination of meshes and geometric primitives. The method provides a compact model while preserving details: Irregular elements such as statues and ornaments are described by meshes, whereas regular structures such as columns and walls are described by primitives (planes, spheres, cylinders, cones, and tori). We adopt a two-step strategy consisting first in segmenting the initial mesh-based surface using a multilabel Markov Random Field-based model and second in sampling primitive and mesh components simultaneously on the obtained partition by a Jump-Diffusion process. The quality of a reconstruction is measured by a multi-object energy model which takes into account both photo-consistency and semantic considerations (i.e., geometry and shape layout). The segmentation and sampling steps are embedded into an iterative refinement procedure which provides an increasingly accurate hybrid representation. Experimental results on complex urban structures and large scenes are presented and compared to state-of-the-art multiview stereo meshing algorithms.
INDEX TERMS
Image reconstruction, Three dimensional displays, Shape, Adaptation models, Semantics, Stereo image processing,jump-diffusion, 3D modeling, multiview stereo, urban scenes, hybrid representation
CITATION
F. Lafarge, R. Keriven, M. Brédif, Hoang-Hiep Vu, "A Hybrid Multiview Stereo Algorithm for Modeling Urban Scenes", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.35, no. 1, pp. 5-17, Jan. 2013, doi:10.1109/TPAMI.2012.84
REFERENCES
[1] H. Mayer, "Object Extraction in Photogrammetric Computer Vision," J. Photogrammetry and Remote Sensing, vol. 63, no. 2, pp. 213-222, 2008.
[2] Z. Zhu and T. Kanade, "Special Issue on Modeling and Representations of Large-Scale 3D Scenes," Int'l J. Computer Vision, vol. 78, nos. 2-3, 2008.
[3] Y. Furukawa and J. Ponce, "Accurate, Dense, and Robust Multi-View Stereopsis," Proc. IEEE Conf. Computer Visison and Pattern Recognition, 2007.
[4] M. Goesele, N. Snavely, B. Curless, H. Hoppe, and S. Seitz, "Multi-View Stereo for Community Photo Collections," Proc. 11th IEEE Int'l Conf. Computer Vision, 2007.
[5] N. Campbell, G. Vogiatzis, C. Hernandez, and R. Cipolla, "Using Multiple Hypotheses to Improve Depth-Maps for Multi-View Stereo," Proc. 10th European Conf. Computer Vision, 2008.
[6] K. Kolev and D. Cramers, "Integration of Multiview Stereo and Silhouettes via Convex Functionals on Convex Domains," Proc. 10th European Conf. Computer Vision, 2008.
[7] D. Scharstein and R. Szeliski, "A Taxonomy and Evaluation of Dense 2-Frame Stereo Correspondence Algorithms," Int'l J. Computer Vision, vol. 47, nos. 1-3, pp. 7-42, 2002.
[8] M.Z. Brown, D. Burschka, and G.D. Hager, "Advances in Computational Stereo," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 8, pp. 993-1008, Aug. 2003.
[9] H. Hirschmuller, "Stereo Processing by Semi-Global Matching and Mutual Information," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 328-341, Feb. 2008.
[10] J.-P. Pons, R. Keriven, and O. Faugeras, "Multi-View Stereo Reconstruction and Scene Flow Estimation with A Global Image-Based Matching Score," Int'l J. Computer Vision, vol. 72, no. 2, pp. 179-193, 2007.
[11] H. Vu, R. Keriven, P. Labatut, and J. Pons, "Towards High-Resolution Large-Scale Multiview Stereo," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[12] P. Labatut, J.-P. Pons, and R. Keriven, "Efficient Multi-View Reconstruction of Large-Scale Scenes Using Interest Points, Delaunay Triangulation and Graph Cuts," Proc. 11th IEEE Int'l Conf. Computer Vision, 2007.
[13] C. Fruh and A. Zakhor, "An Automated Method for Large-Scale, Ground-Based City Model Acquisition," Int'l J. Computer Vision, vol. 60, no. 1, pp. 5-24, 2004.
[14] A. Banno, T. Masuda, T. Oishi, and K. Ikeuchi, "Flying Laser Range Sensor for Large-Scale Site-Modeling and Its Applications in Bayon Digital Archival Project," Int'l J. Computer Vision, vol. 78, nos. 2-3, pp. 207-222, 2008.
[15] M. Pollefeys et al., "Detailed Real-Time Urban 3D Reconstruction from Video," Int'l J. Computer Vision, vol. 78, nos. 2-3, pp. 143-167, 2008.
[16] N. Agarwal, S. and Snavely, I. Simon, S. Seitz, and R. Szeliski, "Building Rome in a Day," Proc. IEEE Int'l Conf. Computer Vision, 2009.
[17] J.-M. Frahm, P. Fite Georgel, D. Gallup, T. Johnson, R. Raguram, C. Wu, Y.H. Jen, E. Dunn, B. Clipp, S. Lazebnik, and M. Pollefeys, "Building Rome on a Cloudless Day," Proc. 11th European Conf. Computer Vision, 2010.
[18] S. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, "A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[19] C. Strecha, W. Von Hansen, L. Van Gool, P. Fua, and U. Thoennessen, "On Benchmarking Camera Calibration and Multi-View Stereo for High Resolution Imagery," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[20] C. Baillard and A. Zisserman, "Automatic Reconstruction of Piecewise Planar Models from Multiple Views," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
[21] A.-L. Chauve, P. Labatut, and J.-P. Pons, "Robust Piecewise-Planar 3D Reconstruction and Completion from Large-Scale Unstructured Point Data," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[22] S.N. Sinha, D. Steedly, and R. Szeliski, "Piecewise Planar Stereo for Image-Based Rendering," Proc. 12th IEEE Int'l Conf. Computer Vision, 2009.
[23] J. Xiao, T. Fang, P. Tan, P. Zhao, E. Ofek, and L. Quan, "Image-Based Facade Modeling," ACM Trans. Graphics, vol. 27, no. 5, Article 161, 2008.
[24] Y. Furukawa, B. Curless, S. Seitz, and R. Szeliski, "Manhattan-World Stereo," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[25] J.M. Coughlan and A.L. Yuille, "The Manhattan World Assumption: Regularities in Scene Statistics Which Enable Bayesian Inference," Proc. Conf. Neural Information Processing Systems, 2000.
[26] A. Dick, P. Torr, and R. Cipolla, "Modelling and Interpretation of Architecture from Several Images," Int'l J. Computer Vision, vol. 60, no. 2, pp. 111-134, 2004.
[27] F. Lafarge, X. Descombes, J. Zerubia, and M. Pierrot-Deseilligny, "Building Reconstruction from a Single DEM," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[28] C. Brenner and N. Ripperda, "Extraction of Facades Using RJMCMC and Constraint Equations," Proc. Conf. Photogrammetric and Computer Vision, 2006.
[29] P. Muller, G. Zeng, P. Wonka, and L. Van Gool, "Image-Based Procedural Modeling of Facades," Proc. ACM Siggraph, 2007.
[30] P. Koutsourakis, O. Teboul, L. Simon, G. Tziritas, and N. Paragios, "Single View Reconstruction Using Shape Grammars for Urban Environments," Proc. 12th IEEE Int'l Conf. Computer Vision, 2009.
[31] C. Vanegas, D. Aliaga, and B. Benes, "Building Reconstruction Using Manhattan-World Grammars," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[32] S. Lee and R. Nevatia, "Extraction and Integration of Window in a 3D Building Model from Ground View Images," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2004.
[33] F. Han and S. Zhu, "Bottom-Up/Top-Down Image Parsing by Attribute Graph Grammar," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005.
[34] M. Brédif, D. Boldo, M. Pierrot-Deseilligny, and H. Maître, "3D Building Reconstruction with Parametric Roof Superstructures," Proc. IEEE Int'l Conf. Image Processing, 2007.
[35] C. Shen, J.F. O'Brien, and J. Shewchuk, "Interpolating and Approximating Implicit Surfaces from Polygon Soup," ACM Trans. Graphics, vol. 23, no. 3, pp. 896-904, 2004.
[36] S. Osher and R. Fedkiw, Level Set Methods and Dynamic Implicit Surfaces. Springer, 2003.
[37] R. Keriven and O. Faugeras, "Variational Principles, Surface Evolution, PDEs, Level Set Methods and the Stereo Problem," IEEE Trans. Image Processing, vol. 7, no. 3, pp. 336-344, Mar. 1998.
[38] F. Lafarge, R. Keriven, and M. Brédif, "Insertion of 3D-Primitives in Mesh-Based Representations: Towards Compact Models Preserving the Details," IEEE Trans. Image Processing, vol. 19, no. 7, pp. 1683-1694, July 2010.
[39] D. Gallup, J.-M. Frahm, and M. Pollefeys, "Piecewise Planar and Non-Planar Stereo for Urban Scene Reconstruction," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[40] P. Labatut, J.-P. Pons, and R. Keriven, "Hierarchical Shape-Based Surface Reconstruction for Dense Multi-View Stereo," Proc. 12th IEEE Int'l Computer Vision Workshops, 2009.
[41] U. Grenander and M. Miller, "Representations of Knowledge in Complex Systems," J. Royal Statistical Soc., vol. 56, no. 4, pp. 549-603, 1994.
[42] F. Lafarge, R. Keriven, M. Brédif, and H. Vu, "Hybrid Multi-View Reconstruction by Jump-Diffusion," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2010.
[43] Y. Boykov, O. Veksler, and R. Zabih, "Fast Approximate Energy Minimization via Graph Cuts," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[44] M. Attene, S. Katz, M. Mortara, G. Patane, M. Spagnuolo, and A. Tal, "Mesh Segmentation—A Comparative Study," Proc. IEEE Int'l Conf. Shape Modeling and Applications, 2006.
[45] A. Shamir, "A Survey on Mesh Segmentation Techniques," Computer Graphics Forum, vol. 27, no. 6, pp. 1539-1556, 2008.
[46] L. Kobbelt, S. Campagna, J. Vorsatz, and H.-P. Seidel, "Interactive Multi-Resolution Modeling on Arbitrary Meshes," Proc. ACM Siggraph, 1998.
[47] F. Han, Z.W. Tu, and S. Zhu, "Range Image Segmentation by an Effective Jump-Diffusion Method," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 9, pp. 1138-1153, Sept. 2004.
[48] Z. Tu, X. Chen, A. Yuille, and S. Zhu, "Image Parsing: Unifying Segmentation, Detection, and Recognition," Proc. IEEE Ninth Int'l Conf. Computer Vision, vol. 63, no. 2, 2005.
[49] A. Srivastava, M. Miller, and U. Grenander, "Multiple Target Direction of Arrival Tracking," IEEE Trans. Signal Processing, vol. 43, no. 5, pp. 1282-1285, May 1995.
[50] P. Green, "Reversible Jump Markov Chains Monte Carlo Computation and Bayesian Model Determination," Biometrika, vol. 82, no. 4, pp. 711-732, 1995.
[51] W. Hastings, "Monte Carlo Sampling Using Markov Chains and Their Applications," Biometrika, vol. 57, no. 1, pp. 97-109, 1970.
[52] S. Geman and C. Huang, "Diffusion for Global Optimization," SIAM J. Control and Optimization, vol. 24, no. 5, pp. 1031-1043, 1986.
[53] P. Salamon, P. Sibani, and R. Frost, "Facts, Conjectures, and Improvements for Simulated Annealing," SIAM Monographs on Math. Modeling and Computation, 2002.
[54] www.cgal.org, 2012.
[55] S. White, "Concepts of Scale in Simulated Annealing," Proc. IEEE Int'l Conf. Computer Design, 1984.
[56] D. Marshall, G. Lukacs, and R. Martin, "Robust Segmentation of Primitives from Range Data in the Presence of Geometric Degeneracy," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 304-314, Mar. 2001.
[57] M. Pauly, N.J. Mitra, J. Wallner, H. Pottmann, and L. Guibas, "Discovering Structural Regularity in 3D Geometry," ACM Trans. Graphics, vol. 27, no. 3, Article 43, 2008.
62 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool