The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.03 - March (2012 vol.34)
pp: 493-505
Kalin Kolev , TU Munchen, Munich
Thomas Brox , Albert-Ludwigs University Freiburg, Germany
Daniel Cremers , TU Munchen, Munich
ABSTRACT
We propose a probabilistic formulation of joint silhouette extraction and 3D reconstruction given a series of calibrated 2D images. Instead of segmenting each image separately in order to construct a 3D surface consistent with the estimated silhouettes, we compute the most probable 3D shape that gives rise to the observed color information. The probabilistic framework, based on Bayesian inference, enables robust 3D reconstruction by optimally taking into account the contribution of all views. We solve the arising maximum a posteriori shape inference in a globally optimal manner by convex relaxation techniques in a spatially continuous representation. For an interactively provided user input in the form of scribbles specifying foreground and background regions, we build corresponding color distributions as multivariate Gaussians and find a volume occupancy that best fits to this data in a variational sense. Compared to classical methods for silhouette-based multiview reconstruction, the proposed approach does not depend on initialization and enjoys significant resilience to violations of the model assumptions due to background clutter, specular reflections, and camera sensor perturbations. In experiments on several real-world data sets, we show that exploiting a silhouette coherency criterion in a multiview setting allows for dramatic improvements of silhouette quality over independent 2D segmentations without any significant increase of computational efforts. This results in more accurate visual hull estimation, needed by a multitude of image-based modeling approaches. We made use of recent advances in parallel computing with a GPU implementation of the proposed method generating reconstructions on volume grids of more than 20 million voxels in up to 4.41 seconds.
INDEX TERMS
Shape from silhouettes, interactive segmentation, convex optimization.
CITATION
Kalin Kolev, Thomas Brox, Daniel Cremers, "Fast Joint Estimation of Silhouettes and Dense 3D Geometry from Multiple Images", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.34, no. 3, pp. 493-505, March 2012, doi:10.1109/TPAMI.2011.150
REFERENCES
[1] X. Bai and G. Sapiro, “A Geodesic Framework for Fast Interactive Image and Video Segmentation and Matting,” Proc. IEEE Int'l Conf. Computer Vision, 2007.
[2] B. Baumgart, “Geometric Modeling for Computer Vision,” PhD thesis, Dept. of Computer Science, Stanford Univ., 1974.
[3] A. Blake, C. Rother, M. Brown, P. Perez, and P. Torr, “Interactive Image Segmentation Using an Adaptive GMMRF Model,” Proc. European Conf. Computer Vision, pp. 428-441, 2004.
[4] Y. Boykov and V. Kolmogorov, “Computing Geodesics and Minimal Surfaces via Graph Cuts,” Proc. IEEE Int'l Conf. Computer Vision, pp. 26-33, 2003.
[5] Y.Y. Boykov and M.P. Jolly, “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in N-D Images,” Proc. IEEE Int'l Conf. Computer Vision, vol. 1, pp. 105-112, 2001.
[6] N.D.F. Campbell, G. Vogiatzis, C. Hernández, and R. Cipolla, “Automatic 3D Object Segmentation in Multiple Views Using Volumetric Graph-Cuts,” Proc. 18th British Machine Vision Conf., vol. 1, pp. 530-539, 2007.
[7] T. Chan, S. Esedolu, and M. Nikolova, “Algorithms for Finding Global Minimizers of Image Segmentation and Denoising Models,” SIAM J. Applied Math., vol. 66, no. 5, pp. 1632-1648, 2006.
[8] A.X. Falcao, J. Stolfi, and R.A. Lotufo, “The Image Foresting Transform: Theory, Algorithms, and Applications,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 19-29, Jan. 2004.
[9] J.-S. Franco and E. Boyer, “Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid,” Proc. IEEE Int'l Conf. Computer Vision, 2005.
[10] Y. Furukawa and J. Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2007.
[11] L.M. Goldschlager, R.A. Shaw, and J. Staples, “The Maximum Flow Problem Is Log Space Complete for p,” Theoretical Computer Science, vol. 21, pp. 105-111, 1982.
[12] L. Grady, “Random Walks for Image Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1768-1783, Nov. 2006.
[13] L. Guan, J.S. Franco, and M. Pollefeys, “3D Occlusion Inference from Silhouette Cues,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2007.
[14] C. Hernandez and F. Schmitt, “Silhouette and Stereo Fusion for 3D Object Modeling,” Computer Vision and Image Understanding, vol. 96, no. 3, pp. 367-392, 2004.
[15] R. Keriven, “A Variational Framework to Shape from Contours,” Technical Report 2002-221, CERMICS, 2002.
[16] M. Klodt, T. Schoenemann, K. Kolev, M. Schikora, and D. Cremers, “An Experimental Comparison of Discrete and Continuous Shape Optimization Methods,” Proc. European Conf. Computer Vision, Oct. 2008.
[17] K. Kolev, T. Brox, and D. Cremers, “Robust Variational Segmentation of 3D Objects from Multiple Views,” Proc. DAGM Symp. Pattern Recognition, K. Franke et al., eds., pp. 688-697, Sept. 2006.
[18] K. Kolev and D. Cremers, “Integration of Multiview Stereo and Silhouettes via Convex Functionals on Convex Domains,” Proc. European Conf. Computer Vision, Oct. 2008.
[19] K. Kolev, M. Klodt, T. Brox, and D. Cremers, “Continuous Global Optimization in Multiview 3D Reconstruction,” Int'l J. Computer Vision, vol. 84, no. 1, pp. 80-96, Aug. 2009.
[20] A. Laurentini, “The Visual Hull Concept for Visual-Based Image Understanding,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 2, pp. 150-162, Feb. 1994.
[21] Y. Li, J. Sun, and H.-Y. Shum, “Video Object Cut and Paste,” ACM Trans. Graphics, vol. 24, no. 3, pp. 595-600, 2005.
[22] J. Liu and J. Sun, “Parallel Graph-Cuts by Adaptive Bottom-Up Merging,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2010.
[23] W.N. Martin and J.K. Aggarwal, “Volumetric Descriptions of Objects from Multiple Views,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 5, no. 2, pp. 150-158, Mar. 1983.
[24] D. Mumford and J. Shah, “Optimal Approximations by Piecewise Smooth Functions and Associated Variational Problems,” Comm. Pure and Applied Math., vol. 42, pp. 577-685, 1989.
[25] T. Pock, D. Cremers, H. Bischof, and A. Chambolle, “An Algorithm for Minimizing the Piecewise Smooth Mumford-Shah Functional,” Proc. IEEE Int'l Conf. Computer Vision, 2009.
[26] M. Potmesil, “Generating Octree Models of 3D Objects from Their Silhouettes from a Sequence of Images,” Computer Vision, Graphics, and Image Processing, vol. 40, no. 1, pp. 1-29, 1987.
[27] C. Rother, V. Kolmogorov, and A. Blake, “GrabCut: Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Trans. Graphics, vol. 23, no. 3, pp. 309-314, 2004.
[28] C. Rother, V. Kolmogorov, T. Minka, and A. Blake, “Cosegmentation of Image Pairs by Histogram Matching—Incorporating a Global Constraint into MRFs,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 993-1000, 2006.
[29] D. Snow, P. Viola, and R. Zabih, “Exact Voxel Occupancy with Graph Cuts,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 345-353, 2000.
[30] P. Strandmark and F. Kahl, “Parallel and Distributed Graph Cuts by Dual Decomposition,” Proc. Int'l Conf. Computer Vision and Pattern Recognition, 2010.
[31] R. Szeliski, “Rapid Octree Construction from Image Sequences,” Computer Vision, Graphics, and Image Processing, vol. 58, no. 1, pp. 23-32, 1993.
[32] J.K. Udupa and P.K. Saha, “Fuzzy Connectedness and Image Segmentation,” Proc. IEEE, vol. 91, no. 10, pp. 1649-1669, Oct. 2003.
[33] M. Unger, T. Pock, D. Cremers, and H. Bischof, “TVSeg— Interactive Total Variation Based Image Segmentation,” Proc. British Machine Vision Conf., Sept. 2008.
[34] V. Vineet and P.J. Narayanan, “Cuda Cuts: Fast Graph Cuts on the Gpu,” Proc. Computer Vision and Pattern Recognition Workshop, pp. 1-8, 2008.
[35] J. Wang, P. Bhat, R.A. Colburn, M. Agrawala, and M.F. Cohen, “Interactive Video Cutout,” ACM Trans. Graphics, vol. 24, no. 3, pp. 585-594, 2005.
[36] A. Yezzi and S. Soatto, “Stereoscopic Segmentation,” Proc. Eighth IEEE Int'l Conf. Computer Vision, vol. 1, pp. 59-66, July 2001.
29 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool