This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains
June 2011 (vol. 33 no. 6)
pp. 1161-1174
Daniel Cremers, Technical University of Munich, München
Kalin Kolev, Technical University of Munich, München
We propose a convex formulation for silhouette and stereo fusion in 3D reconstruction from multiple images. The key idea is to show that the reconstruction problem can be cast as one of minimizing a convex functional, where the exact silhouette consistency is imposed as convex constraints that restrict the domain of feasible functions. As a consequence, we can retain the original stereo-weighted surface area as a cost functional without heuristic modifications of this energy by balloon terms or other strategies, yet still obtain meaningful (nonempty) reconstructions which are guaranteed to be silhouette-consistent. We prove that the proposed convex relaxation approach provides solutions that lie within a bound of the optimal solution. Compared to existing alternatives, the proposed method does not depend on initialization and leads to a simpler and more robust numerical scheme for imposing silhouette consistency obtained by projection onto convex sets. We show that this projection can be solved exactly using an efficient algorithm. We propose a parallel implementation of the resulting convex optimization problem on a graphics card. Given a photoconsistency map and a set of image silhouettes, we are able to compute highly accurate and silhouette-consistent reconstructions for challenging real-world data sets. In particular, experimental results demonstrate that the proposed silhouette constraints help to preserve fine-scale details of the reconstructed shape. Computation times depend on the resolution of the input imagery and vary between a few seconds and a couple of minutes for all experiments in this paper.

[1] http://phototour.cs.washington.edubundler /, 2010.
[2] B. Baumgart, “Geometric Modeling for Computer Vision,” PhD thesis, Dept. of Computer Science, Stanford Univ., 1974.
[3] Y. Boykov and V. Lempitsky, “From Photohulls to Photoflux Optimization,” Proc. British Machine Vision Conf., vol. 3, pp. 1149-1158, 2006.
[4] J.P. Boyle and R.L. Dykstra, “A Method for Finding Projections onto the Intersection of Convex Sets in Hilbert Spaces,” Lecture Notes in Statistics, pp. 28-47, Springer, 1986.
[5] R. Cipolla and P. Giblin, Visual Motion of Curves and Surfaces. Cambridge Univ. Press, 2000.
[6] G. Cross and A. Zisserman, “Surface Reconstruction from Multiple Views Using Apparent Contours and Surface Texture,” Confluence of Computer Vision and Computer Graphics, pp. 25-47, Kluwer Academic Publishers, 2000.
[7] Y. Duan, L. Yang, H. Qin, and D. Samaras, “Shape Reconstruction from 3D and 2D Data Using PDE-Based Deformable Surfaces,” Proc. European Conf. Computer Vision, pp. 238-251, 2004.
[8] R.L. Dykstra, “An Algorithm for Restricted Least-Squares Regression,” J. Am. Statistical Assoc., vol. 78, pp. 837-842, 1983.
[9] C.H. Esteban and F. Schmitt, “Silhouette and Stereo Fusion for 3D Object Modeling,” Computer Vision and Image Understanding, vol. 96, no. 3, pp. 367-392, 2004.
[10] O. Faugeras and R. Keriven, “Variational Principles, Surface Evolution, PDE's, Level Set Methods, and the Stereo Problem,” IEEE Trans. Image Processing, vol. 7, no. 3, pp. 336-344, Mar. 1998.
[11] J.S. Franco and E. Boyer, “Exact Polyhedral Visual Hulls,” Proc. 14th British Machine Vision Conf., pp. 329-338, Sept. 2003.
[12] Y. Furukawa and J. Ponce, “Carved Visual Hulls for Image-Based Modeling,” Proc. European Conf. Computer Vision, pp. 564-577, 2006.
[13] P. Gargallo, E. Prados, and P. Sturm, “Minimizing the Reprojection Error in Surface Reconstruction from Images,” Proc. IEEE Int'l Conf. Computer Vision, 2007.
[14] C. Hernández, G. Vogiatzis, and R. Cipolla, “Probabilistic Visibility for Multi-View Stereo,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2007.
[15] J. Isidoro and S. Sclaroff, “Stochastic Refinement of the Visual Hull to Satisfy Photometric and Silhouette Consistency Constraints,” Proc. IEEE Int'l Conf. Computer Vision, pp. 1335-1342, 2003.
[16] K. Kolev, T. Brox, and D. Cremers, “Robust Variational Segmentation of 3D Objects from Multiple Views,” Pattern Recognition, K. Franke, K.R. Müller, B. Nickolay, and R. Schäfer, eds., pp. 688-697, Springer-Verlag, Sept. 2006.
[17] K. Kolev and D. Cremers, “Integration of Multiview Stereo and Silhouettes via Convex Functionals on Convex Domains,” Proc. European Conf. Computer Vision, Oct. 2008.
[18] K. Kolev, M. Klodt, T. Brox, and D. Cremers, “Propagated Photoconsistency and Convexity in Variational Multiview 3D Reconstruction,” Proc. Workshop Photometric Analysis for Computer Vision, Oct. 2007.
[19] A. Laurentini, “The Visual Hull Concept for Visual-Based Image Understanding,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 2, pp. 150-162, Feb. 1994.
[20] V. Lempitsky, Y. Boykov, and D. Ivanov, “Oriented Visibility for Multiview Reconstruction,” Proc. European Conf. Computer Vision, pp. 226-238, 2006.
[21] W.N. Martin and J.K. Aggarwal, “Volumetric Descriptions of Objects from Multiple Views,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 5, no. 2, pp. 150-158, Mar. 1983.
[22] Y. Matsumoto, K. Fujimura, and T. Kitamura, “Shape-from-Silhouette/Stereo and Its Application to 3D Digitizer,” Discrete Geometry for Computing Imagery, pp. 177-190, Springer, 1999.
[23] T. Pock, D. Cremers, H. Bischof, and A. Chambolle, “An Algorithm for Minimizing the Piecewise Smooth Mumford-Shah Functional,” Proc. IEEE Int'l Conf. Computer Vision, 2009.
[24] S. Seitz, B. Curless, J. Diebel, D. Scharstein, and R. Szeliski, “A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms,” Proc. IEEE CS Int'l Conf. Computer Vision and Pattern Recognition, pp. 519-528, 2006.
[25] S. Seitz and C. Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring,” Proc. Int'l Conf. Computer Vision and Pattern Recognition, pp. 1067-1073, June 1997.
[26] S. Sinha, P. Mordohai, and M. Pollefeys, “Multiview Stereo via Graph Cuts on the Dual of an Adaptive Tetrahedral Mesh,” Proc. IEEE Int'l Conf. Computer Vision, Oct. 2007.
[27] S. Sinha and M. Pollefeys, “Multi-View Reconstruction Using Photo-Consistency and Exact Silhouette Constraints: A Maximum-Flow Formulation,” Proc. IEEE Int'l Conf. Computer Vision, pp. 349-356, 2005.
[28] N. Snavely, S.M. Seitz, and R. Szeliski, “Modeling the World from Internet Photo Collections,” Int'l. J. Computer Vision, vol. 80, no. 2, pp. 189-210, 2008.
[29] D. Snow, P. Viola, and R. Zabih, “Exact Voxel Occupancy with Graph Cuts,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 345-353, 2000.
[30] S. Tran and L. Davis, “3D Surface Reconstruction Using Graph Cuts with Surface Constraints,” Proc. European Conf. Computer Vision, pp. 219-231, 2006.
[31] C.R. Vogel and M.E. Oman, “Iterative Methods for Total Variation Denoising,” SIAM J. Scientific Computing, vol. 17, pp. 227-238, 1996.
[32] G. Vogiatzis, P. Torr, and R. Cippola, “Multi-View Stereo via Volumetric Graph-Cuts,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 391-399, 2005.
[33] A. Yezzi and S. Soatto, “Stereoscopic Segmentation,” Int'l J. Computer Vision, vol. 53, no. 1, pp. 31-43, 2003.

Index Terms:
Image-based modeling, silhouette and stereo fusion, convex optimization.
Citation:
Daniel Cremers, Kalin Kolev, "Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 6, pp. 1161-1174, June 2011, doi:10.1109/TPAMI.2010.174
Usage of this product signifies your acceptance of the Terms of Use.