This Article 
 Bibliographic References 
 Add to: 
A Quasi-Dense Approach to Surface Reconstruction from Uncalibrated Images
March 2005 (vol. 27 no. 3)
pp. 418-433
This paper proposes a quasi-dense approach to 3D surface model acquisition from uncalibrated images. First, correspondence information and geometry are computed based on new quasi-dense point features that are resampled subpixel points from a disparity map. The quasi-dense approach gives more robust and accurate geometry estimations than the standard sparse approach. The robustness is measured as the success rate of full automatic geometry estimation with all involved parameters fixed. The accuracy is measured by a fast gauge-free uncertainty estimation algorithm. The quasi-dense approach also works for more largely separated images than the sparse approach, therefore, it requires fewer images for modeling. More importantly, the quasi-dense approach delivers a high density of reconstructed 3D points on which a surface representation can be reconstructed. This fills the gap of insufficiency of the sparse approach for surface reconstruction, essential for modeling and visualization applications. Second, surface reconstruction methods from the given quasi-dense geometry are also developed. The algorithm optimizes new unified functionals integrating both 3D quasi-dense points and 2D image information, including silhouettes. Combining both 3D data and 2D images is more robust than the existing methods using only 2D information or only 3D data. An efficient bounded regularization method is proposed to implement the surface evolution by level-set methods. Its properties are discussed and proven for some cases. As a whole, a complete automatic and practical system of 3D modeling from raw images captured by hand-held cameras to surface representation is proposed. Extensive experiments demonstrate the superior performance of the quasi-dense approach with respect to the standard sparse approach in robustness, accuracy, and applicability.

[1] Boujou, 2d3 Ltd, http:/, 2000.
[2] P. Beardsley, P. Torr, and A. Zisserman, “3D Model Acquisition from Extended Image Sequences,” Proc. Fourth European Conf. Computer Vision, B. Buxton and R. Cipolla, eds., pp. 683-695, Apr. 1996.
[3] A.F. Bobick and S.S. Intille, “Large Occlusion Stereo,” Int'l J. Computer Vision, vol. 33, no. 3, pp. 181-200, 1999.
[4] D.C. Brown, “The Bundle Adjustment— Progress and Prospects,” Int'l Archive of Photogrammetry, vol. 21, 1976.
[5] V. Caselles and R. Kimmel, “Minimal Surfaces Based Object Segmentation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 4, pp. 394-398, Apr. 1997.
[6] V. Caselles, R. Kimmel, and G. Sapiro, “Geodesic Active Contours,” Int'l J. Computer Vision, vol. 22, no. 1, pp. 61-79, 1997.
[7] B. Curless and M. Levoy, “A Volumetric Method for Building Complex Models from Range Images,” Proc. SIGGRAPH '96, pp. 303-312, 1996.
[8] F. Devernay and O.D. Faugeras, “Automatic Calibration and Removal of Distortion from Scenes of Structured Environments,” Proc. Int'l Soc. for Optical Engineering Conf. Investigate and Trial Image Processing, vol. 2567, July 1995.
[9] O. Faugeras and R. Keriven, “Complete Dense Stereovision Using Level Set Methods,” Proc. Fifth European Conf. Computer Vision, pp. 379-393, 1998.
[10] O. Faugeras and Q.T. Luong, The Geometry of Multiple Images. MIT Press, 2001.
[11] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Graphics and Image Processing, vol. 24, no. 6, pp. 381-395, June 1981.
[12] A.W. Fitzgibbon and A. Zisserman, “Automatic Camera Recovery for Closed or Open Image Sequences,” Proc. European Conf. Computer Vision, pp. 311-326, June 1998.
[13] P. Fua, “Parametric Models are Versatile: The Case of Model Based Optimization,” Proc. Int'l Soc. Photogrammetry and Remote Sensing Joint Workshop, Sept. 1995.
[14] P. Fua, “From Multiple Stereo Views to Multiple 3D Surfaces,” Int'l J. Computer Vision, vol. 24, no. 1, pp. 19-35, 1997.
[15] S. Gibson, J. Cook, T. Howard, R. Hubbold, and D. Oram, “Accurate Camera Calibration for Off-Line, Video-Based Augmented Reality,” IEEE and ACM Int'l Symp. Mixed and Augmented Reality, pp. 37-46, 2002.
[16] J. Gomes and O. Faugeras, “Reconciling Distance Functions and Level Sets,” J. Visual Comm. and Image Representation, vol. 11, pp. 209-223, 2000.
[17] B. Guenter, C. Crimm, D. Wood, and H. Malvar, “Making Faces,” Proc. SIGGRAPH '98, pp. 55-66, 1998.
[18] C. Harris and M. Stephens, “A Combined Corner and Edge Detector,” Alvey Vision Conf., pp. 147-151, 1988.
[19] R.I. Hartley, “Euclidean Reconstruction from Uncalibrated Views,” Proc. DARPA–ESPRIT Workshop Applications of Invariants in Computer Vision, pp. 187-202, Oct. 1993.
[20] R.I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, June 2000.
[21] A. Heyden, “Geometry and Algebra of Multiple Projective Transformations,” PhD thesis, Lund Inst. of Tech nology, 1995.
[22] H. Hoppe, T. Derose, T. Duchamp, J. McDonalt, and W. Stuetzle, “Surface Reconstruction from Unorganized Points,” Computer Graphics, vol. 26, pp. 71-77, 1992.
[23] S. Kichenassamy, A. Kumar, P. Olver, and A. Tannenbaum, “Gradient Flows and Geometric Active Contour Models,” Proc. Fifth Int'l Conf. Computer Vision, June 1995.
[24] V. Kolmogorov and R. Zabih, “Multi-Camera Scene Reconstruction via Graph Cuts,” Proc. Seventh European Conf. Computer Vision, 2002.
[25] K.N. Kutulakos and S.M. Seitz, “A Theory of Shape by Space Carving,” Proc. Seventh Int'l Conf. Computer Vision, vol. 1, pp. 307-314, 1999.
[26] S. Laveau, “Géométrie d'un Système de $N$ Caméras. Théorie, Estimation, et Applications,” PhD thesis, École Polytechnique, May 1996.
[27] M. Lhuillier and L. Quan, “Match Propagation for Image-Based Modeling and Rendering,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 8, pp. 1140-1146, 2002.
[28] M. Lhuillier and L. Quan, “Quasi-Dense Reconstruction from Image Sequence,” Proc. Seventh European Conf. Computer Vision, vol. 2, pp. 125-139, Aug. 2002.
[29] M. Lhuillier and L. Quan, “Surface Reconstruction by Integrating 3D and 2D Data of Multiple Views,” Proc. Ninth Int'l Conf. Computer Vision, 2003.
[30] B.D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proc. Seventh Int'l Joint Conf. Artificial Intelligence, 1981.
[31] R. Malladi, J.A. Sethian, and B.C. Vemuri, “Shape Modeling with Front Propagation: A Level Set Approach,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 2, pp. 158-175, Feb. 1995.
[32] J. Matas, O. Chum, M. Urban, and T. Padjdla, “Robust Wide Baseline Stereo from Maximally Stable Extremal Regions,” Proc. 13th British Machine Vision Conf., Univ. of Cardiff, Sept. 2002.
[33] P.F. McLauchlan, “Gauge Independence in Optimization Algorithms for 3D Vision,” Proc. Vision Algorithms Workshop, 2000.
[34] D.D. Morris, Gauge Freedoms and Uncertainty Modeling for 3D Computer Vision, PhD thesis, The Robotics Inst., Carnegie Mellon Univ., Mar. 2001.
[35] D.D. Morris, K. Kanatani, and T. Kanade, “Uncertainty Modeling for Optimal Structure from Motion,” Proc. Int'l Workshop Vision Algorithms: Theory and Practice, held during the IEEE Int'l Conf. on Computer Vision (ICCV '99), Sept. 1999.
[36] P.J. Narayanan, P.W. Rander, and T. Kanade, “Constructing Virtual Worlds Using Dense Stereo,” Proc. Fifth European Conf. Computer Vision, pp. 3-10, 1998.
[37] D. Nister, “Reconstruction from Uncalibrated Sequences with a Hierarchy of Trifocal Tensors,” Proc. Sixth European Conf. Computer Vision, pp. 649-663, 2000.
[38] D. Nister, “Automatic Dense Reconstruction from Uncalibrated Video Sequences,” PhD thesis, Dept. of Numerical Analysis and Computer Science, Royal Inst. of Tech nology, Mar. 2001.
[39] S. Osher and J.A. Sethian, “Fronts Propagating with Curvature-Dependent Speed: Algorithms Based on Hamilton-Jacobi Formulations,” J. Computational Physics, vol. 79, pp. 12-49, 1988.
[40] M. Pollefeys, R. Koch, and L. Van Gool, “Self-Calibration and Metric Reconstruction in Spite of Varying and Unknown Internal Camera Parameters,” Proc. Sixth Int'l Conf. Computer Vision, pp. 90-95, Jan. 1998
[41] M. Pollefeys, R. Koch, M. Vergauwen, and L. Van Gool, “Metric 3D Surface Reconstruction from Uncalibrated Image Sequences,” Proc. European Workshop 3D Structure from Multiple Images of Large-Scale Environments, R. Koch and L. Van Gool, eds., pp. 139-154, 1998.
[42] M. Pollefeys, F. Verbiest, and L. Van Gool, “Surviving Dominant Planes in Uncalibrated Structure and Motion Recovery,” Proc. Seventh European Conf. Computer Vision, May 2002.
[43] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C— The Art of Scientific Computing, second ed., Cambridge Univ. Press 1992
[44] P. Pritchett and A. Zisserman, “Wide Baseline Stereo Matching,” Proc. Sixth Int'l Conf. Computer Vision, pp. 754-760, 1998.
[45] L. Quan, “Invariants of Six Points and Points Projective Reconstruction from Three Uncalibrated Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 1, pp. 34-46, Jan. 1995.
[46] S.M. Seitz and C.R. Dyer, “Photo-Realistic Scene Reconstruction by Voxel Coloring,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 1067-1073, June 1997.
[47] J.A. Sethian, Level Set Methods and Fast Marching Methods. The CUP Press, 1999.
[48] Manual of Photogrammetry, fourth ed., C.C. Slama, ed., Am. Soc. Photogrammetry and Remote Sensing, 1980.
[49] C. Strecha, T. Tuytelaars, and L. Van Gool, “Dense Matching of Multiple Wide-Baseline Views,” Proc. Int'l Conf. Computer Vision, pp. 1194-1201, 2003.
[50] R. Szeliski, “Rapid Octree Construction from Image Sequences,” Computer Vision, Graphics, and Image Processing, vol. 58, no. 1, pp. 23-32, July 1993.
[51] R. Szeliski and R. Weiss, “Robust Shape Recovery from Occluding Contours Using a Linear Smoother,” technical report, Digital Equipment Corporation, Cambridge Research Lab, Dec. 1993.
[52] C.K. Tang and G. Medioni, “Curvature-Augmented Tensor Voting for Shape Inference from Noisy 3D Data,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 858-864, June 2002.
[53] B. Tell and S. Carlsson, “Wide Baseline Point Matching Using Affine Invariants Computed from Intensity Profiles,” Proc. Sixth European Conf. Computer Vision, pp. 814-828, 2000.
[54] P.H.S. Torr and D.W. Murray, “The Development and Comparison of Robust Methods for Estimating the Fundamental Matrix,” Int'l J. Computer Vision, vol. 24, no. 3, pp. 271-300, 1997.
[55] P.H.S. Torr and A. Zisserman, “Robust Parameterization and Computation of the Trifocal Tensor,” Image and Vision Computing, vol. 15, pp. 591-607, 1997.
[56] P.H.S. Torr and A. Criminisi, “Dense Stereo Using Pivoted Dynamic Programming,” Proc. British Machine Vision Conf., pp. 414-423, 2002.
[57] B. Triggs, “Auto-Calibration and the Absolute Quadric,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 609-614, June 1997.
[58] B. Triggs, P.F. McLauchlan, R.I. Hartley, and A. Fitzgibbon, “Bundle Adjustment— A Modern Synthesis,” Vision Algorithms: Theory and Practice, B. Triggs, A. Zisserman, R. Szeliski, eds., pp. 298-372, Springer-Verlag, 2000.
[59] T. Tuytelaars and L. Van Gool, “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions,” Proc. 11th British Machine Vision Conf., Sept. 2000.
[60] R. Whitaker, “A Level-Set Approach to 3D Reconstruction from Range Data,” Int'l J. Computer Vision, vol. 29, no. 3, pp. 203-231, 1998.
[61] Z. Zhang and Y. Shan, “A Progressive Scheme for Stereo Matching,” Proc. Second European Workshop 3D Structure from Multiple Images of Large-Scale Environments, pp. 68-85, 2000.
[62] Z. Zhang, Z. Liu, D. Adler, and M.F. Cohen, “Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach,” Technical Report MSR-TR-2001-101, Microsoft Research, 2001.
[63] Z. Zhang, R. Deriche, O. Faugeras, and Q.T. Luong, “A Robust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry,” Artificial Intelligence, vol. 78, pp. 87-119, 1995.
[64] H.K. Zhao, S. Osher, B. Merrigan, and M. Kang, “Implicit and Non-Parametric Shape Reconstruction from Unorganized Points Using Variational Level Set Method,” Computer Vision and Image Understanding, vol. 80, pp. 295-319, 2000.
[65] H.K. Zhao, S. Osher, and R. Fedkiw, “Fast Surface Reconstruction Using the Level Set Method,” IEEE Workshop Variational and Level Set Methods in Computer Vision, July 2001.

Index Terms:
Three-dimensional reconstruction, surface reconstruction, structure from motion, 3D modeling, matching, uncertainty, variational calculus, level-set method.
Maxime Lhuillier, Long Quan, "A Quasi-Dense Approach to Surface Reconstruction from Uncalibrated Images," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 3, pp. 418-433, March 2005, doi:10.1109/TPAMI.2005.44
Usage of this product signifies your acceptance of the Terms of Use.