The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.12 - December (2009 vol.31)
pp: 2115-2128
Oliver Woodford , University of Oxford, Oxford
Philip Torr , Oxford Brookes University, Oxford
Ian Reid , University of Oxford, Oxford
Andrew Fitzgibbon , Microsoft Research Ltd., Cambridge
ABSTRACT
Second-order priors on the smoothness of 3D surfaces are a better model of typical scenes than first-order priors. However, stereo reconstruction using global inference algorithms, such as graph cuts, has not been able to incorporate second-order priors because the triple cliques needed to express them yield intractable (nonsubmodular) optimization problems. This paper shows that inference with triple cliques can be effectively performed. Our optimization strategy is a development of recent extensions to \alpha--expansion, based on the “ QPBO” algorithm. The strategy is to repeatedly merge proposal depth maps using a novel extension of QPBO. Proposal depth maps can come from any source, for example, frontoparallel planes as in \alpha-expansion, or indeed any existing stereo algorithm, with arbitrary parameter settings.
INDEX TERMS
Stereo, second-order prior, discrete optimization, graph cuts.
CITATION
Oliver Woodford, Philip Torr, Ian Reid, Andrew Fitzgibbon, "Global Stereo Reconstruction under Second-Order Smoothness Priors", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.31, no. 12, pp. 2115-2128, December 2009, doi:10.1109/TPAMI.2009.131
REFERENCES
[1] W.E.L. Grimson, From Images to Surfaces: A Computational Study of the Human Early Visual System. MIT Press, 1981.
[2] D. Terzopoulos, “Multilevel Computational Processes for Visual Surface Reconstruction,” Computer Vision, Graphics, and Image Processing, vol. 24, no. 1, pp. 52-96, Oct. 1983.
[3] G. Li and S.W. Zucker, “Surface Geometric Constraints for Stereo in Belief Propagation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2355-2362, 2006.
[4] O.J. Woodford, I.D. Reid, P.H.S. Torr, and A.W. Fitzgibbon, “On New View Synthesis Using Multiview Stereo,” Proc. British Machine Vision Conf., vol. 2, pp. 1120-1129, 2007.
[5] O.J. Woodford, P.H.S. Torr, I.D. Reid, and A.W. Fitzgibbon, “Global Stereo Reconstruction under Second Order Smoothness Priors,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), June 2008.
[6] D. Scharstein and R. Szeliski, “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,” Int'l J. Computer Vision, vol. 47, no. 1, pp. 7-42, 2002.
[7] B.K.P. Horn, Robot Vision. MIT Press, 1986.
[8] M.A. Gennert, “Brightness-Based Stereo Matching,” Proc. Int'l Conf. Computer Vision, pp. 139-143, 1988.
[9] D. Marr and T. Poggio, “A Computational Theory of Human Stereo Vision,” Philosophical Trans. Royal Soc. London A, vol. 204, pp. 301-328, 1979.
[10] Y. Boykov, O. Veksler, and R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[11] V. Kolmogorov and R. Zabih, “Multi-Camera Scene Reconstruction via Graph Cuts,” Proc. European Conf. Computer Vision, vol. 3, pp. 82-96, 2002.
[12] J. Sun, N.-N. Zheng, and H.-Y. Shum, “Stereo Matching Using Belief Propagation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 7, pp. 787-800, July 2003.
[13] P.F. Felzenszwalb and D.P. Huttenlocher, “Efficient Belief Propagation for Early Vision,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 261-268, 2004.
[14] C. Strecha, R. Fransens, and L. Van Gool, “Wide Baseline Stereo from Multiple Views: A Probabilistic Account,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 552-559, June 2004.
[15] Y. Wei and L. Quan, “Asymmetrical Occlusion Handling Using Graph Cut for Multi-View Stereo,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 902-909, 2005.
[16] H. Ishikawa and D. Geiger, “Occlusions, Discontinuities, and Epipolar Lines in Stereo,” Proc. European Conf. Computer Vision, pp. 232-248, 1998.
[17] V. Kolmogorov and R. Zabih, “Computing Visual Correspondence with Occlusions via Graph Cuts,” Proc. Int'l Conf. Computer Vision, pp. 508-515, 2001.
[18] S. Roy and I.J. Cox, “A Maximum-Flow Formulation of the N-Camera Stereo Correspondence Problem,” Proc. Int'l Conf. Computer Vision, pp. 492-502, 1998.
[19] O. Veksler, “Graph Cut Based Optimization for MRFs with Truncated Convex Priors,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[20] S. Baker, R. Szeliski, and P. Anandan, “A Layered Approach to Stereo Reconstruction,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 434-441, 1998.
[21] S. Birchfield and C. Tomasi, “Multiway Cut for Stereo and Motion with Slanted Surfaces,” Proc. Int'l Conf. Computer Vision, pp. 489-495, Sept. 1999.
[22] P.H.S. Torr, R. Szeliski, and P. Anandan, “An Integrated Bayesian Approach to Layer Extraction from Image Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 3, pp. 297-304, Mar. 2001.
[23] H. Tao, H.S. Sawhney, and R. Kumar, “A Global Matching Framework for Stereo Computation,” Proc. Int'l Conf. Computer Vision, pp. 532-539, 2001.
[24] M. Bleyer and M. Gelautz, “A Layered Stereo Algorithm Using Image Segmentation and Global Visibility Constraints,” Proc. Int'l Conf. Image Processing, vol. 5, pp. 2997-3000, Oct. 2004.
[25] L. Hong and G. Chen, “Segment-Based Stereo Matching Using Graph Cuts,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 74-81, 2004.
[26] A. Klaus, M. Sormann, and K. Karner, “Segment-Based Stereo Matching Using Belief Propagation and a Self-Adapting Dissimilarity Measure,” Proc. Int'l Conf. Pattern Recognition, pp. 15-18, 2006.
[27] Q. Yang, L. Wang, R. Yang, H. Stewénius, and D. Nistér, “Stereo Matching with Color-Weighted Correlation, Hierachical Belief Propagation and Occlusion Handling,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 2347-2354, 2006.
[28] M. Bleyer and M. Gelautz, “Graph-Cut-Based Stereo Matching Using Image Segmentation with Symmetrical Treatment of Occlusions,” Signal Processing: Image Comm., vol. 2, pp. 127-143, Feb. 2007.
[29] Y. Boykov and O. Veksler, “Graph Cuts in Vision and Graphics: Theories and Applications,” The Handbook of Math. Models in Computer Vision, Springer, 2006.
[30] P. Kohli, “Minimizing Dynamic and Higher Order Energy Functions Using Graph Cuts,” PhD dissertation, Oxford Brookes Univ., Nov. 2007.
[31] A. Bhusnurmath and C.J. Taylor, “Solving Stereo Matching Problems Using Interior Point Methods” Proc. Fourth Int'l Symp. 3D Data Processing, Visualization, and Transmission, pp. 321-329, June 2008.
[32] R. Szeliski, “A Multi-View Approach to Motion and Stereo,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 157-163, 1999.
[33] D. Geiger, B. Ladendorf, and A. Yuille, “Occlusions and Binocular Stereo,” Int'l J. Computer Vision, vol. 14, pp. 211-226, 1995.
[34] P.N. Belhumeur, “A Bayesian Approach to Binocular Steropsis,” Int'l J. Computer Vision, vol. 19, no. 3, pp. 237-260, Aug. 1996.
[35] J. Sun, Y. Li, S.B. Kang, and H. Shum, “Symmetric Stereo Matching for Occlusion Handling,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2005.
[36] V. Kolmogorov and C. Rother, “Comparison of Energy Minimization Algorithms for Highly Connected Graphs,” Proc. European Conf. Computer Vision, vol. 2, pp. 1-15, 2006.
[37] R. Szeliski, R. Zabih, D. Scharstein, O. Veksler, V. Kolmogorov, A. Agarwala, M. Tappen, and C. Rother, “A Comparative Study of Energy Minimization Methods for Markov Random Fields,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 1068-1080, June 2008.
[38] M. Isard, “PAMPAS: Real-Valued Graphical Models for Computer Vision,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 613-620, 2003.
[39] E.B. Sudderth, A.T. Ihler, W.T. Freeman, and A.S. Willsky, “Nonparametric Belief Propagation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 605-612, 2003.
[40] B. Potetz, “Efficient Belief Propagation for Vision Using Linear Constraint Nodes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[41] V. Kolmogorov and R. Zabih, “What Energy Functions Can be Minimized via Graph Cuts?” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 147-159, Feb. 2004.
[42] H. Ishikawa, “Exact Optimization for Markov Random Fields with Convex Priors,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 10, pp. 1333-1336, Oct. 2003.
[43] D. Schlesinger and B. Flach, “Transforming an Arbitrary Minsum Problem into a Binary One,” Technical Report TUD-FI06-01, Dresden Univ. of Tech nology, 2006.
[44] V. Lempitsky, C. Rother, and A. Blake, “LogCut-Efficient Graph Cut Optimization for Markov Random Fields,” Proc. Int'l Conf. Computer Vision, 2007.
[45] O. Woodford, I.D. Reid, P.H.S. Torr, and A.W. Fitzgibbon, “Fields of Experts for Image-Based Rendering,” Proc. British Machine Vision Conf., vol. 3, pp. 1109-1108, 2006.
[46] J. Winn and J. Shotton, “The Layout Consistent Random Field for Recognizing and Segmenting Partially Occluded Objects,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[47] C. Rother, S. Kumar, V. Kolmogorov, V. Lempitsky, and A. Blake, “Digital Tapestry,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 589-596, 2005.
[48] P.L. Hammer, P. Hansen, and B. Simeone, “Roof Duality, Complementation and Persistency in Quadratic 0-1 Optimization,” Math. Programming, vol. 28, pp. 121-155, 1984.
[49] V. Kolmogorov and C. Rother, “Minimizing Non-Submodular Functions with Graph Cuts—A Review,” Technical Report MSR-TR-2006-100, Microsoft Research, 2006.
[50] E. Boros, P.L. Hammer, and G. Tavares, “PreProcessing of Unconstrained Quadratic Binary Optimization,” Technical Report RRR 10-2006, Rutgers Center for Operations Research, Apr. 2006.
[51] C. Rother, V. Kolmogorov, V. Lempitsky, and M. Szummer, “Optimizing Binary MRFs via Extended Roof Duality,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[52] A. Raj, G. Singh, and R. Zabih, “MRF's for MRI's: Bayesian Reconstruction of MR Images via Graph Cuts,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1061-1068, 2006.
[53] O.J. Woodford, I.D. Reid, and A.W. Fitzgibbon, “Efficient New View Synthesis Using Pairwise Dictionary Priors,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[54] V. Lempitsky, S. Roth, and C. Rother, “FusionFlow: Discrete-Continuous Optimization for Optical Flow Estimation,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[55] M. Pollefeys, R. Koch, and L. Van Gool, “Self Calibration and Metric Reconstruction in spite of Varying and Unknown Internal Camera Parameters,” Proc. Int'l Conf. Computer Vision, pp. 90-96, 1998.
[56] P. Gargallo and P. Sturm, “Bayesian 3D Modeling from Images Using Multiple Depth Maps,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 885-891, June 2005.
[57] S. Birchfield and C. Tomasi, “A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 4, pp. 401-406, Apr. 1998.
[58] R. Szeliski and D. Scharstein, “Sampling the Disparity Space Image,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 3, pp. 419-425, Mar. 2004.
[59] J. Besag, “Spatial Interaction and the Statistical Analysis of Lattice Systems,” J. Royal Statistical Soc., Series B, vol. 36, no. 2, pp. 192-236, 1974.
[60] P. Fua, “A Parallel Stereo Algorithm that Produces Dense Depth Maps and Preserves Image Features,” Machine Vision and Applications, vol. 6, no. 1, pp. 35-49, 1993.
[61] A. Alvarez, R. Deriche, J. Sánchez, and J. Weickert, “Dense Disparity Map Estimation Respecting Image Discontinuities: A PDE and Scale-Space Based Approach,” J. Visual Comm. and Image Representation, vol. 13, no. 1, pp. 3-21, Mar. 2002.
[62] D. Scharstein and C. Pal, “Learning Conditional Random Fields for Stereo,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[63] E. Gamble and T. Poggio, “Visual Integration and Detection of Discontinuities: The Key Role of Intensity Edges,” Technical Report AI Memo No. 970, MIT Artificial Intelligence Lab., Oct. 1987.
[64] D. Comaniciu and P. Meer, “Mean Shift: A Robust Approach toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[65] P. Kohli, M.P. Kumar, and P.H.S. Torr, “${\cal P}^3$ & Beyond: Solving Energies with Higher Order Cliques,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[66] A. Billionnet and B. Jaumard, “A Decomposition Method for Minimizing Quadratic Pseudoboolean Functions,” Operations Research Letters, vol. 8, no. 3, pp. 161-163, June 1989.
18 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool