This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Stereo Matching with Linear Superposition of Layers
February 2006 (vol. 28 no. 2)
pp. 290-301
In this paper, we address stereo matching in the presence of a class of non-Lambertian effects, where image formation can be modeled as the additive superposition of layers at different depths. The presence of such effects makes it impossible for traditional stereo vision algorithms to recover depths using direct color matching-based methods. We develop several techniques to estimate both depths and colors of the component layers. Depth hypotheses are enumerated in pairs, one from each layer, in a nested plane sweep. For each pair of depth hypotheses, matching is accomplished using spatial-temporal differencing. We then use graph cut optimization to solve for the depths of both layers. This is followed by an iterative color update algorithm which we proved to be convergent. Our algorithm recovers depth and color estimates for both synthetic and real image sequences.

[1] J.R. Bergen, P.J. Burt, R. Hingorani, and S. Peleg, “A Three-Frame Algorithm for Estimating Two-Component Image Motion,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 9, pp. 886-896, Sept. 1992.
[2] R.C. Bolles, H.H. Baker, and D.H. Marimont, “Epipolar-Plane Image Analysis: An Approach to Determining Structure from Motion,” Int'l J. Computer Vision, vol. 1, pp. 7-55, 1987.
[3] Y. Boykov, O. Veksler, and R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001.
[4] A.M. Bronstein, M.M. Bronstein, M. Zibulevsky, and Y.Y. Zeevi, “Blind Separation of Reflections Using Sparse ICA,” Proc. Fourth Int'l Symp. ICA and BSS, Apr. 2003.
[5] R.T. Collins, “A Space-Sweep Approach to True Multi-Image Matching,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 358-363, June 1996.
[6] T.J. Darrell and A.P. Pentland, “Cooperative Robust Estimation Using Layers of Support,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 17, no. 5, pp. 474-487, May 1995.
[7] H. Farid and E.H. Adelson, “Separating Reflections and Lighting Using Independent Components Analysis,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 262-267, June 1999.
[8] R.I. Hartley and A. Zisserman, Multiple View Geometry. Cambridge, U.K.: Cambridge Univ. Press, Sept. 2000.
[9] B.K.P. Horn, Robot Vision Cambridge, Mass.: MIT Press, 1986.
[10] M. Irani, B. Rousso, and S. Peleg, “Detecting and Tracking Multiple Moving Objects Using Temporal Integration,” Proc. European Conf. Computer Vision, pp. 282-287, May 1992.
[11] S.X. Ju, M.J. Black, and A.D. Jepson, “Skin and Bones: Multi-Layer, Locally Affine, Optical Flow, and Regularization with Transparency,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 307-314, June 1996.
[12] S.B. Kang, R. Szeliski, and J. Chai, “Handling Occlusions in Dense Multi-View Stereo,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. I, pp. 103-110, Dec. 2001.
[13] K.N. Kutulakos and S.M. Seitz, “A Theory of Shape by Space Carving,” Int'l J. Computer Vision, vol. 38, no. 3, pp. 199-218, 2000.
[14] A. Levin, A. Zomet, and Y. Weiss, “Learning to Perceive Transparency from the Statistics of Natural Scenes,” Advances in Neural Information Processing Systems, Dec. 2002.
[15] S.G. Narasimhan and S.K. Nayar, “Vision and the Atmosphere,” Int'l J. Computer Vision, vol. 48, no. 3, pp. 233-254, Aug. 2002.
[16] M. Okutomi and T. Kanade, “A Multiple Baseline Stereo,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 4, pp. 353-363, Apr. 1993.
[17] J. Rissanen, “Modeling by Shortest Data Description,” Automatica, vol. 14, no. 14, pp. 465-471, 1978.
[18] D. Scharstein and R. Szeliski, “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,” Int'l J. Computer Vision, vol. 47, no. 1, pp. 7-42, May 2002.
[19] Y.Y. Schechner, N. Kiryati, and R. Basri, “Separation of Transparent Layers Using Focus,” Proc. Int'l Conf. Computer Vision, pp. 1061-1066, Jan. 1998.
[20] Y.Y. Schechner, N. Kiryati, and J. Shamir, “Blind Recovery of Transparent and Semireflected Scenes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 38-43, June 2000.
[21] Y.Y. Schechner, J. Shamir, and N. Kiryati, “Polarization-Based Decorrelation of Transparent Layers: The Inclination Angle of an Invisible Surface,” Proc. Int'l Conf. Computer Vision, pp. 814-819, Sept. 1999.
[22] M. Shizawa, “Direct Estimation of Multiple Disparities for Transparent Multiple Surfaces in Binocular Stereo,” Proc. Int'l Conf. Computer Vision, pp. 447-454, May 1993.
[23] M. Shizawa, “Reconstruction of Multiple Overlapping Surfaces via Standard Regularization Techniques,” Proc. Int'l Pattern Recognition, vol. A, pp. 321-325, Oct. 1994.
[24] M. Shizawa and K. Mase, “Simultaneous Multiple Optical Flow Estimation,” Proc. Int'l Conf. Pattern Recognition, vol. I, pp. 274-278, June 1990.
[25] M. Shizawa and K. Mase, “Principle of Superposition: A Common Computational Framework for Analysis of Multiple Motion,” Proc. IEEE Workshop Visual Motion, pp. 164-172, Oct. 1991.
[26] M. Singh and B. Anderson, “Toward a Perceptual Theory of Transparency,” Psychological Rev., vol. 109, no. 3, pp. 492-519, July 2002.
[27] R. Swaminathan, S.B. Kang, R. Szeliski, A. Criminisi, and S.K. Nayar, “On the Motion and Appearance of Specularities in Image Sequences,” Proc. European Conf. Computer Vision, vol. 1, pp. 508-523, May/June 2002.
[28] R. Szeliski, S. Avidan, and P. Anandan, “Layer Extraction from Multiple Images Containing Reflections and Transparency,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 246-253, June 2000.
[29] R. Szeliski and P. Golland, “Stereo Matching with Transparency and Matting,” Int'l J. Computer Vision, vol. 32, no. 1, pp. 45-61, Aug. 1999.
[30] J.Y. A. Wang and E.H. Adelson, “Representing Moving Images with Layers,” IEEE Trans. Image Processing, vol. 3, no. 5, pp. 625-638, Sept. 1994.

Index Terms:
Index Terms- Stereo matching, layer extraction, reflection, transparency, plane sweep.
Citation:
Yanghai Tsin, Sing Bing Kang, Richard Szeliski, "Stereo Matching with Linear Superposition of Layers," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 2, pp. 290-301, Feb. 2006, doi:10.1109/TPAMI.2006.42
Usage of this product signifies your acceptance of the Terms of Use.