This Article 
 Bibliographic References 
 Add to: 
Dynamic Refraction Stereo
August 2011 (vol. 33 no. 8)
pp. 1518-1531
Nigel J.W. Morris, University of Toronto, Toronto
Kiriakos N. Kutulakos, University of Toronto, Toronto
In this paper we consider the problem of reconstructing the 3D position and surface normal of points on an unknown, arbitrarily-shaped refractive surface. We show that two viewpoints are sufficient to solve this problem in the general case, even if the refractive index is unknown. The key requirements are 1) knowledge of a function that maps each point on the two image planes to a known 3D point that refracts to it, and 2) light is refracted only once. We apply this result to the problem of reconstructing the time-varying surface of a liquid from patterns placed below it. To do this, we introduce a novel “stereo matching” criterion called refractive disparity, appropriate for refractive scenes, and develop an optimization-based algorithm for individually reconstructing the position and normal of each point projecting to a pixel in the input views. Results on reconstructing a variety of complex, deforming liquid surfaces suggest that our technique can yield detailed reconstructions that capture the dynamic behavior of free-flowing liquids.

[1] Y. Adato, Y. Vasilyev, O. Ben-Shahar, and T. Zickler, "Toward a Theory of Shape from Specular Flow," Proc. IEEE 11th Int'l Conf. Computer Vision, pp. 1-8, 2007.
[2] S. Agarwal, S.P. Mallick, D. Kriegman, and S. Belongie, "On Refractive Optical Flow," Proc. Eighth European Conf. Computer Vision, pp. 483-494, 2004.
[3] S. Baker and I. Matthews, "Lucas-Kanade 20 Years On: A Unifying Framework," Int'l J. Computer Vision, vol. 56, no. 3, pp. 221-255, 2004.
[4] M. Ben-Ezra and S. Nayar, "What Does Motion Reveal about Transparency?" Proc. Ninth Int'l Conf. Computer Vision, pp. 1025-1032, 2003.
[5] A. Blake, "Specular Stereo," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 973-976, 1985.
[6] T. Bonfort and P. Sturm, "Voxel Carving for Specular Surfaces," Proc. Ninth Int'l Conf. Computer Vision, pp. 591-596, 2003.
[7] T. Bonfort, P. Sturm, and P. Gargallo, "General Specular Surface Triangulation," Proc. Asian Conf. Computer Vision, pp. 872-881, 2006.
[8] J.-Y. Bouguet, "MATLAB Camera Calibration Toolbox," /, 2010.
[9] T. Chen, M. Goesele, and H.-P. Seidel, "Mesostructure from Specularity," Proc. 2006 IEEE CS Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 1825-1832, 2006.
[10] Y.-Y. Chuang, D.E. Zonkger, J. Hirdorff, B. Curless, and D.H. Salesin, "Environment Matting Extensions: Toward Higher Accuracy and Real-Time Capture," Proc. ACM SIGGRAPH, pp. 121-130, 2000.
[11] J.M. Daida, D. Lund, C. Wolf, G.A. Meadows, K. Schroeder, J. Vesecky, D.R. Lyzenga, B.C. Hannan, and R.R. Bertram, "Measuring Topography of Small-Scale Water Surface Waves," Proc. Geoscience and Remote Sensing Symp. Conf., vol. 3, pp. 1881-1883, 1995.
[12] Y. Ding and J. Yu, "Recovering Shape Characteristics on Near-Flat Specular Surfaces," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[13] Y. Ding, J. Yu, and P. Sturm, "Recovering Specular Surfaces Using Curved Line Images," Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2009.
[14] D. Enright, S. Marschner, and R. Fedkiw, "Animation and Rendering of Complex Water Surfaces," Proc. ACM SIGGRAPH, pp. 736-744, 2002.
[15] P. Flach and H.-G. Maas, "Vision-Based Techniques for Refraction Analysis in Applications of Terrestrial Geodesy," Proc. Int'l Archives of Photogrammetry and Remote Sensing, pp. 195-201, 2000.
[16] Y. Francken, T. Cuypers, T. Mertens, J. Gielis, and P. Bekaert, "High Quality Mesostructure Acquisition Using Specularities," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-7, June 2008.
[17] C.B. Moler, G.E. Forsyth, and M.A. Malcom, Computer Methods for Mathematical Computations. Prentice Hall, 1976.
[18] A.S. Glassner, Principles of Digital Image Synthesis. Morgan Kaufmann, 1995.
[19] M.A. Halstead, B.A. Barsky, S.A. Klein, and R.B. Mandell, "Reconstructing Curved Surfaces from Specular Reflection Patterns Using Spline Surface Fitting of Normals," Proc. ACM SIGGRAPH, pp. 335-342, 1996.
[20] F.H. Harlow and J.E. Welch, "Numerical Calculation of Time-Dependent Viscous Incompressible Flow," Physics of Fluids, vol. 8, pp. 2182-2189, 1965.
[21] C. Harris and M. Stephens, "A Combined Edge and Corner Detector," Proc. Fourth Alvey Vision Conf., pp. 189-192, 1988.
[22] R.I. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2000.
[23] R.I. Hartley and P. Sturm, "Triangulation," Computer Vision and Image Understanding, vol. 68, no. 2, pp. 146-157, 1997.
[24] J. Höhle, "Reconstruction of the Underwater Object," Proc. Photogrammetric Eng., pp. 948-954, 1971.
[25] M.B. Hullin, M. Fuchs, I. Ihrke, H.-P. Seidel, and H.P.A. Lensch, "Fluorescent Immersion Range Scanning," ACM Trans. Graphics, vol. 27, no. 3, pp. 87-1-87-10, Aug. 2008.
[26] K. Ikeuchi, "Determining the Surface Orientations of Specular Surfaces by Using the Photometric Stereo Method," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 3, no. 6, pp. 661-669, Nov. 1981.
[27] K. Ikeuchi and B.K.P. Horn, "Numerical Shape from Shading and Occluding Boundaries," Artificial Intelligence, vol. 17, pp. 141-184, 1981.
[28] B. Jähne, J. Klinke, P. Geissler, and F. Hering, "Image Sequence Analysis of Ocean Wind Waves," Proc. Int'l Seminar on Imaging in Transport Processes, 1992.
[29] B. Jähne, J. Klinke, and S. Waas, "Imaging of Short Ocean Wind Waves: A Critical Theoretical Review," J. Optical Soc. Am. A, vol. 11, no. 8, pp. 2197-2209, 1994.
[30] B. Jähne, M. Schmidt, and R. Rocholz, "Combined Optical Slope/Height Measurements of Short Wind Waves: Principle and Calibration," Measurement Science and Technology, vol. 16, no. 10, pp. 1937-1944, 2005.
[31] J. Kaminski, S. Lowitzsch, M.C. Knauer, and G. Häusler, "Full-Field Shape Measurement of Specular Surfaces," Proc. Fifth Int'l Workshop Automatic Processing of Fringe Patterns, 2005.
[32] W.C. Keller and B.L. Gotwols, "Two-Dimensional Optical Measurement of Wave Slope," Applied Optics, vol. 22, no. 22, pp. 3476-3478, 1983.
[33] J.J. Koenderink and A.J. van Doorn, "The Structure of Two-Dimensional Scalar Fields with Applications to Vision," Biological Cybernetics, vol. 33, pp. 151-158, 1979.
[34] K. Kolev, M. Klodt, T. Brox, and D. Cremers, "Continuous Global Optimization in Multiview 3d Reconstruction," Int'l J. Computer Vision, vol. 84, no. 1, pp. 80-96, 2009.
[35] K.N. Kutulakos and E. Steger, "A Theory of and Refractive and Specular 3d Shape by Light-Path Triangulation," Proc. 10th Int'l Conf. Computer Vision, pp. 1448-1455, 2005.
[36] F. Losasso, F. Gibou, and R. Fedkiw, "Simulating Water and Smoke with an Octree Data Structure," ACM Trans. Graphics, vol. 23, no. 3, pp. 457-462, 2004.
[37] H.-G. Maas, "New Developments in Multimedia Photogrammetry," Optical 3D Measurement Techniques III. Wichmann Verlag, 1995.
[38] N. Morris and K.N. Kutulakos, "Dynamic Refraction Stereo," Proc. 10th Int'l Conf. Computer Vision, pp. 1573-1580, 2005.
[39] M. Müller, D. Charypar, and M. Gross, "Particle-Based Fluid Simulation for Interactive Applications," Proc. 2003 ACM SIGGRAPH/Eurographics Symp. Computer Animation, pp. 154-159, 2003.
[40] H. Murase, "Surface Shape Reconstruction of an Undulating Transparent Object," Proc. Third Int'l Conf. Computer Vision, pp. 313-317, 1990.
[41] D. Nehab, S. Rusinkiewicz, J. Davis, and R. Ramamoorthi, "Efficiently Combining Positions and Normals for Precise 3D Geometry," Proc. ACM SIGGRAPH, pp. 536-543, 2005.
[42] A. Okamoto, "Orientation Problem of Two-Media Photographs with Curved Boundary Surfaces," Proc. Photogrammetric Eng. and Remote Sensing, pp. 303-316, 1984.
[43] M. Oren and S.K. Nayar, "A Theory of Specular Surface Geometry," Proc. Fifth Int'l Conf. Computer Vision, pp. 740-747, 1995.
[44] W.H. Press, B.P. Flannery, S.A. Teukolsky, and W.T. Vetterling, Numerical Recipies in C. Cambridge Univ. Press, 1988.
[45] A.C. Sanderson, L.E. Weiss, and S.K. Nayar, "Structured Highlight Inspection of Specular Surfaces," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 10, no. 1, pp. 44-55, Jan. 1988.
[46] S. Savarese and P. Perona, "Local Analysis for 3D Reconstruction of Specular Surfaces—Part II," Proc. Seventh European Conf. Computer Vision, pp. 759-774, 2002.
[47] H. Schultz, "Retrieving Shape Information from Multiple Images of a Specular Surface," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 2, pp. 195-201, Feb. 1994.
[48] O.H. Shemdin, "Measurement of Short Surface Waves with Stereophotography," Proc. Eng. in the Ocean Environment Conf., pp. 568-571, 1990.
[49] S.A. Sullivan, "Experimental Study of the Absorption in Distilled Water, Artificial Sea Water, and Heavy Water in the Visible Region of the Spectrum," J. Optical Soc. Am., vol. 53, pp. 962-968, 1963.
[50] M. Tarini, H.P.A. Lensch, M. Goesele, and H.-P. Seidel, "3D Acquisition of Mirroring Objects," Technical Report MPI-I-2003-4-001, Max Planck Institut für Informatik, 2003.
[51] T. Treibitz, Y.Y. Schechner, and H. Singh, "Flat Refractive Geometry," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[52] B. Trifonov, D. Bradley, and W. Heidrich, "Tomographic Reconstruction of Transparent Objects," Proc. 17th Eurographics Symp. Rendering, pp. 51-60, 2006.
[53] B. Triggs, P.F. McLauchlan, R.I. Hartley, and A.W. Fitzgibbon, "Bundle Adjustment—A Modern Synthesis," Proc. Int'l Workshop Vision Algorithms, pp. 298-372, 2000.
[54] Y. Vasilyev, Y. Adato, T. Zickler, and O. Ben-Shahar, "Dense Specular Shape from Multiple Specular Flows," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[55] J. Wang and K.J. Dana, "Relief Texture from Specularities," IEEE Trans. Pattern Analysis Machine Intelligence, vol. 28, no. 3, pp. 446-457, Mar. 2006.
[56] Z. Wu and G.A. Meadows, "2-D Surface Reconstruction of Water Waves," Proc. Eng. in the Ocean Environment. Conf., pp. 416-421, 1990.
[57] L. Zhang, N. Snavely, B. Curless, and S.M. Seitz, "Spacetime Faces: High Resolution Capture for Modeling and Animation," ACM Trans. Graphics, vol. 23, no. 3, pp. 548-558, 2004.
[58] X. Zhang and C.S. Cox, "Measuring the Two-Dimensional Structure of a Wavy Water Surface Optically: A Surface Gradient Detector," Experiments in Fluids, vol. 17, pp. 225-237, 1994.
[59] L. Zhou, C. Kambhamettu, and D.B. Goldgof, "Fluid Structure and Motion Analysis from Multi-Spectrum 2D Cloud Image Sequences," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 744-751, 2000.

Index Terms:
Stereo, time-varying imagery, shape-from-X, transparency, refractive index estimation.
Nigel J.W. Morris, Kiriakos N. Kutulakos, "Dynamic Refraction Stereo," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 8, pp. 1518-1531, Aug. 2011, doi:10.1109/TPAMI.2011.24
Usage of this product signifies your acceptance of the Terms of Use.