This Article 
 Bibliographic References 
 Add to: 
Three-Dimensional Scene Flow
March 2005 (vol. 27 no. 3)
pp. 475-480
Just as optical flow is the two-dimensional motion of points in an image, scene flow is the three-dimensional motion of points in the world. The fundamental difficulty with optical flow is that only the normal flow can be computed directly from the image measurements, without some form of smoothing or regularization. In this paper, we begin by showing that the same fundamental limitation applies to scene flow; however, many cameras are used to image the scene. There are then two choices when computing scene flow: 1) perform the regularization in the images or 2) perform the regularization on the surface of the object in the scene. In this paper, we choose to compute scene flow using regularization in the images. We describe three algorithms, the first two for computing scene flow from optical flows and the third for constraining scene structure from the inconsistencies in multiple optical flows.

[1] E. Adelson and J. Bergen, “The Plenoptic Function and the Elements of Early Vision,” Computational Models of Visual Processing, Landy and Movshon, eds., pp. 3-20, 1991.
[2] S. Avidan and A. Shashua, “Nonrigid Parallax for 3D Linear Motion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 62-66, 1998.
[3] J.L. Barron, D.J. Fleet, and S.S. Beauchemin, “Performance of Optical Flow Techniques,” Int'l J. Computer Vision, vol. 12, no. 1, pp. 43-77, 1994.
[4] M. Black and P. Anandan, “A Framework for the Robust Estimation of Optical Flow,” Proc. Fourth Int'l Conf. Computer Vision, pp. 231-236, 1993.
[5] R. Carceroni and K. Kutulakos, “Multi-View Scene Capture by Surfel Sampling: From Video Streams to Nonrigid 3D Motion, Shape and Reflectance,” Proc. IEEE Int'l Conf. Computer Vision, pp. 60-67, 2001.
[6] J.P. Costeira and T. Kanade, “A Multibody Factorization Method for Independently Moving Objects,” Int'l J. Computer Vision, vol. 29, no. 3, pp. 159-179, 1998.
[7] B.K.P. Horn, Robot Vision. McGraw Hill, 1986.
[8] T. Kanade, H. Saito, and S. Vedula, “The 3D Room: Digitizing Time-Varying 3D Events by Synchronized Multiple Video Streams,” Technical Report CMU-RI-TR-98-34, Robotics Inst., Carnegie Mellon Univ., 1998.
[9] W.-H. Liao, S.J. Aggrawal, and J.K. Aggrawal, “The Reconstruction of Dynamic 3D Structure of Biological Objects Using Stereo Microscope Images,” Machine Vision and Applications, vol. 9, pp. 166-178, 1997.
[10] B.D. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proc. Seventh Int'l Joint Conf. Artificial Intelligence, pp. 674-679, 1981.
[11] S. Malassiotis and M.G. Strintzis, “Model-Based Joint Motion and Structure Estimation from Stereo Images,” Computer Vision Image Understanding, vol. 65, no. 1, pp. 79-94, 1997.
[12] D. Metaxas and D. Terzopoulos, “Shape and Nonrigid Motion Estimation through Physics-Based Synthesis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 580-591, June 1993.
[13] S. Negahdaripour and B.K.P. Horn, “Direct Passive Navigation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, no. 1, pp. 168-176, Jan. 1987.
[14] M.A. Penna, “The Incremental Approximation of Nonrigid Motion,” Computer Vision, Graphics, and Image Processing, vol. 60, no. 2, pp. 141-156, 1994.
[15] A.P. Pentland and B. Horowitz, “Recovery of Nonrigid Motion and Structure,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 7, pp. 730-742, July 1991.
[16] S.M. Seitz and C.R. Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring,” Int'l J. Computer Vision, vol. 35, no. 2, pp. 151-173, 1999.
[17] K. Torrance and E. Sparrow, “Theory for Off-Specular Reflection from Rough Surfaces,” J. Optical Soc. of Am., vol. 57, pp. 1105-1114, Sept. 1967.
[18] S. Ullman, The Interpretation of Visual Motion. MIT Press, 1979.
[19] S. Ullman, “Maximizing the Rigidity: The Incremental Recovery of 3-D Shape and Nonrigid Motion,” Perception, vol. 13, pp. 255-274, 1984.
[20] S. Vedula, “Image Based Spatio-Temporal Modeling and View Interpolation of Dynamic Events,” PhD thesis, Robotics Inst., Carnegie Mellon Univ., 2001.
[21] S. Vedula, S. Baker, and T. Kanade, “Spatio-Temporal View Interpolation,” Proc. 13th ACM Eurographics Workshop Rendering, pp. 65-76, June 2002.
[22] S. Vedula, S. Baker, P. Rander, R. Collins, and T. Kanade, “Three-Dimensional Scene Flow,” Proc. IEEE Int'l Conf. Computer Vision, pp. 722-729, 1999.
[23] A.M. Waxman and J.H. Duncan, “Binocular Image Flows: Steps toward Stereo-Motion Fusion,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 8, no. 6, pp. 715-729, Nov. 1986.
[24] G.S. Young and R. Chellappa, “3-D Motion Estimation Using a Sequence of Noisy Stereo Images: Models, Estimation, and Uniqueness,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 8, pp. 735-759, Aug. 1999.

Index Terms:
Scene flow, three-dimensional dense nonrigid motion, optical flow, the brightness constancy constraint, normal flow, three-dimensional normal flow.
Sundar Vedula, Simon Baker, Peter Rander, Robert Collins, Takeo Kanade, "Three-Dimensional Scene Flow," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 3, pp. 475-480, March 2005, doi:10.1109/TPAMI.2005.63
Usage of this product signifies your acceptance of the Terms of Use.