This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation
March 2011 (vol. 33 no. 3)
pp. 500-513
Thomas Brox, University of California at Berkeley, Berkeley
Jitendra Malik, University of California at Berkeley, Berkeley
Optical flow estimation is classically marked by the requirement of dense sampling in time. While coarse-to-fine warping schemes have somehow relaxed this constraint, there is an inherent dependency between the scale of structures and the velocity that can be estimated. This particularly renders the estimation of detailed human motion problematic, as small body parts can move very fast. In this paper, we present a way to approach this problem by integrating rich descriptors into the variational optical flow setting. This way we can estimate a dense optical flow field with almost the same high accuracy as known from variational optical flow, while reaching out to new domains of motion analysis where the requirement of dense sampling in time is no longer satisfied.

[1] L. Alvarez, J. Weickert, and J. Sánchez, "Reliable Estimation of Dense Optical Flow Fields with Large Displacements," Int'l J. Computer Vision, vol. 39, no. 1, pp. 41-56, Aug. 2000.
[2] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, "From Contours to Regions: An Empirical Evaluation," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2009.
[3] S. Baker, D. Scharstein, J.P. Lewis, S. Roth, M.J. Black, and R. Szeliski, "A Database and Evaluation Methodology for Optical Flow," Proc. IEEE Int'l Conf. Computer Vision, 2007.
[4] A. Berg, T. Berg, and J. Malik, "Shape Matching and Object Recognition Using Low Distortion Correspondence," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2005.
[5] A. Berg and J. Malik, "Geometric Blur for Template Matching," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 607-614, 2001.
[6] M.J. Black and P. Anandan, "Robust Dynamic Motion Estimation over Time," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 292-302, June 1991.
[7] M.J. Black and P. Anandan, "The Robust Estimation of Multiple Motions: Parametric and Piecewise Smooth Flow Fields," Computer Vision and Image Understanding, vol. 63, no. 1, pp. 75-104, Jan. 1996.
[8] A. Blake and A. Zisserman, Visual Reconstruction. MIT Press, 1987.
[9] T. Brox, "From Pixels to Regions: Partial Differential Equations in Image Analysis," PhD thesis, Faculty of Math. and Computer Science, Saarland Univ., Apr. 2005.
[10] T. Brox, C. Bregler, and J. Malik, "Large Displacement Optical Flow," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2009.
[11] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert, "High Accuracy Optical Flow Estimation Based on a Theory for Warping," Proc. Eighth European Conf. Computer Vision, T. Pajdla and J. Matas, eds., pp. 25-36, May 2004.
[12] A. Bruhn and J. Weickert, "Towards Ultimate Motion Estimation: Combining Highest Accuracy with Real-Time Performance," Proc. 10th IEEE Int'l Conf. Computer Vision, pp. 749-755, Oct. 2005.
[13] H. Chui and A. Rangarajan, "A New Point Matching Algorithm for Non-Rigid Registration," Computer Vision and Image Understanding, vol. 89, no. 23, pp. 114-141, 2003.
[14] I. Cohen, "Nonlinear Variational Method for Optical Flow Computation," Proc. Eighth Scandinavian Conf. Image Analysis, vol. 1, pp. 523-530, May 1993.
[15] N. Dalal and B. Triggs, "Histograms of Oriented Gradients for Human Detection," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 886-893, 2005.
[16] Nearest Neighbor Methods in Learning and Vision: Theory and Practice, T. Darrell, P. Indyk, and G. Shakhnarovich, eds. MIT Press, 2006.
[17] P. Heas, E. Mémin, N. Papadakis, and A. Szantai, "Layered Estmation of Atmospheric Mesoscale Dynamics from Satellite Imagery," IEEE Trans. Geoscience and Remote Sensing, vol. 45, no. 12, pp. 4087-4104, Dec. 2007.
[18] B. Horn and B. Schunck, "Determining Optical Flow," Artificial Intelligence, vol. 17, pp. 185-203, 1981.
[19] I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning Realistic Human Actions from Movies," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2008.
[20] C. Lei and Y.-H. Yang, "Optical Flow Estimation on Coarse-to-Fine Region-Trees Using Discrete Optimization," Proc. IEEE Int'l Conf. Computer Vision, 2009.
[21] V. Lempitsky, S. Roth, and C. Rother, "Discrete-Continuous Optimization for Optical Flow Estimation," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2008.
[22] C. Liu, J. Yuen, and A. Torralba, "Nonparametric Scene Parsing: Label Transfer via Dense Scene Alignment," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2009.
[23] C. Liu, J. Yuen, A. Torralba, J. Sivic, and W.T. Freeman, "SIFT Flow: Dense Correspondence across Different Scenes," Proc. European Conf. Computer Vision, pp. 28-42, 2008.
[24] D. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[25] B. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision," Proc. Seventh Int'l Joint Conf. Artificial Intelligence, pp. 674-679, Aug. 1981.
[26] E. Mémin and P. Pérez, "Dense Estimation and Object-Based Segmentation of the Optical Flow with Robust Techniques," IEEE Trans. Image Processing, vol. 7, no. 5, pp. 703-719, May 1998.
[27] Y. Mileva, A. Bruhn, and J. Weickert, "Illumination-Invariant Variational Optical Flow with Photometric Invariants," Proc. Ann. Symp. German Assoc. Pattern Recognition, pp. 152-162, 2007.
[28] N. Papenberg, A. Bruhn, T. Brox, S. Didas, and J. Weickert, "Highly Accurate Optic Flow Computation with Theoretically Justified Warping," Int'l J. Computer Vision, vol. 67, pp. 141-158, 2006.
[29] A. Shekhovtsov, I. Kovtun, and V.V. Hlaváč, "Efficient MRF Deformation Model for Non-Rigid Image Matching," Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, 2007.
[30] L. Sigal and M. Black, "Humaneva: Synchronized Video and Motion Capture Dataset for Evaluation of Articulated Human Motion," Technical Report CS-06-08, Brown Univ., 2006.
[31] F. Steinbruecker, T. Pock, and D. Cremers, "Large Displacement Optical Flow Computation without Warping," Proc. IEEE Int'l Conf. Computer Vision, 2009.
[32] D. Sun, S. Roth, J.P. Lewis, and M.J. Black, "Learning Optical Flow," Proc. European Conf. Computer Vision, pp. 83-87, 2008.
[33] J. Weber and J. Malik, "Robust Computation of Optical Flow in a Multi-Scale Differential Framework," Int'l J. Computer Vision, vol. 14, pp. 67-81, 1995.
[34] A. Wedel, T. Pock, C. Zach, D. Cremers, and H. Bischof, "An Improved Algorithm for TV-L1 Optical Flow," Proc. Statistical and Geometrical Approaches to Visual Motion Analysis, pp. 23-45, 2009.
[35] J. Wills and S. Belongie, "A Feature Based Method for Determining Dense Long Range Correspondences," Proc. European Conf. Computer Vision, pp. 170-182, 2004.
[36] C. Zach, T. Pock, and H. Bischof, "A Duality Based Approach for Realtime TV-L1 Optical Flow," Proc. Ann. Symp. German Assoc. Pattern Recognition, pp. 214-223, 2007.
[37] H. Zimmer, A. Bruhn, J. Weickert, L. Valgaerts, A. Salgado, B. Rosenhahn, and H.-P. Seidel, "Complementary Optic Flow," Proc. Int'l Workshop Enery Minimization Methods in Computer Vision, pp. 207-220, 2009.

Index Terms:
Motion, video, optical flow.
Citation:
Thomas Brox, Jitendra Malik, "Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 3, pp. 500-513, March 2011, doi:10.1109/TPAMI.2010.143
Usage of this product signifies your acceptance of the Terms of Use.