This Article 
 Bibliographic References 
 Add to: 
Real-Time Visual Tracking of Complex Structures
July 2002 (vol. 24 no. 7)
pp. 932-946

This paper presents a novel framework for three-dimensional model-based tracking. Graphical rendering technology is combined with constrained active contour tracking to create a robust wire-frame tracking system. It operates in real time at video frame rate (25 Hz) on standard hardware. It is based on an internal CAD model of the object to be tracked which is rendered using a binary space partition tree to perform hidden line removal. The visible edge features are thus identified online at each frame and correspondences are found in the video feed. A Lie group formalism is used to cast the motion computation problem into simple geometric terms so that tracking becomes a simple optimization problem solved by means of iterative reweighted least squares. A visual servoing system constructed using this framework is presented together with results showing the accuracy of the tracker. The system also incorporates real-time online calibration of internal camera parameters. The paper then describes how this tracking system has been extended to provide a general framework for tracking in complex configurations, including the use of multiple cameras, the tracking of structures with articulated components, or of multiple structures with constraints. The methodology used to achieve this exploits the simple geometric nature of the Lie group formalism which renders the constraints linear and homogeneous. The adjoint representation of the group is used to transform measurements into common coordinate frames. The constraints are then imposed by means of Lagrange multipliers. Results from a number of experiments performed using this framework are presented and discussed.

[1] J.K. Aggarwal, Q. Cai, W. Liao, and B. Sabata, “Nonrigid Motion Analysis: Articulated and Elastic Motion,” Computer Vision and Image Understanding, vol. 70, no. 2, pp. 142-156, 1998.
[2] M. Armstrong and A. Zisserman, “Robust Object Tracking,” Proc. Second Asian Conf. Computer Vision, pp. 58-62, 1995.
[3] R. Basri, E. Rivlin, and I. Shimshoni, “Visual Homing: Surfing on the Epipoles,” Proc. Int'l Conf. Computer Vision (ICCV '98), pp. 863-869, 1998.
[4] M.J. Black and A.D. Jepson, “Eigentracking: Robust Matching and Tracking of Articulated Objects Using a View-Based Representations,” Proc. Fourth European Conf. Computer Vision, B.F. Buxton and R. Cipolla, eds., p. 329-342, Apr. 1996.
[5] C. Bregler and J. Malik, “Tracking People with Twists and Exponential Maps,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 8–15, June 1998.
[6] T.J. Cham and J. Rehg, “A Multiple Hypothesis Approach to Figure Tracking,” Proc. Conf. Computer Vision and Pattern Recognition, vol. II, pp. 239–245, June 1999.
[7] S.Y. Lee and F. Hsu, “Spatial Reasoning and Similarity Retrieval of Images using 2D C-Strings Knowledge Representation,” Pattern Recognition, vol. 25, no. 3, pp. 305-318, 1992.
[8] R. Cipolla and A. Blake, “Image Divergence and Deformation from Closed Curves,” Int'l J. Robotics Research, vol. 16, no. 1, pp. 77-96, 1997.
[9] N. Daucher, M. Dhome, J. T. Lapresté, and G. Rives, “Modelled Object Pose Estimation and Tracking by Monocular Vision,” Proc. British Machine Vision Conf., pp. 249-258, 1993.
[10] Q. Delamarre and O. Faugeras, “3D Articulated Models and Multi-View Tracking with Silhouettes,” Proc. 17th Int'l Conf. Computer Vision, pp. 716–721, Sept. 1999.
[11] T. Drummond and R. Cipolla, “Real-Time Tracking of Multiple Articulated Structures in Multiple Views,” Proc. Sixth European Conf. Computer Vision, vol. 2, pp. 20-36, June 2000.
[12] T. Drummond and R. Cipolla, “Application of Lie Algebras to Visual Servoing,” Int'l J. Computer Vision, vol. 37, no. 1, pp. 21-41, 2000.
[13] B. Espiau, F. Chaumette, and P. Rives, “A New Approach to Visual Servoing in Robotics,” IEEE T-Robotics and Automation, vol. 8, no. 3, 1992.
[14] O.D. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint.Cambridge, Mass.: MIT Press, 1993.
[15] D.M. Gavrila and L.S. Davis, “3-D Model-Based Tracking of Humans in Action: A Multi-View Approach,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 73–80, June 1996.
[16] M. Haag and H.-H. Nagel, “Tracking of Complex Driving Manoeuvres in Traffic Image Sequences,” Image and Vision Computing, vol. 16, pp. 517-527, 1998.
[17] G. Hager, G. Grunwald, and K. Toyama, “Feature-Based Visual Servoing and Its Application to Telerobotics,” Intelligent Robotic Systems, V. Graefe, ed., Elsevier, 1995.
[18] C.G. Harris, “Geometry from Visual Motion,” Active Vision, A. Blake and A. Yuille, eds., 1992.
[19] C. Harris, “Tracking with Rigid Models,” Active Vision, A. Blake and A. Yuille, eds., chapter 4, pp. 59-73, MIT Press, 1992.
[20] P.J. Huber, Robust Statistics. Wiley, 1981.
[21] S. Hutchinson, G.D. Hager, and P. Corke, "A Tutorial Introduction to Visual Servo Control," IEEE Trans. Robotics and Automation, vol. 12, no. 5, pp. 651-670, 1996.
[22] M. Isard and A. Blake, “Condensation-Conditional Density Propagation for Visual Tracking,” Int'l J. Computer Vision, vol. 29, pp. 5-28, 1998.
[23] D.G. Lowe, "Fitting Parameterized Three-Dimensional Models to Images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 5, pp. 441-450, May 1991.
[24] D.G. Lowe, “Robust Model-Based Motion Tracking through the Integration of Search and Estimation,” Int'l J. Computer Vision, vol. 8, no. 2, pp. 113-122, 1992.
[25] J. MacCormick and A. Blake, “Spatial Dependence in the Observation of Visual Contours,” Proc. Fifth European Conf. Computer Vision (ECCV '98), pp. 765-781, 1998.
[26] E. Marchand, P. Bouthemy, F. Chaumette, and V. Moreau, “Robust Real-Time Visual Tracking Using a 2D-3D Model-Based Approach,” Proc. Int'l Conf. Computer Vision ,'99, vol. 1, pp. 262-268, Sept. 1999.
[27] N. Papanikolopoulos, “Selection of Features and Evaluation of Visual Measurements During Robotic Visual Servoing Tasks,” J. Intelligent Robotic Systems, vol. 13, pp. 279-304, 1995.
[28] M. Paterson and F. Yao, “Efficient Binary Space Partitions for Hidden Surface Removal and Solid Modeling,” Discrete and Computational Geometry, vol. 5, no. 5, pp. 485-503, 1990.
[29] A.C. Sanderson, L.E. Weiss, and C.P. Neumann, “Dynamic Sensor Based Control of Robots with Visual Feedback,” IEEE J. Robotics and Automation, vol. 3, pp. 404-417, 1987.
[30] D.H. Sattinger and O.L. Weaver, Lie Groups and Algebras with Applications to Physics, Geometry, and Mechanics. Springer-Verlag, 1986.
[31] J. Shi and C. Tomasi, Good Features to Track Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 593-600, 1994.
[32] C. Taylor and D. Kriegman, “Minimization on the Lie Group SO(3) and Related Manifolds,” Technical Report 9405, Yale Univ., Apr. 1994.
[33] D. Terzopoulos and R. Szeliski, “Tracking with Kalman Snakes,” Active Vision, A. Blake and A. Yuille, eds., pp. 3-20, MIT Press, 1992.
[34] W.J. Wilson, C.C. Williams Hulls, and G.S. Bell, “Relative End-Effector Control Using Cartesian Position Based Visual Servoing,” IEEE T-Robotics and Automation, vol. 12, no. 5, pp. 684-696, 1996.
[35] A.D. Worrall, G.D. Sullivan, and K.D. Baker, “Pose Refinement of Active Models Using Forces in 3D,” Proc. Third European Conf. Computer Vision (ECCV '94), J. Eklundh, ed., vol. 2, pp. 341-352, May 1994.
[36] P. Wunsch and G. Hirzinger, “Real-Time Visual Tracking of 3-D Objects with Dynamic Handling of Occlusion,” Proc. 1997 Int'l Conf. Robotics and Automation, pp. 2868-2873, 1997.

Index Terms:
Visual tracking, real-time, 3D, Lie groups, articulated motion.
Tom Drummond, Roberto Cipolla, "Real-Time Visual Tracking of Complex Structures," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 7, pp. 932-946, July 2002, doi:10.1109/TPAMI.2002.1017620
Usage of this product signifies your acceptance of the Terms of Use.