This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Real-Time Markerless Tracking for Augmented Reality: The Virtual Visual Servoing Framework
July/August 2006 (vol. 12 no. 4)
pp. 615-628

Abstract—Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through” monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. Here, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.

[1] R. Azuma, “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385, Aug. 1997.
[2] R. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre, “Recent Advances in Augmented Reality,” IEEE Computer Graphics and Application, vol. 21, no. 6, pp. 34-47, Nov. 2001.
[3] M. Billinghurst, H. Kato, and I. Poupyrev, “The Magicbook: Moving Seamlessly between Reality and Virtuality,“ IEEE Computer Graphics and Applications, vol. 21, no. 3, pp. 6-8, May 2001.
[4] P. Bouthemy, “A Maximum Likelihood Framework for Determining Moving Edges,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 5, pp. 499-511, May 1989.
[5] K.-W. Chia, A.-D. Cheok, and S. Prince, “Online 6 DOF Augmented Reality Registration from Natural Features,” Proc. IEEE Int'l Symp. Mixed and Augmented Reality, pp. 305-316, Sept. 2002.
[6] A.I. Comport, D. Kragic, E. Marchand, and F. Chaumette, “Robust Real-Time Visual Tracking: Comparison, Theoretical Analysis and Performance Evaluation,” Proc. IEEE Int'l Conf. Robotics and Automation, pp. 2852-2857, Apr. 2005.
[7] A.J. Davison, “Real-Time Simultaneous Localisation and Mapping with a Single Camera,” Proc. IEEE Int'l Conf. Computer Vision, pp. 1403-1410, 2003.
[8] S. de Ma, “Conics-Based Stereo, Motion Estimation and Pose Determination,” Int'l J. Computer Vision, vol. 10, no. 1, pp. 7-25, 1993.
[9] D. Dementhon and L. Davis, “Model-Based Object Pose in 25 Lines of Code,” Int'l J. Computer Vision, vol. 15, pp. 123-141, 1995.
[10] M. Dhome, J.-T. Lapresté, G. Rives, and M. Richetin, “Determination of the Attitude of Modelled Objects of Revolution in Monocular Perspective Vision,” Proc. European Conf. Computer Vision, pp. 475-485, Apr. 1990.
[11] M. Dhome, M. Richetin, J.-T. Lapresté, and G. Rives, “Determination of the Attitude of 3D Objects from a Single Perspective View,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 11, no. 12, pp. 1265-1278, Dec. 1989.
[12] T. Drummond and R. Cipolla, “Real-Time Visual Tracking of Complex Structures,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 7, pp. 932-946, July 2002.
[13] B. Espiau, F. Chaumette, and P. Rives, “A New Approach to Visual Servoing in Robotics,” IEEE Trans. Robotics and Automation, vol. 8, no. 3, pp. 313-326, June 1992.
[14] O.D. Faugeras and G. Toscani, “Camera Calibration for 3D Computer Vision,” Proc. Int'l Workshop Machine Vision and Machine Intelligence, pp. 240-247, Feb. 1987.
[15] N. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Application to Image Analysis and Automated Cartography,” Comm. ACM, vol. 24, no. 6, pp. 381-395, June 1981.
[16] S. Ganapathy, “Decomposition of Transformation Matrices for Robot Vision,” Pattern Recognition Letter, vol 2, pp. 401-412, 1984.
[17] A. Glassner, “Everyday Computer Graphics, IEEE Computer Graphics and Applications, vol. 23, no. 6, pp. 76-82, Nov. 2003.
[18] R. Haralick, H. Joo, C. Lee, X. Zhuang, V Vaidya, and M. Kim, “Pose Estimation from Corresponding Point Data,” IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 6, pp. 1426-1445, Nov. 1989.
[19] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge Univ. Press, 2001.
[20] Visual Servoing: Real Time Control of Robot Manipulators Based on Visual Sensory Feedback, World Scientific Series in Robotics and Automated Systems, vol. 7, K. Hashimoto, ed. Singapore: World Scientific Press, 1993.
[21] P.-W. Holland and R.-E. Welsch, “Robust Regression Using Iteratively Reweighted Least-Squares,” Comm. Statistics Theory and Methods, vol. A6, pp. 813-827, 1977.
[22] R. Horaud, B. Conio, O. Leboulleux, and B. Lacolle, “An Analytic Solution for the Perspective Four-Points Problem,” Computer Vision, Graphics, and Image Processing, vol. 47, no. 1, pp. 33-44, July 1989.
[23] P.-J. Huber, Robust Statistics. New York: Wiley, 1981.
[24] S. Hutchinson, G. Hager, and P. Corke, “A Tutorial on Visual Servo Control,” IEEE Trans. Robotics and Automation, vol. 12, no. 5, pp. 651-670, Oct. 1996.
[25] M. Isard and A. Blake, “Condensation— Conditional Density Propagation for Visual Tracking,” Int. J. Computer Vision, vol. 29, no. 1, pp. 5-28 Jan. 1998.
[26] H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto, and K. Tachibana, “Virtual Object Manipulation on a Table-Top AR Environment,” Proc. Int'l Symp. Augmented Reality 2000, Oct. 2000.
[27] G. Klein and T. Drummond, “Robust Visual Tracking for Non-Instrumented Augmented Reality,” Proc. ACM/IEEE Int'l Symp. Mixed and Augmented Reality, pp. 113-122, Oct. 2003.
[28] R. Koch, “Dynamic 3-D Scene Analysis through Synthesis Feedback Control,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 556-568, June 1993.
[29] R. Kumar and A.R. Hanson, “Robust Methods for Estimating Pose and a Sensitivity Analysis,” CVGIP: Image Understanding, vol. 60, no. 3, pp. 313-342, Nov. 1994.
[30] K. Kutulakos and J. Vallino, “Calibration-Free Augmented Reality,” IEEE Trans. Visualization and Computer Graphics, vol. 4, no. 1, pp. 1-20, Jan. 1998.
[31] V. Lepetit, J. Pilet, and P. Fua, “Point Matching as Classification Problem for Fast and Robust Object Pose Estimation,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, June 2004.
[32] Y. Liu, T.S. Huang, and O.D. Faugeras, “Determination of Camera Location from 2D to 3D Line and Point Correspondences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 1, pp. 28-37, Jan. 1990.
[33] D. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[34] D.G. Lowe, “Three-Dimensional Object Recognition from Single Two-Dimensional Images,” Artificial Intelligence, vol. 31, no. 3, pp. 355-394, Mar. 1987.
[35] D.G. Lowe, “Fitting Parameterized Three-Dimensional Models to Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 5, pp. 441-450, May 1991.
[36] C.P. Lu, G.D. Hager, and E. Mjolsness, “Fast and Globally Convergent Pose Estimation from Video Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 6, pp. 610-622, June 2000.
[37] Y. Ma, S. Soatto, J. Košecká, and S. Sastry, An Invitation to 3-D Vision. Springer, 2004.
[38] E. Marchand, P. Bouthemy, F. Chaumette, and V. Moreau, “Robust Real-Time Visual Tracking Using a 2D-3D Model-Based Approach,” Proc. IEEE Int'l Conf. Computer Vision, vol. 1, pp. 262-268, Sept. 1999.
[39] E. Marchand and F. Chaumette, “Virtual Visual Servoing: A Framework for Real-Time Augmented Reality,” Eurographics Conf. Proc., pp. 289-298, Sept. 2002.
[40] E. Marchand, F. Spindler, and F. Chaumette, “ViSP for Visual Servoing: A Generic Software Platform with a Wide Class of Robot Control Skills,” IEEE Robotics and Automation Magazine, vol. 12, no. 4, pp. 40-52, Dec. 2005.
[41] N. Navab, “Developing Killer Apps for Industrial Augmented Reality,” IEEE Computer Graphics and Applications, vol. 24, no. 3, pp. 16-20, May 2004.
[42] A.N. Netravali and J. Salz, “Algorithms for Estimation of Three-Dimensional Motion,” AT&T Technical J., vol. 64, no. 2, pp. 335-346, 1985.
[43] U. Neumann, S. You, Y. Cho, J. Lee, and J. Park, “Augmented Reality Tracking in Natural Environments,” Proc. Int'l Symp. Mixed Realities, 1999.
[44] D. Nister, “Preemptive Ransac for Live Structure and Motion Estimation,” Proc. IEEE Int'l Conf. Computer Vision, vol. 1, pp. 199-207, Nov. 2003.
[45] J. Park, B. Jiang, and U. Neumann, “Vision-Based Pose Computation: Robust and Accurate Augmented Reality Tracking,” Proc. ACM/IEEE Int'l Workshop Augmented Reality, pp. 3-12, Oct. 1998.
[46] S. Prince, K. Xu, and A. Cheok, “Augmented Reality Camera Tracking with Homographies,” IEEE Computer Graphics and Applications, vol. 22, no. 6, pp. 39-45, Nov. 2002.
[47] R. Safaee-Rad, I. Tchoukanov, B. Benhabib, and K.C. Smith, “Three-Dimensional Location Estimation of Circular Features for Machine Vision,” IEEE Trans. Robotics and Automation, vol. 8, no. 2, pp. 624-639, Oct. 1992.
[48] Y. Seo, M.H. Ahn, and K.-S. Hong, “Real-Time Camera Calibration for Virtual Studio,” IEEE Trans. Visualization and Computer Graphics, vol. 6, no. 4, pp. 346-359, Oct. 2000.
[49] J. Shi and C. Tomasi, “Good Features to Track,” Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition, pp. 593-600, June 1994.
[50] G. Simon and M.-O. Berger, “Pose Estimation for Planar Structures,” IEEE Computer Graphics and Applications, vol. 22, no. 6, pp. 46-53, Nov. 2002.
[51] G. Simon, A. Fitzgibbon, and A. Zisserman, “Markerless Tracking Using Planar Structures in the Scene,” Proc. IEEE/ACM Int'l Symp. Augmented Reality, pp. 120-128, Oct. 2000.
[52] C.-V. Stewart, “Robust Parameter Estimation in Computer Vision,” SIAM Rev., vol. 41, no. 3, pp. 513-537, Sept. 1999.
[53] V. Sundareswaran and R. Behringer, “Visual Servoing-Based Augmented Reality,“ Proc. IEEE Int'l Workshop Augmented Reality, Nov. 1998.
[54] H. Wang and D. Suter, “Robust Adaptive-Scale Parametric Estimation for Computer Vision,“ IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 11, pp. 1459-1474, Nov. 2004.
[55] X. Zhang, S. Fronz, and N. Navab, “Visual Marker Detection and Decoding in AR Systems: A Comparative Study,” Proc. IEEE Int'l Symp. Mixed and Augmented Reality, pp. 79-106, Sept. 2002.

Index Terms:
Augmented reality, virtual visual servoing, robust estimators, real-time, model-based tracking, model-free tracking.
Citation:
Andrew I. Comport, Eric Marchand, Muriel Pressigout, Fran?ois Chaumette, "Real-Time Markerless Tracking for Augmented Reality: The Virtual Visual Servoing Framework," IEEE Transactions on Visualization and Computer Graphics, vol. 12, no. 4, pp. 615-628, July-Aug. 2006, doi:10.1109/TVCG.2006.78
Usage of this product signifies your acceptance of the Terms of Use.