This Article 
 Bibliographic References 
 Add to: 
Shape Recognition and Pose Estimation for Mobile Augmented Reality
October 2011 (vol. 17 no. 10)
pp. 1369-1379
Nate Hagbi, Ben-Gurion University, Israel
Oriel Bergig, Ben-Gurion University, Israel
Jihad El-Sana, Ben-Gurion University, Israel
Mark Billinghurst, University of Canterbury, New Zealand
Nestor is a real-time recognition and camera pose estimation system for planar shapes. The system allows shapes that carry contextual meanings for humans to be used as Augmented Reality (AR) tracking targets. The user can teach the system new shapes in real time. New shapes can be shown to the system frontally, or they can be automatically rectified according to previously learned shapes. Shapes can be automatically assigned virtual content by classification according to a shape class library. Nestor performs shape recognition by analyzing contour structures and generating projective-invariant signatures from their concavities. The concavities are further used to extract features for pose estimation and tracking. Pose refinement is carried out by minimizing the reprojection error between sample points on each image contour and its library counterpart. Sample points are matched by evolving an active contour in real time. Our experiments show that the system provides stable and accurate registration, and runs at interactive frame rates on a Nokia N95 mobile phone.

[1] H. Kato, M. Billinghurst, I. Pouipyrev, K. Imamoto, and K. Tachibana, “Virtual Object Manipulation on a Table-Top AR Environment,” Proc. Int'l Symp. Augmented Reality, pp. 111-119, 2000.
[2] G.A. Lee, C. Nelles, M. Billinghurst, and G.J. Kim, “Immersive Authoring of Tangible Augmented Reality Applications,” Proc. Third IEEE/ACM Int'l Symp. Mixed and Augmented Reality, pp. 172-181, 2004.
[3] V. Lepetit and P. Fua, “Monocular Model-Based 3D Tracking of Rigid Objects: A Survey,” Foundations and Trends in Computer Graphics and Vision, vol. 1, no. 1, pp. 1-89, 2005.
[4] N. Hagbi, O. Bergig, J. El-Sana, and M. Billinghurst, “Shape Recognition and Pose Estimation for Mobile Augmented Reality,” Proc. Int'l Symp. Mixed and Augmented Reality, pp. 65-71, 2009.
[5] N. Hagbi, O. Bergig, J. El-Sana, K. Kedem, and M. Billinghurst, “In-Place Augmented Reality,” Proc. Int'l Symp. Mixed and Augmented Reality, pp. 135-138, 2008.
[6] W.A. Hoff, K. Nguyen, and T. Lyon, “Computer Vision-Based Registration Techniques for Augmented Reality,” Proc. Intelligent Robots and Control Systems XV, Intelligent Control Systems and Advanced Manufacturing, pp. 538-548, 1996.
[7] A. State, G. Hirota, D. Chen, W. Garett, and M. Livingston, “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking,” Proc. SIGGRAPH Computer Graphics, pp. 429-438, 1996.
[8] H. Kato and M. Billinghurst, “Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System,” Proc. Second Int'l Workshop Augmented Reality, 1999.
[9] M. Fiala, “ARTag, A Fiducial Marker System Using Digital Techniques,” Proc. Computer Vision and Pattern Recognition, pp. 590-596, 2005.
[10] D. Schmalstieg and D. Wagner, “Experiences with Handheld Augmented Reality,” Proc. Sixth IEEE and ACM Int'l Symp. Mixed and Augmented Reality, 2007.
[11] M. Rohs, J. Schöning, A. Krüger, and B. Hecht, “Towards Real-Time Markerless Tracking of Magic Lenses on Paper Maps,” Proc. Int'l Conf. Pervasive Computing, pp. 69-72, 2007.
[12] C.A. Rothwell, A. Zisserman, D.A. Forsyth, and J.L. Mundy, “Canonical Frames for Planar Object Recognition,” Proc. Second European Conf. Computer Vision, pp. 757-772, 1992.
[13] A. Zisserman, D. Forsyth, J. Mundy, C. Rothwell, J. Liu, and N. Pillow, “3D Object Recognition Using Invariance,” Artificial Intelligence, vol. 78, nos. 1/2, pp. 239-288, 1995.
[14] T. Drummond and R. Cipolla, “Visual Tracking and Control Using Lie Algebras,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 652-657, 1999.
[15] A. Ruiz, E.T. López de Teruel, and L. Fernández, “Robust Homography Estimation from Planar Contours Based on Convexity,” Proc. European Conf. Computer Vision, pp. 107-120, 2006.
[16] W.A. Fitzgibbon, “Robust Registration of 2D and 3D Point Sets,” Proc. British Machine Vision Conf., pp. 411-420, 2001.
[17] D. Bradley and G. Roth, “Adaptive Thresholding Using the Integral Image,” J. Graphics Tools, vol. 12, no. 2, pp. 13-21, 2007.
[18] Y. Lamdan, J.T. Schwartz, and H.J. Wolfson, “Object Recognition by Affine Invariant Matching,” Proc. CS Conf. Computer Vision and Pattern Recognition, pp. 335-344, 1988.
[19] R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision, second ed. Cambridge Univ. Press, 2003.
[20] T.R. Riklin, N. Sochen, and N. Kiryati, “Mutual Segmentation with Level-Sets,” Proc. Conf. Computer Vision and Pattern Recognition Workshop, 2006.
[21] J.J. LaViolaJr., “Double Exponential Smoothing: An Alternative to Kalman Filter-Based Predictive Tracking,” Proc. Workshop Virtual Environments, pp. 199-206, 2003.
[22] J.J. LaViolaJr., “An Experiment Comparing Double Exponential Smoothing and Kalman Filter-Based Predictive Tracking Algorithms,” Proc. IEEE Virtual Reality, pp. 283-284, 2003.
[23] S. Belongie and J. Malik, “Matching with Shape Contexts,” Proc. IEEE Workshop Content-Based Access of Image and Video Libraries, pp. 20-26, 2000.
[24] H. Ling and D.W. Jacobs, “Shape Classification Using the Inner-Distance,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 2, pp. 286-299, Feb. 2007.
[25] N. Hagbi, R. Grasset, O. Bergig, M. Billinghurst, and J. El-Sana, “In-Place Sketching for Content Authoring in Augmented Reality Games,” Proc. IEEE Virtual Reality, 2010.
[26] C.A. Rothwell, A. Zisserman, D. Forsyth, and J. Mundy, “Planar Object Recognition Using Projective Shape Representation,” Int'l J. Computer Vision, vol. 16, pp. 57-99, 1995.
[27] S. Carlsson, “Projectively Invariant Decomposition and Recognition of Planar Shapes,” Int'l J. Computer Vision, vol. 17, no. 2, pp. 193-209, 1996.
[28] A.W. Fitzgibbon, M. Pilu, and R.B. Fisher, “Direct Least Square Fitting of Ellipses,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 476-480, May 1999.
[29] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active Contour Models,” Int'l J. Computer Vision, vol. 1, no. 4, pp. 321-331, 1988.
[30] V. Caselles, R. Kimmel, and G. Sapiro, “Geodesic Active Contours,” Int'l J. Computer Vision, vol. 22, no. 1, pp. 61-79, 1997.

Index Terms:
Multimedia information systems, artificial, augmented, and virtual realities, image processing and computer vision, scene analysis, tracking.
Nate Hagbi, Oriel Bergig, Jihad El-Sana, Mark Billinghurst, "Shape Recognition and Pose Estimation for Mobile Augmented Reality," IEEE Transactions on Visualization and Computer Graphics, vol. 17, no. 10, pp. 1369-1379, Oct. 2011, doi:10.1109/TVCG.2010.241
Usage of this product signifies your acceptance of the Terms of Use.