This Article 
 Bibliographic References 
 Add to: 
Calibration-Free Augmented Reality
January-March 1998 (vol. 4 no. 1)
pp. 1-20

Abstract—Camera calibration and the acquisition of Euclidean 3D measurements have so far been considered necessary requirements for overlaying three-dimensional graphical objects with live video. In this article, we describe a new approach to video-based augmented reality that avoids both requirements: It does not use any metric information about the calibration parameters of the camera or the 3D locations and dimensions of the environment's objects. The only requirement is the ability to track across frames at least four fiducial points that are specified by the user during system initialization and whose world coordinates are unknown.

Our approach is based on the following observation: Given a set of four or more noncoplanar 3D points, the projection of all points in the set can be computed as a linear combination of the projections of just four of the points. We exploit this observation by 1) tracking regions and color fiducial points at frame rate, and 2) representing virtual objects in a non-Euclidean, affine frame of reference that allows their projection to be computed as a linear combination of the projection of the fiducial points. Experimental results on two augmented reality systems, one monitor-based and one head-mounted, demonstrate that the approach is readily implementable, imposes minimal computational and hardware requirements, and generates real-time and accurate video overlays even when the camera parameters vary dynamically.

[1] R.T. Azuma, "A Survey of Augmented Reality," Presence: Teleoperators and Virtual Environments, vol. 6, no. 4, pp. 355-385, 1997.
[2] T.P. Caudell and D.W. Mizell, “Augmented Reality: An Application of Heads-Up Display Technology to Manual Manufacturing Processes,” Proc. Hawaii Int’l Conf. System Sciences, Vol. 2, 1992, pp. 659-669.
[3] S. Feiner, B. MacIntyre, and D. Seligmann, "Knowledge-Based Augmented Reality," Comm. ACM, July 1993, pp. 53-62.
[4] W.E.L. Grimson, T. Lozano-Perez, S.J. White, W.M. Wells III, R. Kikinis, and G.J. Ettinger, “An Automatic Registration Method for Frameless Stereotaxy, Image Guided Surgery, and Enhanced Reality Visualization,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 430-436, June 1994.
[5] W. Grimson, G. Ettinger, S. White, P. Gleason, T. Lozano-Perez, W. Wells, and R. Kikinis, “Evaluating and Validating an Automated Registration System for Enhanced Reality Visualization in Surgery,” Proc. Computer Vision, Virtual Reality, and Robotics in Medicine '95 (CVRMed '95), pp. 3-12, Apr. 1995.
[6] M. Uenohara and T. Kanade, "Vision-Based Object Registration for Real-Time Image Overlay," Proc. CVRMED '95, pp. 14-22, 1995.
[7] M. Bajura, H. Fuchs, and R. Ohbuchi, "Merging Virtual Objects with the Real World: Seeing Ultrasound Imagery within the Patient," Proc. Siggraph 92, ACM Press, New York, 1992, pp. 203-210.
[8] A. State, M.A. Livingston, W.F. Garrett, G. Hirota, M.C. Whitton, E.D. Pisano, and H. Fuchs, “Technologies for Augmented Reality Systems: Realizing Ultrasound-Guided Needle Biopsies,” Proc. 23rd Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '96), pp. 439-446, Aug. 1996.
[9] P. Wellner, "Interaction with Paper on the Digital Desk, Comm. ACM, vol. 36, no. 7, 1993, pp. 87-96.
[10] T. Darrell, P. Maes, B. Blumberg, and A.P. Pentland, "A Novel Environment for Situated Vision and Action," IEEE Workshop Visual Behaviors, pp. 68-72, 1994.
[11] M.M. Wloka and B.G. Anderson, "Resolving Occlusion in Augmented Reality," Proc. Symp. Interactive 3D Graphics, pp. 5-12, 1995.
[12] M. Tuceryan, D.S. Greer, R.T. Whitaker, D.E. Breen, C. Crampton, E. Rose, and K.H. Ahlers, “Calibration Requirements and Procedures for a Monitor-Based Augmented Reality System,” IEEE Trans. Visualization and Computer Graphics, vol. 1, no. 3, pp. 255-273, 1995.
[13] R. Holloway, Registration Errors in Augmented Reality Systems, PhD dissertation, Dept. of Computer Science, Univ. North Carolina, Chapel Hill, N.C., 1995.
[14] J. Mellor, "Enhanced Reality Visualization in a Surgical Environment," Master's thesis, Massachusetts Inst. of Tech nology, 1995.
[15] M. Bajura and U. Neumann, "Dynamic Registration Correction in Video-Based Augmented Reality Systems," IEEE Computer Graphics and Applications, vol. 15, no. 5, Sept. 1995, pp. 52-61.
[16] A. State, G. Hirota, D.T. Chen, W.F. Garrett, and M.A. Livingston, “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking,” Proc. 23rd Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '96), pp. 429-438, Aug. 1996.
[17] S. Ravela, B. Draper, J. Lim, and R. Weiss, Adaptive Tracking and Model Registration Across Distinct Aspects Proc. Int'l Conf. Intelligent Robots and Systems, pp. 174-180, 1995.
[18] D.G. Lowe, “Robust Model-Based Motion Tracking through the Integration of Search and Estimation,” Int'l J. Computer Vision, vol. 8, no. 2, pp. 113-122, 1992.
[19] D.G. Lowe, "Fitting Parameterized Three-Dimensional Models to Images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 5, pp. 441-450, May 1991.
[20] G. Verghese, K.L. Gale, and C.R. Dyer, "Real-Time Motion Tracking of Three-Dimensional Objects," Proc. IEEE Conf. Robotics and Automation, pp. 1,998-2,003, 1990.
[21] J.J. Koenderink and A.J. van Doorn, "Affine Structure From Motion," J. Optical Soc. Am., vol. A, no. 2, pp. 377-385, 1991.
[22] S. Ullman and R. Basri, "Recognition by Linear Combinations of Models," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 992-1006, 1991.
[23] O.D. Faugeras, "Stratification of Three-Dimensional Vision: Projective, Affine, and Metric Representations," J. Optical Soc. Am., vol. A, vol. 12, no. 3, pp. 465-484, 1995.
[24] Geometric Invariance in Computer Vision, J.L. Mundy and A. Zisserman, eds. MIT Press, 1992.
[25] O.D. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint.Cambridge, Mass.: MIT Press, 1993.
[26] D. Weinshall and C. Tomasi, “Linear and Incremental Acquisition of Invariant Shape Models from Image Sequences,” Proc. Fourth Int'l Conf. Computer Vision, pp. 675-682, May 1993.
[27] Y. Lamdan, J. Schwartz, and H. Wolfson, "Object Recognition by Affine Invariant Matching," Proc. IEEE Computer Vision and Pattern Recognition Conf., IEEE Computer Society, 1988, pp. 335-344.
[28] R. Cipolla, P.A. Hadfield, and N.J. Hollinghurst, "Uncalibrated Stereo Vision With Pointing for a Man-Machine Interface," Proc. IAPR Workshop on Machine Vision Applications, 1994.
[29] A. Azarbayejani,T. Starner,B. Horowitz,, and A. Pentland,“Visually controlled graphics,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 15, no. 6, pp. 602-605, June 1993. (Special Section on 3D Modeling in Image Analysis and Synthesis).
[30] J.D. Foley,A. van Dam,S.K. Feiner,, and J.F. Hughes,Computer Graphics: Principles and Practice,Menlo Park, Calif.: Addison-Wesley, 1990.
[31] Y. Bar-Shalom and T.E. Fortmann, Tracking and Data Association. Academic Press, 1988.
[32] J.-R. Wu and M. Ouhyoung, "A 3D Tracking Experiment on Latency and Its Compensation Methods in Virtual Environments," Proc. Eighth ACM Symp. User Interface Software and Technology, pp. 41-49, 1995.
[33] R. Azuma and G. Bishop, “Improving Static and Dynamic Registration in an Optical See-Through HMD,” Computer Graphics (Proc. Siggraph), ACM Press, 1994, pp. 197-204.
[34] M. Gleicher and A. Witkin, "Through-the-Lens Camera Control," Proc. SIGGRAPH '92, pp. 331-340, 1992.
[35] L.S. Shaprio, A. Zisserman, and M. Brady, "3D Motion Recovery Via Affine Epipolar Geometry," Int'l J. Computer Vision, vol. 16, pp. 147-182, 1995.
[36] W.B. Thompson and J.L. Mundy, "Three-Dimensional Model Matching From An Unconstrained Viewpoint," Proc. IEEE Conf. Robotics and Automation, pp. 208-220, 1987.
[37] A. Shashua, "A Geometric Invariant for Visual Recognition and 3D Reconstruction From Two Perspective/Orthographic Views," Proc. IEEE Workshop Qualitative Vision, pp. 107-117, 1993.
[38] E.B. Barrett,M.H. Brill,N.N. Haag,, and P.M. Payton,“Invariant linear methods in photogrammetry and model-matching,” J.L. Mundy and Zisserman, eds., Applications of Invariances in Computer Vision. MIT Press, 1992.
[39] S.M. Seitz and C.R. Dyer, "Complete Scene Structure From Four Point Correspondences," Proc. Fifth Int'l Conf. Computer Vision, pp. 330-337, 1995.
[40] G. Hager, “Calibration-Free Visual Control Using Projective Invariance,” Proc. Fifth Int'l Conf. Computer Vision, June 1995.
[41] Active Vision, A. Blake and A. Yuille, eds. MIT Press, 1992.
[42] Real-Time Computer Vision, C.M. Brown and D. Terzopoulos, eds. Cambridge Univ. Press, 1994.
[43] A. Blake and M. Isard, ”Three-Dimensional Position and Shape Input Using Video Tracking of Hands and Lips,” Proc. SIGGRAPH '94, pp. 185-192, 1994.
[44] G. Hager and P. Belhumeur, “Real-Time Tracking of Image Regions with Changes in Geometry and Illumination,” Proc. Computer Vision and Pattern Recognition, 1996.
[45] D.H. Ballard and C.M. Brown, Computer Vision, Prentice Hall, Upper Saddle River, N.J., 1982.
[46] C. Tomasi and T. Kanade, "Shape and Motion From Image Streams Under Orthography: A Factorization Method," Int'l J. Computer Vision, vol. 9, no. 2, pp. 137-154, 1992.
[47] K. Jack, Video Demystified: A Handbook for the Digital Engineer. HighText Publications Inc., 1993.
[48] R. Tsai, "A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses," IEEE J. Robotics and Automation, vol. 3, no. 4, pp. 323-344, Aug. 1987.
[49] B. Boufama, D. Weinshall, and M. Werman, "Shape from motion algorithms: A comparative analysis of scaled orthography and perspective," J.-O. Eklundh, Computer Vision—ECCV '94, pp. I.199-I.204.Berlin-Heidelberg: Springer-Verlag, 1994.
[50] G. Turk and M. Levoy, “Zippered Polygon Meshes from Range Images,” Proc. SIGGRAPH '94, pp. 311-318, 1994.
[51] E.K.-Y. Jeng and Z. Xiang, "Moving Cursor Plane for Interactive Sculpting," ACM Trans. Graphics, vol. 15, no. 3, pp. 211-222, 1996.
[52] W. Wang and A.F. Kaufman, “Volume Sculpting,” Proc. Symp. Interactive 3D Graphics Proceedings, pp. 151-156, Apr. 1995.
[53] H. Qin and D. Terzopoulos, “D-NURBS: A Physics-Based Framework for Geometric Design,” IEEE Trans. Visualization and Computer Graphics, vol. 2, no. 1, pp. 85-96, Mar. 1996.
[54] P.E. Debevec, C.J. Taylor, and J. Malik, “Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach,” Proc. SIGGRAPH '96, pp. 11-20, Aug. 1996.
[55] S.A. Tebo, D.A. Leopold, D.M. Long, S.J. Zinreich, and D.W. Kennedy, "An Optical 3D Digitizer for Frameless Stereotactic Surgery," IEEE Computer Graphics and Applications, vol. 16, pp. 55-64, Jan. 1996.
[56] M. Agrawala, A.C. Beers, and M. Levoy, "3D Painting on Scanned Surfaces," Proc. Symp. Interactive 3D Graphics, pp. 145-150, 1995.
[57] K.N. Kutulakos and J. Vallino, Affine Object Representations for Calibration-Free Augmented Reality: Example MPEG Sequences. / cse/cse.html u/kyros/mpegsTVCG.html, 1996.
[58] C. Wiles and M. Brady, "On the Appropriateness of Camera Models," Proc. Fourth European Conf. Computer Vision, pp. 228-237, 1996.
[59] O. Faugeras, "What can be seen in three dimensions with an uncalibrated stereo rig?" Second European Conf. Computer Vision, pp. 563-578, 1992.
[60] R. Mohr, B. Boufama, and P. Brand, “Accurate Projective Reconstruction,” Applications of Invariance in Computer Vision, J.L. Mundy, A. Zisserman, and D. Forsyth, eds. pp. 297-316. Springer-Verlag, 1994.
[61] J. Dorsey, J. Arvo, and D. Greenberg, "Interactive Design of Complex Time-Dependent Lighting," IEEE Computer Graphics and Applications, vol. 15, no. 2, pp. 26-36, Mar. 1995.
[62] A. Shashua, “Geometry and Photometry in 3D Visual Recognition,” PhD dissertation, Dept. of Brain and Cognitive Sciences, Massachusetts Inst. of Technology, Cambridge, Nov. 1992.
[63] P.N. Belhumeur and D.J. Kriegman, "What is the Set of Images of an Object under all Possible Lighting Conditions?," IEEE Proc. Conf. Computer Vision and Pattern Recognition, 1996.
[64] P.F. McLauchlan, I.D. Reid, and D.W. Murray, “Recursive Affine Structure and Motion from Image Sequence,” Proc. Third European Conf. Computer Vision, pp. 217-224, Stockholm, Sweden, May 1994.
[65] L. O'Gorman, "Subpixel Precision of Straight-Edged Shapes for Registration and Measurement," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 7, pp. 746-751, July 1991.
[66] K. Toyama and G. Hager, “Incremental Focus of Attention for Robust Visual Tracking,” Proc. Computer Vision and Pattern Recognition, pp. 189-195, 1996.
[67] S.K. Nayar, “Catadioptric Omnidirectional Cameras,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 482-488, June 1997.

Index Terms:
Augmented reality, real-time computer vision, calibration, registration, affine representations, feature tracking, 3D interaction techniques.
Kiriakos N. Kutulakos, James R. Vallino, "Calibration-Free Augmented Reality," IEEE Transactions on Visualization and Computer Graphics, vol. 4, no. 1, pp. 1-20, Jan.-March 1998, doi:10.1109/2945.675647
Usage of this product signifies your acceptance of the Terms of Use.