This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Using Photo-Consistency to Register 2D Optical Images of the Human Face to a 3D Surface Model
November 2001 (vol. 23 no. 11)
pp. 1266-1280

Abstract—In this paper, we propose a novel method to register two or more optical images to a 3D surface model. The potential applications of such a registration method could be in medicine; for example, in image guided interventions, surveillance and identification, industrial inspection, computer assisted manufacture, computer assisted maintenance, or telemanipulation in remote or hostile environments. Registration is performed by optimizing a similarity measure with respect to the transformation parameters. We propose a novel similarity measure based on “photo-consistency.” For each surface point, the similarity measure computes how consistent the corresponding optical image information in each view is with a lighting model. The relative pose of the optical images must be known. We validate the system using data from an optical-based surface reconstruction system and surfaces derived from magnetic resonance (MR) images of the human face. We test the accuracy and robustness of the system with respect to the number of video images, video image noise, errors in surface location and area, and complexity of the matched surfaces. We demonstrate the algorithm working on 10 further optical-based reconstructions of the human head and skin surfaces derived from MR images of the heads of five volunteers. Matching four optical images to a surface model produced a 3D error of between 1.45 and 1.59 mm, at a success rate of 100 percent, where the initial misregistration was up to 16 mm or degrees from the registration position.

[1] M.J. Clarkson, D. Rueckert, D.L.G. Hill, and D.J. Hawkes, “A Multiple 2D Video-3D Medical Image Registration Algorithm,” Proc. SPIE Conf. Medical Imaging: Image Processing, pp. 342-352, 2000.
[2] P.J. Edwards, A.P. King, C.R. MaurerJr., D.A. de Cunha, D.J. Hawkes, D.L.G. Hill, R.P. Gaston, M.R. Fenlon, A. Juczyzck, A.J. Strong, and C.L. Chandler, “Design and Evaluation of a System for Microscope-Assisted Guided Interventions (MAGI),” IEEE Trans. Medical Imaging, vol. 19, no. 11, pp. 1082-1093, 2000.
[3] W.E.L. Grimson, R. Kikinis, F.A. Jolesz, and P.M. Black, “Image-Guided Surgery,” Scientific Am., vol. 280, no. 6, pp. 62-69, 1999.
[4] J.D. Foley,A. van Dam,S.K. Feiner,, and J.F. Hughes,Computer Graphics: Principles and Practice,Menlo Park, Calif.: Addison-Wesley, 1990.
[5] K.N. Kutulakos and S.M. Seitz, “A Theory of Shape by Space Carving,” Int'l J. Computer Vision, vol. 38, no. 3, pp. 199-218, 2000.
[6] M.A. Fischler and R.C. Bolles, “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,” Graphics and Image Processing, vol. 24, no. 6, pp. 381–395, June 1981.
[7] R.M. Haralick, H. Joo, C.-N. Lee, X. Zhuang, and M.B. Kim, “Pose Estimation from Corresponding Point Data,” IEEE Trans. Systems, Man, and Cybernetics, vol. 19, no. 6, p. 1426, 1989.
[8] R.M. Haralick, C. Lee, K. Ottenberg, and M. Nolle, “Review and Analysis of Solutions of the Three Point Perspective Pose Estimation,” Int'l J. Computer Vision, vol. 13, no. 3, pp. 331-356, 1994.
[9] D.F. DeMenthon and L.S. Davis, "Exact and Approximate Solutions of the Perspective Three-Point Problem," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no, 11, pp. 1,100-1,105, 1992.
[10] Y. Liu, T.S. Huang, and O.D. Faugeras, “Determination of Camera Location from 2D to 3D Line and Point Correspondences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 12, no. 1, pp. 28-37, Jan. 1990.
[11] T.Q. Phong, R. Horaud, A. Yassine, and P.D. Tao, “Object Pose from 2D to 3D Point and Line Correspondences,” Int'l J. Computer Vision, vol. 15, pp. 225-243, 1995.
[12] A.C.F. Colchester, J. Zhao, S.K. Holton-Tainter, C.J. Henri, N. Maitland, P.T.E. Roberts, C.G. Harris, and R.J. Evans, “Development and Preliminary Evaluation of VISLAN, a Surgical Planning and Guidance System Using Intra-Operative Video Imaging,” Medical Image Analysis, vol. 1, no. 1, pp. 73-90, 1996.
[13] W.E.L. Grimson, G.J. Ettinger, S.J. White, T. Lozano-Perez, W.M. Wells, and R. Kikinis, “An Automatic Registration Method for Frameless Stereotaxy, Image Guided Surgery, and Enhanced Reality Visualization,” IEEE Trans. Medical Imaging, vol. 15, no. 2, pp. 129-140, 1996.
[14] W.E.L. Grimson, M. Leventon, G. Ettinger, A. Chabrerie, C.F. Ozlen, S. Nakajima, H. Atsumi, R. Kikinis, and P. Black, “Clinical Experience with a High Precision Image-Guided Neurosurgery System,” Proc. First Int'l Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI '98), pp. 63-73, 1998.
[15] P.J. Besl and N.D. McKay, "A Method for Registration of 3D Shapes," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239-256, Feb. 1992.
[16] H. Kollnig and H.H. Nagel, “3D Pose Estimation by Directly Matching Polyhedral Models to Gray Value Gradients,” Int'l J. Computer Vision, vol. 23, no. 3, pp. 283-302, 1997.
[17] L. Robert, “Camera Calibration without Feature Extraction,” Computer Vision and Image Understanding, vol. 63, no. 2, pp. 314-325, 1996.
[18] P. Viola and W.M. WellsIII, “Alignment by Maximization of Mutual Information,” Int'l J. Computer Vision, vol. 24, no. 2, pp. 137-154, 1997.
[19] M.E. Leventon, W.M. WellsIII, and W.E.L. Grimson, “Multiple View 2D-3D Mutual Information Registration,” Proc. Image Understanding Workshop, 1997.
[20] M.J. Clarkson, D. Rueckert, D.L.G. Hill, and D.J. Hawkes, “Registration of Multiple Video Images to Pre-Operative CT for Image Guided Surgery,” Proc. SPIE Conf. Medical Imaging: Image Processing, pp. 14-23, 1999.
[21] M.J. Clarkson, D. Rueckert, A.P. King, P.J. Edwards, D.L.G. Hill, and D.J. Hawkes, “Registration of Video Images to Tomographic Images by Optimizing Mutual Information Using Texture Mapping,” Proc. Second Int'l Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI '99), pp. 579-588, 1999.
[22] M. La Cascia, S. Sclaroff, and V. Athitsos, “Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Registration of Texture Mapped 3D Models,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 4, pp. 322-336, Apr. 2000.
[23] P. Fua, “From Multiple Stereo Views to Multiple 3D Surfaces,” Int'l J. Computer Vision, vol. 24, no. 1, pp. 19-35, 1997.
[24] P. Fua, “Regularized Bundle-Adjustment to Model Heads from Image Sequences without Calibration Data,” Int'l J. Computer Vision, vol. 38, no. 2, July 2000.
[25] G.P. Penney, J. Weese, J.A. Little, P. Desmedt, D.L.G. Hill, and D.J. Hawkes, “A Comparison of Similarity Measures for Use in 2D-3D Medical Image Registration,” IEEE Trans. Medical Imaging, vol. 17, no. 4, pp. 586-595, 1998.
[26] P. Agouris and T. Schenk, “Automated Aerotriangulation Using Image Multipoint Matching,” Photogrammetric Eng. and Remote Sensing, vol. 62, no. 6, pp. 703-710, 1996.
[27] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C. Cambridge Univ. Press, 1992.
[28] TCTi, Harefield, Middlesex UB9 6JA UK,http://vision.stanford.edu/~birch/publications/ http://cs1.cs.nyu.edu/phd_students/ ishikawa/index.htmlwww.neci.nj.nec.com/ homepages/olgahttp:/www.tcti.com. 2001.
[29] R.Y. Tsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,” IEEE J. Robotics and Automation, vol. 3, no. 4, pp. 323-344, 1987.
[30] W.E. Lorensen and H.E. Cline, “Marching Cubes: A High Resolution 3D Surface Construction Algorithm,” Computer Graphics (SIGGRAPH '87 Proc.), vol. 21, pp. 163-169, 1987.
[31] W. Schroeder, K. Martin, B. Lorensen, L. Avila, , R. Avila, and C. Law, The Visualization Toolkit An Object-Oriented Approach to 3D Graphics. Prentice-Hall, 1997.
[32] K.S. Arun, T.S. Huang, and S.D. Blostein, "Least Squares Fitting of Two 3-(D) Point Sets," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 9, pp. 698-700, 1987.
[33] M. Clarkson, “Registration of Optical Images to 3D Medical Images,” PhD thesis, The Univ. of London, United Kingdom, 2000. http://www-ipg.umds.ac.uk/m.clarksonthesis .
[34] D.G. Lowe, "Fitting Parameterized Three-Dimensional Models to Images," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, no. 5, pp. 441-450, May 1991.
[35] M.W. Powell, S. Sakar, and D. Goldgof, “Calibration of Light Sources” Proc. Conf. Computer Vision and Pattern Recognition, pp. 263-269, 2000.

Index Terms:
2D-3D registration, similarity measures, photo-consistency, pose estimation, extrinsic parameter calibration.
Citation:
Matthew J. Clarkson, Daniel Rueckert, Derek L.G. Hill, David J. Hawkes, "Using Photo-Consistency to Register 2D Optical Images of the Human Face to a 3D Surface Model," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1266-1280, Nov. 2001, doi:10.1109/34.969117
Usage of this product signifies your acceptance of the Terms of Use.