This Article 
 Bibliographic References 
 Add to: 
Fisheye Video Correction
Oct. 2012 (vol. 18 no. 10)
pp. 1771-1783
Jin Wei, Tsinghua University, Beijing
Chen-Feng Li, Swansea University, Swansea
Shi-Min Hu, Tsinghua University, Beijing
Ralph R. Martin, Cardiff University, Cardiff
Chiew-Lan Tai, Hong Kong University of Science and Technology, Hong Kong
Various types of video can be captured with fisheye lenses; their wide field of view is particularly suited to surveillance video. However, fisheye lenses introduce distortion, and this changes as objects in the scene move, making fisheye video difficult to interpret. Current still fisheye image correction methods are either limited to small angles of view, or are strongly content dependent, and therefore unsuitable for processing video streams. We present an efficient and robust scheme for fisheye video correction, which minimizes time-varying distortion and preserves salient content in a coherent manner. Our optimization process is controlled by user annotation, and takes into account a wide set of measures addressing different aspects of natural scene appearance. Each is represented as a quadratic term in an energy minimization problem, leading to a closed-form solution via a sparse linear system. We illustrate our method with a range of examples, demonstrating coherent natural-looking video output. The visual quality of individual frames is comparable to those produced by state-of-the-art methods for fisheye still photograph correction.

[1] J.P. Hornak, Encyclopedia of Imaging Science and Technology. Wiley-Interscience, 2002.
[2] A. Flocon and A. Barre, Curvilinear Perspective from Visual Space to the Constructed Image. Univ. of California Press, 1988.
[3] R. Carroll, M. Agrawala, and A. Agarwala, "Optimizing Content-Preserving Projections for Wide-Angle Images," ACM Trans. Graphics, vol. 28, no. 3, p. 43, 2009.
[4] J.P. Snyder, Flattening the Earth: Two Thousand Years of Map Projections. Univ. of Chicago Press, 1997.
[5] M. Kubovy, The Psychology of Perspective and Renaissance Art. Cambridge Univ. Press, 1988.
[6] M.H. Pirenne, Optics, Painting & Photography. Cambridge Univ. Press, 1970.
[7] D. Zorin and A.H. Barr, "Correction of Geometric Perceptual Distortions in Pictures," Proc. ACM SIGGRAPH, pp. 257-264, 1995.
[8] L. Zelnik-Manor, G. Peters, and P. Perona, "Squaring the Circle in Panoramas," Proc. IEEE Int'l Conf. Computer Vision (ICCV '05), pp. 1292-1299, 2005.
[9] J. Kopf, D. Lischinski, O. Deussen, D. Cohen-Or, and M. Cohen, "Locally Adapted Projections to Reduce Panorama Distortions," Computer Graphics Forum, vol. 28, no. 4, pp. 1083-1089, 2009.
[10] R. Swaminathan and S. Nayar, "Nonmetric Calibration of Wide-Angle Lenses and Polycameras," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 10, pp. 1172-1178, Oct. 2000.
[11] F. Devernay and O. Faugeras, "Straight Lines Have to Be Straight - Automatic Calibration and Removal of Distortion from Scenes of Structured Environments," Machine Vision and Applications, vol. 13, no. 1, pp. 14-24, 2001.
[12] J. Kannala and S.S. Brandt, "A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 8, pp. 1335-1340, Aug. 2006.
[13] R. Hartley and S.B. Kang, "Parameter-Free Radial Distortion Correction with Center of Distortion Estimation," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 29, no. 8, pp. 1309-1321, Aug. 2007.
[14] G.-X. Zhang, M.-M. Cheng, S.-M. Hu, and R.R. Martin, "A Shape-Preserving Approach to Image Resizing," Computer Graphics Forum, vol. 28, no. 7, pp. 1897-1906, 2009.
[15] Y.-S. Wang, H. Fu, O. Sorkine, T.-Y. Lee, and H.-P. Seidel, "Motion-Aware Temporal Coherence for Video Resizing," ACM Trans. Graphics, vol. 28, no. 5, p. 127, 2009.
[16] T. Igarashi, T. Moscovich, and J. Hughes, "As-Rigid-as-Possible Shape Manipulation," ACM Trans. Graphics, vol. 24, no. 3, pp. 1134-1141, 2005.
[17] S. Schaefer, T. McPhail, and J. Warren, "Image Deformation Using Moving Least Squares," ACM Trans. Graphics, vol. 25, no. 3, pp. 533-540, 2006.
[18] J. Huang, X. Shi, X. Liu, K. Zhou, L.-Y. Wei, S.-H. Teng, H. Bao, B. Guo, and H.-Y. Shum, "Subspace Gradient Domain Mesh Deformation," ACM Trans. Graphics, vol. 25, no. 3, pp. 1126-1134, 2006.
[19] M. Ben-Chen, O. Weber, and C. Gotsman, "Variational Harmonic Maps for Space Deformation," ACM Trans. Graphics, vol. 28, no. 3, pp. 1-11, 2009.
[20] S.F. Ray, Applied Photographic Optics. Focal Press, 2002.
[21] P. Kraehenbuehl, M. Lang, A. Hornung, and M. Gross, "A System for Retargeting of Streaming Video," ACM Trans. Graphics, vol. 28, no. 5, p. 126, 2009.
[22] A. Yilmaz, O. Javed, and M. Shah, "Object Tracking: A Survey," ACM Computing Surveys, vol. 38, 2006.
[23] J.-Y. Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker Description of the Algorithm," 2000.
[24] B. Georgescu, S. Member, P. Meer, and S. Member, "Point Matching under Large Image Deformations and Illumination Changes," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 674-688, June 2004.
[25] T. Liu, J. Sun, N.-N. Zheng, X. Tang, and H.-Y. Shum, "Learning to Detect a Salient Object," Proc. IEEE Conf. Computer and Vision Pattern Recognition (CVPR), pp. 1-8, 2007.
[26] C.-L. Guo, Q. Ma, and L.-M. Zhang, "Spatio-Temporal Saliency Detection Using Phase Spectrum of Quaternion Fourier Transform," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008.
[27] S. Goferman, L. Zelnik-Manor, and A. Tal, "Context-Aware Saliency Detection," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2010.
[28] B.D. Lucas and T. Kanade, "An Iterative Image Registration Techniue with an Application to Stereo Vision," Proc. Imaging Understanding Workshop, pp. 121-130, 1981.
[29] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes: The Art of Scientific Computing, third ed. Cambridge Univ. Press, 2007.
[30] M. Floater and K. Hormann, "Surface Parameterization: A Tutorial and Survey," Proc. Advances in Multiresolution for Geometric Modelling, pp. 157-186, 2005.
[31] T.J.R. Hughes, Mathematical Foundations of Elasticity. Dover Publications, 1994.
[32] O.C. Zienkiewicz, R.L. Taylor, and J.Z. Zhu, The Finite Element Method: Its Basis and Fundamentals, sixth ed. Elsevier Ltd., 2005.

Index Terms:
Lenses,Streaming media,Cameras,Shape analysis,Optimization,Optical distortion,least-squares minimization.,Fisheye lens,wide-angle lens,video correction,distortion,mesh,optimization
Jin Wei, Chen-Feng Li, Shi-Min Hu, Ralph R. Martin, Chiew-Lan Tai, "Fisheye Video Correction," IEEE Transactions on Visualization and Computer Graphics, vol. 18, no. 10, pp. 1771-1783, Oct. 2012, doi:10.1109/TVCG.2011.130
Usage of this product signifies your acceptance of the Terms of Use.