This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Full-Frame Video Stabilization with Motion Inpainting
July 2006 (vol. 28 no. 7)
pp. 1150-1163
ASCII Text
x 
 
Yasuyuki Matsushita, Eyal Ofek, Weina Ge, Xiaoou Tang, Heung-Yeung Shum, "Full-Frame Video Stabilization with Motion Inpainting," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150-1163, July, 2006.
 
 
BibTex
x
 
@article{ 10.1109/TPAMI.2006.141,
author = {Yasuyuki Matsushita and Eyal Ofek and Weina Ge and Xiaoou Tang and Heung-Yeung Shum},
title = {Full-Frame Video Stabilization with Motion Inpainting},
journal ={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {28},
number = {7},
issn = {0162-8828},
year = {2006},
pages = {1150-1163},
doi = {http://doi.ieeecomputersociety.org/10.1109/TPAMI.2006.141},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
TI - Full-Frame Video Stabilization with Motion Inpainting
IS - 7
SN - 0162-8828
SP1150
EP1163
EPD - 1150-1163
A1 - Yasuyuki Matsushita,
A1 - Eyal Ofek,
A1 - Weina Ge,
A1 - Xiaoou Tang,
A1 - Heung-Yeung Shum,
PY - 2006
KW - Video analysis
KW - video stabilization
KW - video completion
KW - motion inpainting
KW - sharpning and deblurring
KW - video enhancement.
VL - 28
JA - IEEE Transactions on Pattern Analysis and Machine Intelligence
ER -
 
Video stabilization is an important video enhancement technology which aims at removing annoying shaky motion from videos. We propose a practical and robust approach of video stabilization that produces full-frame stabilized videos with good visual quality. While most previous methods end up with producing smaller size stabilized videos, our completion method can produce full-frame videos by naturally filling in missing image parts by locally aligning image data of neighboring frames. To achieve this, motion inpainting is proposed to enforce spatial and temporal consistency of the completion in both static and dynamic image areas. In addition, image quality in the stabilized video is enhanced with a new practical deblurring algorithm. Instead of estimating point spread functions, our method transfers and interpolates sharper image pixels of neighboring frames to increase the sharpness of the frame. The proposed video completion and deblurring methods enabled us to develop a complete video stabilizer which can naturally keep the original image quality in the stabilized videos. The effectiveness of our method is confirmed by extensive experiments over a wide variety of videos.

[1] I. Zoghlami, O. Faugeras, and R. Deriche, “Using Geometric Corners to Build a 2D Mosaic from a Set of Images,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 420-425, 1997.
[2] D. Capel and A. Zisserman, “Automated Mosaicing with Super-Resolution Zoom,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 885-891, 1998.
[3] A. Cenci, A. Fusiello, and V. Roberto, “Image Stabilization by Feature Tracking,” Proc. 10th Int'l Conf. Image Analysis and Processing, pp. 665-670, 1999.
[4] M. Brown and D. Lowe, “Recognizing Panoramas,” Proc. Int'l Conf. Computer Vision, pp. 1218-1225, 2003.
[5] J. Bergen, P. Anandan, K. Hanna, and R. Hingorani, “Hierarchical Model-Based Motion Estimation,” Proc. Second European Conf. Computer Vision, pp. 237-252, 1992.
[6] R. Szeliski, “Image Mosaicing for Tele-Reality Applications,” technical report, CRL-Digital Equipment Corp., 1994.
[7] T.-J. Cham and R. Cipolla, “A Statistical Framework for Longrange Feature Matching in Uncalibrated Image Mosaicing,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 442-447, 1998.
[8] Z. Zhu, G. Xu, Y. Yang, and J. Jin, “Camera Stabilization Based on 2.5D Motion Estimation and Inertial Motion Filtering,” Proc. IEEE Int'l Conf. Intelligent Vehicles, vol. 2, pp. 329-334, 1998.
[9] R. Szeliski, “Image Alignment and Stitching: A Tutorial,” Technical Report MSR-TR-2004-92, Microsoft Corp., 2004.
[10] M. Hansen, P. Anandan, K. Dana, G. van der Wal, and P. Burt, “Real-Time Scene Stabilization and Mosaic Construction,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 54-62, 1994.
[11] C. Buehler, M. Bosse, and L. McMillan, “Non-Metric Image-Based Rendering for Video Stabilization,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 609-614, 2001.
[12] Z. Duric and A. Resenfeld, “Shooting a Smooth Video with a Shaky Camera,” J. Machine Vision and Applications, vol. 13, pp. 303-313, 2003.
[13] M. Pilu, “Video Stabilization as a Variational Problem and Numerical Solution with the Viterbi Method,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 625-630, 2004.
[14] A. Litvin, J. Konrad, and W. Karl, “Probabilistic Video Stabilization Using Kalman Filtering and Mosaicking,” Proc. IS&T/SPIE Symp. Electronic Imaging, Image, and Video Comm., pp. 663-674, 2003.
[15] Y. Wexler, E. Shechtman, and M. Irani, “Space-Time Video Completion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 120-127, 2004.
[16] J. Jia, T. Wu, Y. Tai, and C. Tang, “Video Repairing: Inference of Foreground and Background under Severe Occlusion,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 364-371, 2004.
[17] V. Cheung, B.J. Frey, and N. Jojic, “Video Epitomes,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 42-49, 2005.
[18] R. Fabian and D. Malah, “Robust Identification of Motion and Out-of-Focus Blur Parameters from Blurred and Noisy Images,” CVGIP: Graphical Models and Image Processing, vol. 53, no. 5, pp. 403-412, 1991.
[19] B. Bascle, A. Blake, and A. Zisserman, “Motion Deblurring and Super-Resolution from an Image Sequence,” Proc. Fourth European Conf. Computer Vision, vol. 2, pp. 573-582, 1996.
[20] P. Jansson, Deconvolution of Image and Spectra, second ed. Academic Press, 1997.
[21] A. Rav-Acha and S. Peleg, “Restoration of Multiple Images with Motion Blur in Different Directions,” Proc. Fifth IEEE Workshop Application of Computer Vision, pp. 22-28, 2000.
[22] Y. Yitzhaky, G. Boshusha, Y. Levy, and N. Kopeika, “Restoration of an Image Degraded by Vibrations Using Only a Single Frame,” Optical Eng., vol. 39, no. 8, pp. 2083-2091, 2000.
[23] M. Ben-Ezra and S. Nayar, “Motion-Based Motion Deblurring,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 689-698, June 2004.
[24] E. Adelson, “Depth-of-Focus Image Processing Method,” US Patent 4,661,986, 1987.
[25] J.R. Bergen, “Method and Apparatus for Extended Depth of Field Imaging,” US Patent 6,201,899, 2001.
[26] M. Ben-Ezra, A. Zomet, and S.K. Nayar, “Video Super-Resolution Using Controlled Subpixel Detector Shifts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 977-987, June 2005.
[27] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image Inpainting,” Proc. SIGGRAPH Conf., pp. 417-424, 2000.
[28] A. Criminisi, P. Perez, and K. Toyama, “Object Removal by Exemplar-Based Inpainting,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 721-728, 2003.
[29] H.-Y. Shum and R. Szeliski, “Construction of Panoramic Mosaics with Global and Local Alignment,” Int'l J. Computer Vision, vol. 36, no. 2, pp. 101-130, 2000.
[30] M. Irani and S. Peleg, “Improving Resolution by Image Registration,” CVGIP: Graphical Models and Image Processing, pp. 231-239, 1991.
[31] P. Anandan, “A Computational Framework and an Algorithm for the Measurement of Visual Motion,” Int'l J. Computer Vision, vol. 2, no. 3, pp. 283-310, 1989.
[32] B. Lucas and T. Kanade, “An Iterative Image Registration Technique with an Application to Stereo Vision,” Proc. Int'l Joint Conf. Artificial Intelligence, pp. 674-679, 1981.
[33] J. Bouguet, “Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of the Algorithm,” OpenCV Document, Intel, Microprocessor Research Labs, 2000.
[34] E. Krotkov, “Focusing,” Int'l J. Computer Vision, vol. 1, no. 3, pp. 223-237, 1987.
[35] N. Zhang, M. Postek, R. Larrabee, A. Vladar, W. Keery, and S. Jones, “Image Sharpness Measurement in the Scanning Electron Microscope,” The J. Scanning Microscopies, vol. 21, pp. 246-252, 1999.
[36] C. Tomasi and R. Manduchi, “Bilateral Filtering for Gray and Color Images,” Proc. Int'l Conf. Computer Vision, pp. 836-846, 1998.
[37] J. Foley, A. van Dam, S. Feiner, and J. Hughes, Computer Graphics: Principles and Practice. Addison-Wesley, 1996.
[38] J. Sethian, Level Set Methods: Evolving Interfaces in Geometry, Fluid Mechanics, Computer Vision and Materials Sciences. Cambridge Univ. Press, 1996.
[39] A. Telea, “An Image Inpainting Technique Based on the Fast Marching Method,” J. Graphics Tools, vol. 9, no. 1, pp. 23-34, 2004.
[40] A. Efros and T. Leung, “Texture Synthesis by Nonparametric Sampling,” Proc. Int'l Conf. Computer Vision, pp. 1033-1038, 1999.
[41] R. Strzodka and C. Garbe, “Real-Time Motion Estimation and Visualization on Graphics Cards,” Proc. IEEE Visualization Conf., pp. 545-552, 2004.
[42] C. Rother, V. Kolmogorov, and A. Blake, “‘Grabcut’: Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Trans. Graphics, vol. 23, no. 3, pp. 309-314, 2004.
[43] Y. Boykov and M. Jolly, “Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in N-D Images,” Proc. Int'l Conf. Computer Vision, pp. 105-112, 2001.
[44] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A.F. Bobick, “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” ACM Trans. Graphics, vol. 22, no. 3, pp. 277-286, 2003.

Index Terms:
Video analysis, video stabilization, video completion, motion inpainting, sharpning and deblurring, video enhancement.
Citation:
Yasuyuki Matsushita, Eyal Ofek, Weina Ge, Xiaoou Tang, Heung-Yeung Shum, "Full-Frame Video Stabilization with Motion Inpainting," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150-1163, July 2006, doi:10.1109/TPAMI.2006.141
Usage of this product signifies your acceptance of the Terms of Use.