This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Correction of Spatially Varying Image and Video Motion Blur Using a Hybrid Camera
June 2010 (vol. 32 no. 6)
pp. 1012-1028
Yu-Wing Tai, Korea Advanced Institute of Science and Technology (KAIST), Korea
Hao Du, University of Washington, Seattle
Michael S. Brown, National University of Singapore, Singapore
Stephen Lin, Microsoft Research Asia, Beijing
We describe a novel approach to reduce spatially varying motion blur in video and images using a hybrid camera system. A hybrid camera is a standard video camera that is coupled with an auxiliary low-resolution camera sharing the same optical path but capturing at a significantly higher frame rate. The auxiliary video is temporally sharper but at a lower resolution, while the lower frame-rate video has higher spatial resolution but is susceptible to motion blur. Our deblurring approach uses the data from these two video streams to reduce spatially varying motion blur in the high-resolution camera with a technique that combines both deconvolution and super-resolution. Our algorithm also incorporates a refinement of the spatially varying blur kernels to further improve results. Our approach can reduce motion blur from the high-resolution video as well as estimate new high-resolution frames at a higher frame rate. Experimental results on a variety of inputs demonstrate notable improvement over current state-of-the-art methods in image/video deblurring.

[1] M. Aggarwal and N. Ahuja, "Split Aperture Imaging for High Dynamic Range," Int'l J. Computer Vision, vol. 58, no. 1, pp. 7-17, 2004.
[2] A. Agrawal and R. Raskar, "Resolving Objects at Higher Resolution from a Single Motion-Blurred Image," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[3] S. Baker and T. Kanade, "Limits on Super-Resolution and How to Break Them," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 1167-1183, Sept. 2002.
[4] J. Bardsley, S. Jefferies, J. Nagy, and R. Plemmons, "Blind Iterative Restoration of Images with Spatially-Varying Blur," Optics Express, pp. 1767-1782, 2006.
[5] B. Bascle, A. Blake, and A. Zisserman, "Motion Deblurring and Super-Resolution from an Image Sequence," Proc. European Conf. Computer Vision, pp. 573-582, 1996.
[6] M. Ben-Ezra and S. Nayar, "Motion Deblurring Using Hybrid Imaging," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. I, pp. 657-664, June 2003.
[7] M. Ben-Ezra and S. Nayar, "Motion-Based Motion Deblurring," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 689-698, June 2004.
[8] P. Bhat, C.L. Zitnick, N. Snavely, A. Agarwala, M. Agrawala, B. Curless, M. Cohen, and S.B. Kang, "Using Photographs to Enhance Videos of a Static Scene," Proc. Eurographics Symp. Rendering, pp. 327-338, 2007.
[9] M. Bigas, E. Cabruja, J. Forest, and J. Salvi, "Review of cmos Image Sensors," Microelectronics J., vol. 37, no. 5, pp. 433-451, 2006.
[10] S. Borman and R. Stevenson, "Super-Resolution from Image Sequences—a Review," Proc. Midwest Symp. Circuits and Systems, p. 374, 1998.
[11] J. Chen and C.K. Tang, "Robust Dual Motion Deblurring," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[12] S. Cho, Y. Matsushita, and S. Lee, "Removing Non-Uniform Motion Blur from Images," Proc. Int'l Conf. Computer Vision, 2007.
[13] Y. Chuang, A. Agarwala, B. Curless, D.H. Salesin, and R. Szeliski, "Video Matting of Complex Scenes," ACM Trans. Graphics, pp. 243-248, 2002.
[14] Y. Chuang, B. Curless, D.H. Salesin, and R. Szeliski, "A Bayesian Approach to Digital Matting," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 264-271, 2001.
[15] N. Dey, L. Blanc-Fraud, C. Zimmer, Z. Kam, P. Roux, J. Olivo-Marin, and J. Zerubia, "A Deconvolution Method for Confocal Microscopy with Total Variation Regularization," Proc. IEEE Int'l Symp. Biomedical Imaging: Nano to Macro, 2004.
[16] M. Elad and A. Feuer, "Superresolution Restoration of an Image Sequence: Adaptive Filtering Approach," IEEE Trans. Image Processing, vol. 8, no. 3, pp. 387-395, 1999.
[17] R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis, and W.T. Freeman, "Removing Camera Shake from a Single Photograph," ACM Trans. Graphics, vol. 25, no. 3, pp. 787-794, 2006.
[18] D. Fish, A. Brinicombe, E. Pike, and J. Walker, "Blind Deconvolution by Means of the Richardson-Lucy Algorithm," J. Optical Soc. Am., vol. 12, 1995.
[19] R.C. Gonzalez and R.E. Woods, Digital Image Processing, second ed. Prentice Hall, 2002.
[20] P.C. Hansen, J.G. Nagy, and D.P. OLeary, Deblurring Images: Matrices, Spectra, and Filtering. SIAM, 2006.
[21] M. Irani and S. Peleg, "Improving Resolution by Image Registration," Proc. Conf. Computer Vision, Graphics and Image Processing, vol. 53, no. 3, pp. 231-239, 1991.
[22] M. Irani and S. Peleg, "Motion Analysis for Image Enhancement: Resolution, Occlusion and Transparency," J. Visual Comm. Image Representation, vol. 4, pp. 324-335, 1993.
[23] J. Jia, "Single Image Motion Deblurring Using Transparency," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007.
[24] N. Joshi, W. Matusik, and S. Avidan, "Natural Video Matting Using Camera Arrays," ACM Trans. Graphics, vol. 25, pp. 779-786, 2006.
[25] N. Joshi, R. Szeliski, and D. Kriegman, "PSF Estimation Using Sharp Edge Prediction," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[26] T. Lauer, "Deconvolution with a Spatially-Variant PSF," Astronomical Data Analysis II, vol. 4847, pp. 167-173, 2002.
[27] A. Levin, "Blind Motion Deblurring Using Image Statistics," Proc. Conf. Neural Information Processing Systems, pp. 841-848, 2006.
[28] A. Levin, R. Fergus, F. Durand, and W.T. Freeman, "Image and Depth from a Conventional Camera with a Coded Aperture," ACM Trans. Graphics, 2007.
[29] A. Levin, D. Lischinski, and Y. Weiss, "A Closed Form Solution to Natural Image Matting," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2006.
[30] A. Levin, P. Sand, T.S. Cho, F. Durand, and W.T. Freeman, "Motion-Invariant Photography," ACM Trans. Graphics, 2008.
[31] F. Li, J. Yu, and J. Chai, "A Hybrid Camera for Motion Deblurring and Depth Map Super-Resolution," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[32] B. Lucas and T. Kanade, "An Iterative Image Registration Technique with an Application to Stereo Vision," Proc. Imaging Understanding Workshop, pp. 121-130, 1981.
[33] L. Lucy, "An Iterative Technique for the Rectification of Observed Distributions," Astronomical J., vol. 79, p. 745, 1974.
[34] Y. Matsushita, E. Ofek, W. Ge, X. Tang, and H. Shum, "Full-Frame Video Stabilization with Motion Inpainting," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150-1163, July 2006.
[35] M. McGuire, W. Matusik, H. Pfister, J.F. Hughes, and F. Durand, "Defocus Video Matting," ACM Trans. Graphics, vol. 24, pp. 567-576, 2005.
[36] M. McGuire, W. Matusik, and W. Yerazunis, "Practical, Real-Time Studio Matting Using Dual Imagers," Proc. Eurographics Symp. Rendering, 2006.
[37] A. Patti, M. Sezan, and A.M. Tekalp, "Superresolution Video Reconstruction with Arbitrary Sampling Lattices and Nonzero Aperture Time," IEEE Trans. Image Processing, vol. 6, no. 8, pp. 1064-1076, Aug. 1997.
[38] R. Raskar, A. Agrawal, and J. Tumblin, "Coded Exposure Photography: Motion Deblurring Using Fluttered Shutter," ACM Trans. Graphics, vol. 25, no. 3, pp. 795-804, 2006.
[39] A. Rav-Acha and S. Peleg, "Two Motion Blurred Images Are Better than One," Pattern Recognition Letters, vol. 26, pp. 311-317, 2005.
[40] W. Richardson, "Bayesian-Based Iterative Method of Image Restoration," J. Optical Soc. Am., vol. 62, no. 1, pp. 55-59, 1972.
[41] Q. Shan, J. Jia, and A. Agarwala, "High-Quality Motion Deblurring from a Single Image," ACM Trans. Graphics, 2008.
[42] Q. Shan, W. Xiong, and J. Jia, "Rotational Motion Deblurring of a Rigid Object from a Single Image," Proc. Int'l Conf. Computer Vision, 2007.
[43] E. Shechtman, Y. Caspi, and M. Irani, "Space-Time Super-Resolution," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 4, pp. 531-544, Apr. 2005.
[44] A. Smith and J.F. Blinn, "Blue Screen Matting," Proc. ACM SIGGRAPH, 1996.
[45] F. Sroubek, G. Cristobal, and J. Flusser, "A Unified Approach to Superresolution and Multichannel Blind Deconvolution," IEEE Trans. Image Processing, vol. 16, no. 9, pp. 2322-2332, Sept. 2007.
[46] J. Sun, J. Jia, C. Tang, and H. Shum, "Poisson Matting," ACM Trans. Graphics, 2004.
[47] Y. Tai, H. Du, M. Brown, and S. Lin, "Image/Video Deblurring Using a Hybrid Camera," Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2008.
[48] J. Wang, M. Agrawala, and M. Cohen, "Soft Scissors: An Interactive Tool for Realtime High Quality Matting," ACM Trans. Graphics, 2007.
[49] J. Wang and M. Cohen, "An Iterative Optimization Approach for Unified Image Segmentation and Matting," Proc. Int'l Conf. Computer Vision, 2005.
[50] N. Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series. Wiley, 1949.
[51] J. Xiao, H. Cheng, H. Sawhney, C. Rao, and M. Isnardi, "Bilateral Filtering-Based Optical Flow Estimation with Occlusion Detection," Proc. European Conf. Computer Vision, 2006.
[52] L. Yuan, J. Sun, L. Quan, and H. Shum, "Image Deblurring with Blurred/Noisy Image Pairs," ACM Trans. Graphics, p. 1, 2007.
[53] L. Yuan, J. Sun, L. Quan, and H.-Y. Shum, "Progressive Inter-Scale and Intra-Scale Non-Blind Image Deconvolution," ACM Trans. Graphics, 2008.
[54] G. Zhang, J. Jia, W. Xiong, T. Wong, P. Heng, and H. Bao, "Moving Object Extraction with a Hand-Held Camera," Proc. Int'l Conf. Computer Vision, 2007.
[55] W. Zhao and H.S. Sawhney, "Is Super-Resolution with Optical Flow Feasible?" Proc. European Conf. Computer Vision, pp. 599-613, 2002.

Index Terms:
Motion deblurring, spatially varying motion blur, hybrid camera.
Citation:
Yu-Wing Tai, Hao Du, Michael S. Brown, Stephen Lin, "Correction of Spatially Varying Image and Video Motion Blur Using a Hybrid Camera," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 6, pp. 1012-1028, June 2010, doi:10.1109/TPAMI.2009.97
Usage of this product signifies your acceptance of the Terms of Use.