This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Video Repairing under Variable Illumination Using Cyclic Motions
May 2006 (vol. 28 no. 5)
pp. 832-839
Jiaya Jia, IEEE Computer Society
Yu-Wing Tai, IEEE Computer Society
Tai-Pang Wu, IEEE Computer Society
Chi-Keung Tang, IEEE Computer Society
This paper presents a complete system capable of synthesizing a large number of pixels that are missing due to occlusion or damage in an uncalibrated input video. These missing pixels may correspond to the static background or cyclic motions of the captured scene. Our system employs user-assisted video layer segmentation, while the main processing in video repair is fully automatic. The input video is first decomposed into the color and illumination videos. The necessary temporal consistency is maintained by tensor voting in the spatio-temporal domain. Missing colors and illumination of the background are synthesized by applying image repairing. Finally, the occluded motions are inferred by spatio-temporal alignment of collected samples at multiple scales. We experimented on our system with some difficult examples with variable illumination, where the capturing camera can be stationary or in motion.

[1] S. Baker, R. Szeliski, and P. Anandan, “A Layered Approach to Stereo Reconstruction,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 434-441, 1998.
[2] M. Bertalmio, A.L. Bertozzi, and G. Sapiro, “Navier-Stokes, Fluid Dynamics, and Image and Video Inpainting,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. I355-362, 2001.
[3] M. Bertalmio, G. Sapiro, C. Ballester, and V. Caselles, “Image Inpainting,” Proc. SIGGRAPH, pp. 417-424, 2000.
[4] M.J. Black and A.D. Jepson, “Estimating Optical-Flow in Segmented Images Using Variable-Order Parametric Models with Local Deformations,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 10, pp. 972-986, Oct. 1996.
[5] Y. Caspi and M. Irani, “A Step towards Sequence-to-Sequence Alignment,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. II682-689, 2000.
[6] Y.-Y. Chuang, A. Agarwala, B. Curless, D.H. Salesin, and R. Szeliski, “Video Matting of Complex Scenes,” Proc. 29th Ann. Conf. Computer Graphics and Interactive Techniques, pp. 243-248, 2002.
[7] D. Comaniciu, V. Ramesh, and P. Meer, “Real-Time Tracking of Non-Rigid Objects Using Mean Shift,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. II142-149, 2000.
[8] R. Cutler and L.S. Davis, “Robust Real-Time Periodic Motion Detection, Analysis, and Applications,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 781-796, Aug. 2000.
[9] J.E. Davis, “Mosaics of Scenes with Moving Objects,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 354-360, 1998.
[10] G. Doretto, A. Chiuso, Y.N. Wu, and S. Soatto, “Dynamic Textures,” Proc. Int'l Conf. Computer Vision, pp. II439-446, 2001.
[11] I. Drori, D. Cohen-Or, and H. Yeshurun, “Fragment-Based Image Completion,” Proc. SIGGRAPH, pp. 303-312, 2003.
[12] A. Efros and T.K. Leung, “Texture Synthesis by Non-Parametric Sampling,” Proc. Int'l Conf. Computer Vision, pp. 1033-1038, 1999.
[13] J. Jia and C.K. Tang, “Inference of Segmented Color and Texture Description by Tensor Voting,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 6, pp. 771-786, June 2004.
[14] J. Jia and C.K. Tang, “Tensor Voting for Image Correction by Global and Local Intensity Alignment,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 1, pp. 36-50, Jan. 2005.
[15] J. Jia, T.P. Wu, Y.W. Tai, and C.K. Tang, “Video Repairing: Inference of Foreground and Background under Severe Occlusion,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. I364-371, 2004.
[16] V. Kwatra, A. Schodl, I. Essa, G. Turk, and A. Bobick, “Graphcut Textures: Image and Video Synthesis Using Graph Cuts,” ACM Trans. Graphics, SIGGRAPH 2003, vol. 22, no. 3, pp. 277-286, July 2003.
[17] Y. Li, J. Sun, C.K. Tang, and H.Y. Shum, “Lazy Snapping,” ACM Trans. Graphics, vol. 23, no. 3, pp. 303-308, 2004.
[18] G. Medioni, M.S. Lee, and C.K. Tang, A Computational Framework for Feature Extraction and Segmentation. Amsderstam: Elsevier Science, 2000.
[19] A. Schodl, R. Szeliski, D. Salesin, and I. Essa, “Video Textures,” Proc. SIGGRAPH, pp. 489-498, 2000.
[20] S.M. Seitz and C.R. Dyer, “View Morphing,” Proc. 23rd Ann. Conf. Computer Graphics and Interactive Techniques, pp. 21-30, 1996.
[21] S.M. Seitz and C.R. Dyer, “View Invariant Analysis of Cyclic Motion,” Int'l J. Computer Vision, vol. 25, no. 3, pp. 231-251, Dec. 1997.
[22] R. Szeliski, “Video Mosaics for Virtual Environments,” IEEE Computer Graphics and Applications, pp. 22-30, Mar. 1996.
[23] J. Wang, Y. Xu, H.Y. Shum, and M.F. Cohen, “Video Tooning,” ACM Trans. Graphics, vol. 23, no. 3, pp. 574-583, 2004.
[24] J.Y.A. Wang and E.H. Adelson, “Layered Representation for Motion Analysis,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 361-366, 1993.
[25] Y. Weiss, “Deriving Intrinsic Images from Image Sequences,” Proc. Ninth IEEE Int'l Conf. Computer Vision, pp. 68-75, 2001.
[26] Y. Wexler, E. Shechtman, and M. Irani, “Space-Time Video Completion,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. I120-127, 2004.

Index Terms:
Video restoration, spatio-temporal consistence, illumination consistence, tensor voting, applications.
Citation:
Jiaya Jia, Yu-Wing Tai, Tai-Pang Wu, Chi-Keung Tang, "Video Repairing under Variable Illumination Using Cyclic Motions," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 5, pp. 832-839, May 2006, doi:10.1109/TPAMI.2006.108
Usage of this product signifies your acceptance of the Terms of Use.