Subscribe
Issue No.05 - May (2011 vol.33)
pp: 995-1008
Uma Mudenagudi , BVB College of Engineering and Technology, Hubli
Subhashis Banerjee , IIT Delhi, New Delhi
Prem Kumar Kalra , IIT Delhi, New Delhi
ABSTRACT
We address the problem of super-resolution—obtaining high-resolution images and videos from multiple low-resolution inputs. The increased resolution can be in spatial or temporal dimensions, or even in both. We present a unified framework which uses a generative model of the imaging process and can address spatial super-resolution, space-time super-resolution, image deconvolution, single-image expansion, removal of noise, and image restoration. We model a high-resolution image or video as a Markov random field and use maximum a posteriori estimate as the final solution using graph-cut optimization technique. We derive insights into what super-resolution magnification factors are possible and the conditions necessary for super-resolution. We demonstrate spatial super-resolution reconstruction results with magnifications higher than predicted limits of magnification. We also formulate a scheme for selective super-resolution reconstruction of videos to obtain simultaneous increase of resolutions in both spatial and temporal directions. We show that it is possible to achieve space-time magnification factors beyond what has been suggested in the literature by selectively applying super-resolution constraints. We present results on both synthetic and real input sequences.
INDEX TERMS
Super-resolution, Markov random field (MRF), maximum a posteriori (MAP), graph-cut, space-time, nonlinear, minimization.
CITATION
Uma Mudenagudi, Subhashis Banerjee, Prem Kumar Kalra, "Space-Time Super-Resolution Using Graph-Cut Optimization", IEEE Transactions on Pattern Analysis & Machine Intelligence, vol.33, no. 5, pp. 995-1008, May 2011, doi:10.1109/TPAMI.2010.167
REFERENCES
 [1] S. Geman and D. Geman, “Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 6, no. 6, pp. 721-741, Nov. 1984. [2] Y. Boykov, O. Veksler, and R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1222-1239, Nov. 2001. [3] U. Mudenagudi, R. Singla, P. Kalra, and S. Banerjee, “Super Resolution Using Graph-Cut,” Proc. Asian Conf. Computer Vision, pp. 385-394, Jan. 2006. [4] U. Mudenagudi, S. Banerjee, and P. Kalra, “On Improving Space-Time Super Resolution Using a Small Set of Video Inputs,” Proc. Sixth Indian Conf. Computer Vision, Graphics and Image Processing, pp. 320-327, Dec. 2008. [5] M. Irani and S. Peleg, “Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency,” J. Visual Comm. and Image Representation, vol. 4, no. 4, pp. 324-335, Dec. 1993. [6] S. Borman and R. Stevenson, “Linear Models for Multi-Frame Super-Resolution Restoration under Non-Affine Registration and Spatially Varying PSF,” Proc. SPIE, C. Bouman and E. Miller, eds., pp. 234-245, Jan. 2004. [7] D. Rajan and S. Chaudhuri, “Simultaneous Estimation of Super-Resolved Scene and Depth Map for Low Resolution Defocused Observations,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 9, pp. 1102-1117, Sept. 2003. [8] M. Joshi and S. Chaudhuri, “A Learning Based Method for Image Super-Resolution from Zoomed Observations,” Proc. Fifth Int'l Conf. Advances in Pattern Recognition, pp. 179-182, 2003. [9] S. Chaudhuri and M.V. Joshi, Motion-Free Super-Resolution. Springer, 2004. [10] D. Glasner, S. Bagon, and M. Irani, “Super-Resolution from a Single Image,” Proc. IEEE Int'l Conf. Computer Vision, Oct. 2009. [11] H. Takeda, P. Milanfar, M. Protter, and M. Elad, “Super-Resolution without Explicit Subpixel Motion Estimation,” IEEE Trans. Image Processing, vol. 18, no. 9, pp. 1958-1975, Sept. 2009. [12] D. Mitzel, T. Pock, T. Schoenemann, and D. Cremers, “Video Super Resolution Using Duality Based ${TV-L^1}$ Optical Flow,” Proc. 31st DAGM Symp. Pattern Recognition, pp. 432-441, 2009. [13] S.H. Keller, F. Lauze, and M. Nielsen, “Motion Compensated Video Super Resolution,” Proc. First Int'l Conf. Scale Space and Variational Methods in Computer Vision, F. Sgallari et al., eds., pp. 801-812, 2007. [14] M. Ebrahimi and A.L. Martel, “A PDE Approach to Coupled Super-Resolution with Non-Parametric Motion,” Proc. Seventh Int'l Conf. Energy Minimization Methods in Computer Vision and Pattern Recognition D. Cremers et al., eds., pp. 112-125, 2009. [15] D. Capel and A. Zisserman, “Computer Vision Applied to Super Resolution,” IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 75-86, May 2003. [16] L.C. Pickup, S.J. Roberts, and A. Zisserman, “A Sampled Texture Prior for Image Super-Resolution,” Proc. Neural Information Processing Systems, 2003. [17] S. Baker and T. Kanade, “Limits on Super-Resolution and How to Break Them,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2000. [18] S. Baker and T. Kanade, “Limits on Super-Resolution and How to Break Them,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 1167-1183, Sept. 2002. [19] Z. Lin and H.-Y. Shum, “Fundamental Limits of Reconstruction-Based Super-Resolution Algorithms under Local Translation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 1, pp. 83-97, Jan. 2004. [20] W.T. Freeman, T.R. Jones, and E.C. Pasztor, “Example-Based Super-Resolution,” IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56-65, Mar./Apr. 2002. [21] E. Shechtman, Y. Caspi, and M. Irani, “Increasing Space-Time Resolution in Video,” Proc. Seventh European Conf. Computer Vision, vol. 1, pp. 753-768, 2002. [22] E. Shechtman and Y. Caspi, “Space-Time Super-Resolution,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 4, pp. 531-545, Apr. 2005. [23] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hingorani, “Hierarchical Model-Based Motion Estimation,” Proc. Second European Conf. Computer Vision, pp. 237-252, 1992. [24] Y. Caspi and M. Irani, “Spatio-Temporal Alignment of Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 11, pp. 1409-1421, Nov. 2002. [25] V. Kolmogorov and R. Zabih, “What Energy Functions Can Be Minimized via Graph Cuts?” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 26, no. 2, pp. 147-159, Feb. 2004. [26] A. Raj and R. Zabih, “A Graph Cut Algorithm for Generalized Image Deconvolution,” Proc. IEEE Int'l Conf. Computer Vision, Jan. 2005. [27] A. Raj, G. Singh, and R. Zabih, “MRFs for MRIs: Bayesian Reconstruction of MR Images via Graph Cuts,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, June 2006. [28] http://www.cs.ucl.ac.uk/staff/V.Kolmogorov software.html, 2009.