This Article 
 Bibliographic References 
 Add to: 
Space-Time Super-Resolution
April 2005 (vol. 27 no. 4)
pp. 531-545
We propose a method for constructing a video sequence of high space-time resolution by combining information from multiple low-resolution video sequences of the same dynamic scene. Super-resolution is performed simultaneously in time and in space. By "temporal super-resolution,” we mean recovering rapid dynamic events that occur faster than regular frame-rate. Such dynamic events are not visible (or else are observed incorrectly) in any of the input sequences, even if these are played in "slow-motion.” The spatial and temporal dimensions are very different in nature, yet are interrelated. This leads to interesting visual trade-offs in time and space and to new video applications. These include: 1) treatment of spatial artifacts (e.g., motion-blur) by increasing the temporal resolution and 2) combination of input sequences of different space-time resolutions (e.g., NTSC, PAL, and even high quality still images) to generate a high quality video sequence. We further analyze and compare characteristics of temporal super-resolution to those of spatial super-resolution. These include: How many video cameras are needed to obtain increased resolution? What is the upper bound on resolution improvement via super-resolution? What is the temporal analogue to the spatial "ringing” effect?

[1] M.I. Sezan, A.J. Patti, and A.M. Tekalp, “Superresolution Video Reconstruction with Arbitrary Sampling Lattices and Nonzero Aperture Time,” IEEE Trans. Image Processing, vol. 6, pp. 1064-1076, Aug. 1997.
[2] A. Blake, B. Bascle, and A. Zisserman, “Motion Deblurring and Super-Resolution from an Image Sequence,” Proc. European Conf. Computer Vision, pp. 312-320, 1996.
[3] S. Baker and T. Kanade, “Limits on Super-Resolution and How to Break Them,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 9, Sept. 2002.
[4] S. Borman and R. Stevenson, “Spatial Resolution Enhancement of Low-Resolution Image Sequences— A Comprehensive Review with Directions for Future Research,” technical report, Laboratory for Image and Signal Analysis (LISA), Univ. of Notre Dame, July 1998.
[5] M. Born and E. Wolf, Principles of Optics. Permagon Press, 1965.
[6] D. Capel and A. Zisserman, “Automated Mosaicing with Super-Resolution Zoom,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, pp. 885-891, June 1998.
[7] D. Capel and A. Zisserman, “Super-Resolution Enhancement of Text Image Sequences,” Proc. Int'l Conf. Pattern Recognition, pp. 600-605, 2000.
[8] D.P. Capel, “Image Mosaicing and Super-Resolution,” PhD thesis, Dept. of Eng. Science, Univ. of Oxford, 2001.
[9] Y. Caspi and M. Irani, “Spatio-Temporal Aignment of Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 11, pp. 1409-1425, Nov. 2002.
[10] G. de Haan, “Progress in Motion Estimation for Video Format Conversion,” IEEE Trans. Consumer Electronics, vol. 46, no. 3, pp. 449-459, Aug. 2000.
[11] M. Elad, “Super-Resolution Reconstruction of Images,” PhD thesis, Technion Israel Inst. of Tech nology, Dec. 1996.
[12] H. Greenspan, G. Oz, N. Kiryati, and S. Peled, “MRI Inter-Slice Reconstruction Using Super Resolution,” Magnetic Resonance Imaging, vol. 20, pp. 437-446, 2002.
[13] T.S. Huang and R.Y. Tsai, “Multi-Frame Image Restoration and Registration,” Advances in Computer Vision and Image Processing, T.S. Huang, ed., vol. 1, pp. 317-339, JAI Press Inc., 1984.
[14] M. Irani and S. Peleg, “Improving Resolution by Image Registration,” CVGIP: Graphical Models and Image Processing, vol. 53, pp. 231-239, May 1991.
[15] M. Irani and S. Peleg, “Motion Analysis for Image Enhancement: Resolution, Occlusion, and Transparency,” J. Visual Comm. and Image Representation, vol. 4, pp. 324-335, Dec. 1993.
[16] J.R. Price, J. Shin, J. Paik, and M.A. Abidi, “Adaptive Regularized Image Interpolation Using Data Fusion and Steerable Constraints,” SPIE Visual Comm. and Image Processing, vol. 4310, Jan. 2001.
[17] V. Laude and C. Dirson, “Liquid-Crystal Active Lens: Application to Image Resolution Enhancement,” Optics Comm., vol. 163, pp. 72-78, May 1999.
[18] Z. Lin and H.Y. Shum, “On the Fundamental Limits of Reconstruction-Based Super-Resolution Algorithms,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2001.
[19] N. Nguyen, P. Milanfar, and G. Golub, “A Computationally Efficient Superresolution Image Reconstruction Algorithm,” IEEE Trans. Image Processing, vol. 10, no. 4, pp. 573-583, Apr. 2001.
[20] REALVIZ™, Retimer,, 2000.
[21] E. Shechtman, “Space-Time Super-Resolution in Video,” MSc thesis, Dept. of Computer Science and Applied Math., Weizmann Inst. Science, Feb. 2003.
[22] E. Shechtman, Y. Capsi, and M. Irani, “Increasing Space-Time Resolution in Video,” Proc. European Conf. Computer Vision, vol. 1, pp. 753-768, 2002.
[23] U. Trottenber, C. Oosterlee, and A. Schüller, Multigrid. Academic Press, Nov. 2000.

Index Terms:
Super-resolution, space-time analysis, temporal resolution, motion blur, motion aliasing, high-quality video, fast cameras.
Eli Shechtman, Yaron Caspi, Michal Irani, "Space-Time Super-Resolution," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 4, pp. 531-545, April 2005, doi:10.1109/TPAMI.2005.85
Usage of this product signifies your acceptance of the Terms of Use.