This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Limits on Super-Resolution and How to Break Them
September 2002 (vol. 24 no. 9)
pp. 1167-1183

Abstract—Nearly all super-resolution algorithms are based on the fundamental constraints that the super-resolution image should generate the low resolution input images when appropriately warped and down-sampled to model the image formation process. (These reconstruction constraints are normally combined with some form of smoothness prior to regularize their solution.) In the first part of this paper, we derive a sequence of analytical results which show that the reconstruction constraints provide less and less useful information as the magnification factor increases. We also validate these results empirically and show that, for large enough magnification factors, any smoothness prior leads to overly smooth results with very little high-frequency content (however, many low resolution input images are used). In the second part of this paper, we propose a super-resolution algorithm that uses a different kind of constraint, in addition to the reconstruction constraints. The algorithm attempts to recognize local features in the low-resolution images and then enhances their resolution in an appropriate manner. We call such a super-resolution algorithm a hallucination or recogstruction algorithm. We tried our hallucination algorithm on two different data sets, frontal images of faces and printed Roman text. We obtained significantly better results than existing reconstruction-based algorithms, both qualitatively and in terms of RMS pixel error.

[1] S. Baker and T. Kanade, “Hallucinating Faces,” Technical Report CMU-RI-TR-99-32, The Robotics Inst., Carnegie Mellon Univ., 1999.
[2] S. Baker and T. Kanade, “Super-Resolution Optical Flow,” Technical Report CMU-RI-TR-99-36, The Robotics Inst., Carnegie Mellon Univ., 1999.
[3] S. Baker and T. Kanade, “Hallucinating Faces,” Proc. Fourth Int'l Conf. Automatic Face and Gesture Recognition, 2000.
[4] S. Baker and T. Kanade, “Limits on Super-Resolution and How to Break Them,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000.
[5] S. Baker, S.K. Nayar, and H. Murase, “Parametric Feature Detection,” Int'l J. Computer Vision, vol. 27, no. 1, pp. 27-50, 1998.
[6] D.F. Barbe, Charge-Coupled Devices. Springer-Verlag, 1980.
[7] B. Bascle, A. Blake, and A. Zisserman, “Motion Deblurring and Super-Resolution from an Image Sequence,” Proc. Fourth European Conf. Computer Vision, pp. 573-581, 1996.
[8] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hingorani, “Hiercharchical Model-Based Motion Estimation,” Proc. European Conf. Computer Vision, pp. 237-252, 1992.
[9] M. Berthod, H. Shekarforoush, M. Werman, and J. Zerubia, “Reconstruction of High Resolution 3D Visual Information,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 654-657, 1994.
[10] M. Born and E. Wolf, Principles of Optics. Permagon Press, 1965.
[11] P.J. Burt, “Fast Filter Transforms for Image Processing,” Computer Graphics and Image Processing, vol. 16, pp. 20-51, 1980.
[12] P.J. Burt and E.H. Adelson, “The Laplacian Pyramid as a Compact Image Code,” IEEE Trans. Comm., vol. 31, no. 4, pp. 532-540, 1983.
[13] P. Cheeseman, B. Kanefsky, R. Kraft, J. Stutz, and R. Hanson, “Super-Resolved Surface Reconstruction from Multiple Images,” Technical Report FIA-94-12, NASA Ames Research Center, 1994.
[14] M.-C. Chiang and T.E. Boult, “Imaging-Consistent Super-Resolution,” Proc. DARPA Image Understanding Workshop, 1997.
[15] M.-C. Chiang and T.E. Boult, “Local Blur Estimation and Super-Resolution,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 821-826, 1997.
[16] J.S. De Bonet, “Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images,” SIGGRAPH '97 Conf. Proc., pp. 361-368, 1997.
[17] J.S. De Bonet and P. Viola, “Texture Recognition Using a Non-Parametric Multi-Scale Statistical Model,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1998.
[18] F. Dellaert, S. Thrun, and C. Thorpe, “Jacobian Images of Super-Resolved Texture Maps for Model-Based Motion Estimation and Tracking,” Proc. Fourth Workshop Applications of Computer Vision, pp. 2-7, 1998.
[19] G.J. Edwards, C.J. Taylor, and T.F. Cootes, “Learning to Identify and Track Faces in Image Sequences,” Proc. Third Int'l Conf. Automatic Face and Gesture Recognition, pp. 260-265, 1998.
[20] M. Elad, “Super-Resolution Reconstruction of Image Sequences—Adaptive Filtering Approach,” PhD thesis, The Technion—Israel Inst. Technology, Haifa, Israel, 1996.
[21] P. Ciaccia, M. Patella, and P. Zezula, “Processing Complex Similarity Queries with Distance-Based Access Methods,” Proc. Sixth Int'l Conf. Extending Database Technology, Mar. 1998.
[22] M. Elad and A. Feuer, “Super-Resolution Reconstruction of Image Sequences,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 21, no. 9, pp. 817-834, Sept. 1999.
[23] M. Elad and A. Feuer, “Super-Resolution Restoration of Continuous Image Sequence—Adaptive Filtering Approach,” IEEE Trans. Image Processing, vol. 8, no. 3, pp. 387-395, Marc. 1999.
[24] W.T. Freeman and E.H. Adelson, "The Design and Use of Steerable Filters," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 13, pp. 891-906, 1991.
[25] W.T. Freeman, E.C. Pasztor, and O.T. Carmichael, “Learning Low-Level Vision,” Int'l J. Computer Vision, vol. 20, no. 1, pp. 25-47, 2000.
[26] R.C. Hardie, K.J. Barnard, and E.E. Armstrong, “Joint MAP Registration and High-Resolution Image Estimation Using a Sequence of Undersampled Images,” IEEE Trans. Image Processing, vol. 6, no. 12, pp. 1621-1633, 1997.
[27] B.K. Horn, Robot Vision. Cambridge, Mass.: MIT Press, 1986.
[28] T.S. Huang and R. Tsai, “Multi-Frame Image Restoration and Registration,” Advances in Computer Vision and Image Processing, vol. 1, pp. 317-339, 1984.
[29] M. Irani and S. Peleg, “Improving Resolution by Image Restoration,” Computer Vision, Graphics, and Image Processing, vol. 53, pp. 231-239, 1991.
[30] M. Irani and S. Peleg, “Motion Analysis for Image Enhancement: Resolution, Occulsion, and Transparency,” J. Visual Comm. and Image Representation, vol. 4, no. 4, pp. 324-335, 1993.
[31] M. Irani, B. Rousso, and S. Peleg, “Image Sequence Enhancement Using Multiple Motions Analysis,” Proc. 1992 Conf. Computer Vision and Pattern Recognition, pp. 216-221, 1992.
[32] D. Keren, S. Peleg, and R. Brada, “Image Sequence Enhancement Using Sub-Pixel Displacements,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 742-746, 1988.
[33] S. Kim, N. Bose, and H. Valenzuela, “Recursive Reconstruction of High Resolution Image from Noisy Undersampled Multiframes,” IEEE Trans. Acoustics, Speech, and Signal Processing, vol. 38, pp. 1013-1027, 1990.
[34] S. Kim and W.-Y. Su, “Recursive High-Resolution Reconstruction of Blurred Multiframe Images,” IEEE Trans. Image Processing, vol. 2, pp. 534-539, 1993.
[35] S. Mann and R. Picard, “Virtual Bellows: Constructing High Quality Stills from Video,” Proc. First IEEE Int'l Conf. Image Processing, vol. I, pp. 363-367, Nov. 1994.
[36] V.S. Nalwa, A Guided Tour of Computer Vision. Addison-Wesley, 1993.
[37] T. Numnonda, M. Andrews, and R. Kakarala, “High Resolution Image Reconstruction by Simulated Annealing,” Image and Vision Computing, vol. 11, no. 4, pp. 213-220, 1993.
[38] A. Patti, M. Sezan, and A. Tekalp, “Super-resolution Video Reconstruction with Arbitrary Sampling Latices and Nonzero Aperture Time,” IEEE Trans. Image Processing, vol. 6, no. 8, pp. 1064-1076, 1997.
[39] S. Peleg, D. Keren, and L. Schweitzer, “Improving Image Resolution Using Subpixel Motion,” Pattern Recognition Letters, pp. 223-226, 1987.
[40] P.J. Philips, H. Moon, P. Rauss, and S.A. Rizvi, The FERET Evaluation Methodology for Face-Recognition Algorithms Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1997.
[41] W.K. Pratt, Digital Image Processing. Wiley-Interscience, 1991.
[42] W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C, second ed. Cambridge Univ. Press, 1992.
[43] H. Qi and Q. Snyder, “Conditioning Analysis of Missing Data Estimation for Large Sensor Arrays,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2000.
[44] T. Riklin-Raviv and A. Shashua, “The Quotient Image: Class Based Recognition and Synthesis under Varying Illumination,” Proc. Conf. Computer Vision and Pattern Recognition, pp. 566-571, 1999.
[45] H. Rowley, S. Baluja, and T. Kanade, "Neural Network-Based Face Detection," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 1, Jan. 1998, pp. 23-38.
[46] R.R. Schultz and R.L. Stevenson, "A Bayesian Approach to Image Expansion for Improved Definition," IEEE Trans. Image Processing, vol. 3, 1994.
[47] R.S. Schultz and R.L. Stevenson, “Extraction of High-Resolution Frames from Video Sequences,” IEEE Trans. Image Processing, vol. 5, pp. 996-1,011, June 1996.
[48] H. Shekarforoush, “Conditioning Bounds for Multi-Frame Super-Resolution Algorithms,” Technical Report CAR-TR-912, Computer Vision Laboratory, Center for Automation Research, Univ. of Maryland, 1999.
[49] H. Shekarforoush, M. Berthod, J. Zerubia, and M. Werman, “Sub-Pixel Bayesian Estimation of Albedo and Height,” Int'l J. Computer Vision, vol. 19, pp. 289-300, 1996.
[50] V. Smelyanskiy, P. Cheeseman, D. Maluf, and R. Morris, “Bayesian Super-Resolved Surface Reconstruction from Images,” Proc. 2000 IEEE Conf. Computer Vision and Pattern Recognition, 2000.
[51] H. Stark and P. Oskoui, “High-Resolution Image Recovery from Image-Plane Arrays, Using Convex Projections,” J. Optical Soc. Am. A, vol. 6, pp. 1715-1726, 1989.
[52] R. Szeliski and P. Golland, “Stereo Matching with Transparency and Matting,” Proc. Sixth Int'l Conf. Computer Vision (ICCV '98), pp. 517-524, 1998.
[53] H. Ur and D. Gross, “Improved Resolution from Subpixel Shifted Pictures,” Computer Vision, Graphics, and Image Processing, vol. 54, no. 2, pp. 181-186, 1992.

Index Terms:
Super-resolution, analysis of reconstruction constraints, learning, faces, text, hallucination, recogstruction.
Citation:
Simon Baker, Takeo Kanade, "Limits on Super-Resolution and How to Break Them," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 9, pp. 1167-1183, Sept. 2002, doi:10.1109/TPAMI.2002.1033210
Usage of this product signifies your acceptance of the Terms of Use.