The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.09 - September (2011 vol.17)
pp: 1273-1285
Micah K. Johnson , Massachusetts Institute of Technology, Cambridge
Kevin Dale , Harvard University, Cambridge
Shai Avidan , Tel Aviv University, Israel, and Adobe Systems
Hanspeter Pfister , Harvard University, Cambridge
William T. Freeman , Massachusetts Institute of Technology, Cambridge
Wojciech Matusik , Massachusetts Institute of Technology, Cambridge
ABSTRACT
Computer-generated (CG) images have achieved high levels of realism. This realism, however, comes at the cost of long and expensive manual modeling, and often humans can still distinguish between CG and real images. We introduce a new data-driven approach for rendering realistic imagery that uses a large collection of photographs gathered from online repositories. Given a CG image, we retrieve a small number of real images with similar global structure. We identify corresponding regions between the CG and real images using a mean-shift cosegmentation algorithm. The user can then automatically transfer color, tone, and texture from matching regions to the CG image. Our system only uses image processing operations and does not require a 3D model of the scene, making it fast and easy to integrate into digital content creation workflows. Results of a user study show that our hybrid images appear more realistic than the originals.
INDEX TERMS
Image enhancement, image databases, image-based rendering.
CITATION
Micah K. Johnson, Kevin Dale, Shai Avidan, Hanspeter Pfister, William T. Freeman, Wojciech Matusik, "CG2Real: Improving the Realism of Computer Generated Images Using a Large Collection of Photographs", IEEE Transactions on Visualization & Computer Graphics, vol.17, no. 9, pp. 1273-1285, September 2011, doi:10.1109/TVCG.2010.233
REFERENCES
[1] J. Hays and A.A. Efros, "Scene Completion Using Millions of Photographs," ACM Trans. Graphics, vol. 26, no. 3, pp. 1-7, Aug. 2007.
[2] D.J. Heeger and J.R. Bergen, "Pyramid-Based Texture Analysis/Synthesis," Proc. ACM SIGGRAPH, pp. 229-238, 1995.
[3] J.S. De Bonet, "Multiresolution Sampling Procedure for Analysis and Synthesis of Texture Images," Proc. ACM SIGGRAPH, pp. 361-368, 1997.
[4] A.A. Efros and T.K. Leung, "Texture Synthesis by Non-Parametric Sampling," Proc. IEEE Int'l Conf. Computer Vision, pp. 1033-1038, 1999.
[5] A.A. Efros and W.T. Freeman, "Image Quilting for Texture Synthesis and Transfer," Proc. ACM SIGGRAPH, pp. 341-346, 2001.
[6] V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick, "Graphcut Textures: Image and Video Synthesis Using Graph Cuts," ACM Trans. Graphics, vol. 22, no. 3, pp. 277-286, July 2003.
[7] A. Hertzmann, C.E. Jacobs, N. Oliver, B. Curless, and D.H. Salesin, "Image Analogies" Proc. ACM SIGGRAPH, pp. 327-340, Aug. 2001.
[8] W.T. Freeman, T.R. Jones, and E.C. Pasztor, "Example-Based Super-Resolution," IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56-65, Mar./Apr. 2002.
[9] R. Rosales, K. Achan, and B. Frey, "Unsupervised Image Translation," Proc. IEEE Int'l Conf. Computer Vision, pp. 472-478, 2003.
[10] E. Reinhard, M. Ashikhmin, B. Gooch, and P.S. Shirley, "Color Transfer between Images," IEEE Computer Graphics and Applications, vol. 21, no. 5, pp. 34-41, Sept./Oct. 2001.
[11] F. Pitié, A.C. Kokaram, and R. Dahyot, "N-Dimensional Probability Density Function Transfer and Its Application to Colour Transfer," Proc. IEEE Int'l Conf. Computer Vision, pp. 1434-1439, 2005.
[12] S. Bae, S. Paris, and F. Durand, "Two-Scale Tone Management for Photographic Look," ACM Trans. Graphics, vol. 25, no. 3, pp. 637-645, July 2006.
[13] C.-L. Wen, C.-H. Hsieh, B.-Y. Chen, and M. Ouhyoung, "Example-Based Multiple Local Color Transfer by Strokes," Computer Graphics Forum, vol. 27, no. 7, pp. 1765-1772, 2008.
[14] X. Liu, L. Wan, Y. Qu, T.-T. Wong, S. Lin, C.-S. Leung, and P.-A. Heng, "Intrinsic Colorization," ACM Trans. Graphics, vol. 27, no. 3, pp. 1-9, 2008.
[15] K. Dale, M.K. Johnson, K. Sunkavalli, W. Matusik, and H. Pfister, "Image Restoration Using Online Photo Collections," Proc. IEEE Int'l Conf. Computer Vision, pp. 2217-2224, 2009.
[16] J.-F. Lalonde, D. Hoiem, A.A. Efros, C. Rother, J. Winn, and A. Criminisi, "Photo Clip Art," ACM Trans. Graphics, vol. 26, no. 3, pp. 1-10, Aug. 2007.
[17] B.C. Russell, A. Torralba, K.P. Murphy, and W.T. Freeman, "Labelme: A Database and Web-Based Tool for Image Annotation," Int'l J. Computer Vision, vol. 77, nos. 1-3, pp. 157-173, May 2008.
[18] M. Johnson, G.J. Brostow, J. Shotton, O. Arandjelovic, V. Kwatra, and R. Cipolla, "Semantic Photo Synthesis," Computer Graphics Forum, vol. 25, no. 3, pp. 407-414, Sept. 2006.
[19] T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu, "Sketch2Photo: Internet Image Montage," ACM Trans. Graphics, vol. 28, no. 5, pp. 1-10, 2009.
[20] S. Lyu and H. Farid, "How Realistic is Photorealistic?," IEEE Trans. Signal Processing, vol. 53, no. 2, pp. 845-850, Feb. 2005.
[21] J.-F. Lalonde and A.A. Efros, "Using Color Compatibility for Assessing Image Realism," Proc. IEEE Int'l Conf. Computer Vision, pp. 1-8, 2007.
[22] Y. Weiss and W.T. Freeman, "What Makes a Good Model of Natural Images?" Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[23] S. Roth and M.J. Black, "Fields of Experts: A Framework for Learning Image Priors," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 860-867, 2005.
[24] B.C. Russell, A.A. Efros, J. Sivic, W.T. Freeman, and A. Zisserman, "Segmenting Scenes by Matching Image Composites," Proc. Advances in Neural Information Processing Systems, pp. 1580-1588, 2009.
[25] A. Oliva and A. Torralba, "Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope," Int'l J. Computer Vision, vol. 42, no. 3, pp. 145-175, 2001.
[26] J. Zhang, M. Marszalek, S. Lazebnik, and C. Schmid, "Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study," Int'l J. Computer Vision, vol. 73, no. 2, pp. 213-238, 2007.
[27] D.G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[28] S. Lazebnik, C. Schmid, and J. Ponce, "Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 2, pp. 2169-2178, 2006.
[29] K. Grauman and T. Darrell, "The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features," Proc. IEEE Int'l Conf. Computer Vision, vol. 2, pp. 1458-1465, 2005.
[30] Y.-W. Tai, J. Jia, and C.-K. Tang, "Local Color Transfer via Probabilistic Segmentation by Expectation-Maximization," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 747-754, 2006.
[31] C. Rother, T. Minka, A. Blake, and V. Kolmogorov, "Cosegmentation of Image Pairs by Histogram Matching-Incorporating a Global Constraint into MRFs," Proc. IEEE Conf. Computer Vision and Pattern Recognition, vol. 1, pp. 993-1000, 2006.
[32] K. Fukunaga and L.D. Hostetler, "The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition," IEEE Trans. Information Theory, vol. 21, no. 1, pp. 32-40, Jan. 1975.
[33] S. Paris and F. Durand, "A Topological Approach to Hierarchical Segmentation Using Mean Shift," Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2007.
[34] J. Kopf, M.F. Cohen, D. Lischinski, and M. Uyttendaele, "Joint Bilateral Upsampling," Proc. ACM SIGGRAPH, 2007.
[35] E.H. Adelson, E.P. Simoncelli, and R. Hingorani, "Orthogonal Pyramid Transforms for Image Coding," Proc. SPIE Visual Comm. and Image Processing II, vol. 845, pp. 50-58, 1987.
[36] Y. Li, L. Sharan, and E.H. Adelson, "Compressing and Companding High Dynamic Range Images with Subband Architectures," ACM Trans. Graphics, vol. 24, no. 3, pp. 836-844, 2005.
[37] P. Pérez, M. Gangnet, and A. Blake, "Poisson Image Editing," ACM Trans. Graphics, vol. 22, no. 3, pp. 313-318, July 2003.
[38] Y. Horry, K.-I. Anjyo, and K. Arai, "Tour into the Picture: Using a Spidery Mesh Interface to Make Animation from a Single Image," Proc. ACM SIGGRAPH, pp. 225-232, 1997.
[39] Amazon.com, "Amazon Mechanical Turk," http:/www.mturk. com, 2009.
20 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool