The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - Feb. (2014 vol.20)
pp: 262-275
Chenxi Zhang , Center for Visualization & Virtual Environments, Univ. of Kentucky, Lexington, KY, USA
Jizhou Gao , Center for Visualization & Virtual Environments, Univ. of Kentucky, Lexington, KY, USA
Oliver Wang , Disney Res. Zurich, Zurich, Switzerland
Pierre Georgel , Dept. of Comput. Sci., Univ. of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Ruigang Yang , Center for Visualization & Virtual Environments, Univ. of Kentucky, Lexington, KY, USA
James Davis , Dept. of Comput. Sci., Univ. of California at Santa Cruz, Santa Cruz, CA, USA
Jan-Michael Frahm , Dept. of Comput. Sci., Univ. of North Carolina at Chapel Hill, Chapel Hill, NC, USA
Marc Pollefeys , Dept. of Comput. Sci., ETH Zurich, Zurich, Switzerland
ABSTRACT
Given the growth of Internet photo collections, we now have a visual index of all major cities and tourist sites in the world. However, it is still a difficult task to capture that perfect shot with your own camera when visiting these places, especially when your camera itself has limitations, such as a limited field of view. In this paper, we propose a framework to overcome the imperfections of personal photographs of tourist sites using the rich information provided by large-scale Internet photo collections. Our method deploys state-of-the-art techniques for constructing initial 3D models from photo collections. The same techniques are then used to register personal photographs to these models, allowing us to augment personal 2D images with 3D information. This strong available scene prior allows us to address a number of traditionally challenging image enhancement techniques and achieve high-quality results using simple and robust algorithms. Specifically, we demonstrate automatic foreground segmentation, mono-to-stereo conversion, field-of-view expansion, photometric enhancement, and additionally automatic annotation with geolocation and tags. Our method clearly demonstrates some possible benefits of employing the rich information contained in online photo databases to efficiently enhance and augment one's own personal photographs.
INDEX TERMS
Internet, Photography, Image processing, Indexes,photometric enhancement, geotagging and locating, Image enhancement, Internet photo collections, segmentation, 2D to 3D conversion, field-of-view expansion
CITATION
Chenxi Zhang, Jizhou Gao, Oliver Wang, Pierre Georgel, Ruigang Yang, James Davis, Jan-Michael Frahm, Marc Pollefeys, "Personal Photograph Enhancement Using Internet Photo Collections", IEEE Transactions on Visualization & Computer Graphics, vol.20, no. 2, pp. 262-275, Feb. 2014, doi:10.1109/TVCG.2013.77
REFERENCES
[1] N. Snavely, S.M. Seitz, and R. Szeliski, “Photo Tourism: Exploring Photo Collections in 3D,” ACM Trans. Graphics, vol. 25, no. 3, pp. 835-846, 2006.
[2] Y. Furukawa, B. Curless, S.M. Seitz, and R. Szeliski, “Towards Internet-Scale Multi-View Stereo,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1434-1441, 2010.
[3] J.-M. Frahm, P.F. Georgel, D. Gallup, T. Johnson, R. Raguram, C. Wu, Y.H. Jen, E. Dunn, B. Clipp, and S. Lazebnik, “Building Rome on a Cloudless Day,” Proc. European Conf. Computer Vision (ECCV), pp. 368-381, 2010.
[4] R. Kaminsky, N. Snavely, S.M. Seitz, and R. Szeliski, “Alignment of Third Point Clouds to Overhead Images,” Proc. Workshop Internet Vision in Conjunction with CVPR, 2009.
[5] S. Agarwal, N. Snavely, I. Simon, S.M. Seitz, and R. Szeliski, “Building Rome in a Day,” Proc. Int'l Conf. Computer Vision (ICCV), pp. 72-79, 2009.
[6] X. Liu, L. Wan, Y. Qu, T.-T. Wong, S. Lin, C.-S. Leung, and P.-A. Heng, “Intrinsic Colorization,” ACM Trans. Graphics, vol. 27, no. 5, pp. 152:1-152:9, Dec. 2008.
[7] Y.S. Chia, S. Zhuo, R.K. Gupta, Y.-W. Tai, S.-Y. Cho, P. Tan, and S. Lin, “Semantic Colorization with Internet Images,” ACM Trans. Graphics, vol. 30, article 156, 2011.
[8] M.K. Johnson, K. Dale, S. Avidan, H. Pfister, W.T. Freeman, and W. Matusik, “CG2real: Improving the Realism of Computer Generated Images Using a Large Collection of Photographs,” IEEE Trans. Visualization and Computer Graphics, vol. 17, no. 9, pp. 1273-1285, Sept. 2011.
[9] N. Joshi, W. Matusik, E. Adelson, and D. Kriegman, “Personal Photo Enhancement Using Example Images,” ACM Trans. Graphics, vol. 29, no. 2, pp. 1-15, 2010.
[10] J. Hays and A.A. Efros, “Scene Completion Using Millions of Photographs,” Comm. ACM, vol. 51, no. 10, pp. 87-94, 2008.
[11] O. Whyte, J. Sivic, and A. Zisserman, “Get Out of My Picture! Internet-Based Inpainting,” Proc. British machine Vision Conf. (BMVC), 2009.
[12] R. Garg, H. Du, S.M. Seitz, and N. Snavely, “The Dimensionality of Scene Appearance,” Proc. Int'l Conf. Computer Vision (ICCV), pp. 1917-1924, 2009.
[13] K. Dale, M.K. Johnson, K. Sunkavalli, W. Matusik, and H. Pfister, “Image Restoration Using Online Photo Collections,” Proc. Int'l Conf. Computer Vision, pp. 2217-2224, 2009.
[14] P.-Y. Laffont, A. Bousseau, S. Paris, F. Durand, and G. Drettakis, “Coherent Intrinsic Images from Photo Collections,” Proc. ACM SIGGRAPH, 2012.
[15] C.L. Zitnick, S.B. Kang, M. Uyttendaele, S.A.J. Winder, and R. Szeliski, “High-Quality Video View Interpolation Using a Layered Representation,” ACM Trans. Graphics, vol. 23, no. 3, pp. 600-608, 2004.
[16] L. Zhang, W.J. Tam, and D. Wang, “Stereoscopic Image Generation Based on Depth Images,” Proc. Int'l Conf. Image Processing, pp. 2993-2996, 2004.
[17] O. Wang, M. Lang, M. Frei, A. Hornung, A. Smolic, and M.H. Gross, “Stereobrush: Interactive 2D to 3D Conversion Using Discontinuous Warps,” Proc. Eighth Eurographics Symp. Sketch-Based Interfaces and Modeling, pp. 47-54, 2011.
[18] J. Hays and A.A. Efros, “IM2GPS: Estimating Geographic Information from a Single Image,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1-8, 2008.
[19] E. Kalogerakis, O. Vesselova, J. Hays, A.A. Efros, and A. Hertzmann, “Image Sequence Geolocation with Human Travel Priors,” Proc. IEEE 12th Int'l Conf. Computer Vision, pp. 253-260, 2009.
[20] J. Kopf, B. Neubert, B. Chen, M.F. Cohen, D. Cohen-Or, O. Deussen, M. Uyttendaele, and D. Lischinski, “Deep Photo: Model-Based Photograph Enhancement and Viewing,” ACM Trans. Graphics, vol. 27, no. 5,article 116, 2008.
[21] C. Rother, V. Kolmogorov, and A. Blake, ““Grabcut”: Interactive Foreground Extraction Using Iterated Graph Cuts,” ACM Trans. Graphics, vol. 23, no. 3, pp. 309-314, 2004.
[22] V.S. Lempitsky, P. Kohli, C. Rother, and T. Sharp, “Image Segmentation with a Bounding Box Prior,” Proc. Int'l Conf. Computer Vision (ICCV), pp. 277-284, 2009.
[23] Y. Boykov and M.-P. Jolly, “Interactive Graph Cuts for Optimal Boundary and Region Segmentation of Objects in n-d Images,” Proc. Int'l Conf. Computer Vision (ICCV), pp. 105-112, 2001.
[24] Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum, “Lazy Snapping,” ACM Trans. Graphics, vol. 23, no. 3, pp. 303-308, 2004.
[25] M. Raginsky and S. Lazebnik, “Locality Sensitive Binary Codes from Shift-Invariant Kernels,” Proc. Advances in Neural Information Processing Systems (NIPS), 2009.
[26] D.G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints,” Int'l J. Computer Vision, vol. 60, no. 2, pp. 91-110, 2004.
[27] D. Comaniciu, P. Meer, and S. Member, “Mean Shift: A Robust Approach toward Feature Space Analysis,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[28] A. Irschara, C. Zach, J.-M. Frahm, and H. Bischof, “From Structure-from-Motion Point Clouds to Fast Location Recognition,” Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1-8, Mar. 2009.
[29] R. Raguram, J.-M. Frahm, and M. Pollefeys, “A Comparative Analysis of RANSAC Techniques Leading to Adaptive Real-Time Random Sample Consensus,” Proc. European Conf. Computer Vision, pp. 500-513, 2008.
[30] R.M. Haralick, C. Lee, K. Ottenberg, and M. Nölle, “Analysis and Solutions of the Three Point Perspective Pose Estimation Problem,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, 1991.
[31] G.R. Jones, D. Lee, N.S. Holliman, and D. Ezra, “Controlling Perceived Depth in Stereoscopic Images,” Proc. SPIE Stereoscopic Displays and Virtual Reality Systems VIII, vol. 4297, pp. 42-53, June 2001.
[32] P. Pérez, M. Gangnet, and A. Blake, “Poisson Image Editing,” ACM Trans. Graphics, vol. 22, no. 3, pp. 313-318, 2003.
[33] S. Avidan and A. Shamir, “Seam Carving for Content-Aware Image Resizing,” ACM Trans. Graphics, vol. 26, no. 3,article 10, 2007.
[34] A.A. Efros and W.T. Freeman, “Image Quilting for Texture Synthesis and Transfer,” Proc. ACM SIGGRAPH, pp. 341-346, Aug. 2001.
[35] K. Tuite, N. Snavely, D. yu Hsiao, N. Tabing, and Z. Popovic, “PhotoCity: Training Experts at Large-Scale Image Acquisition through a Competitive Game,” Proc. SIGCHI Conf. Human Factors in Computing Systems, pp. 1383-1392, 2011.
54 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool