The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.02 - Feb. (2013 vol.19)
pp: 210-224
P. Laffont , REVES, INRIA Sophia Antipolis, Sophia-Antipolis, France
A. Bousseau , REVES, INRIA Sophia Antipolis, Sophia-Antipolis, France
G. Drettakis , REVES, INRIA Sophia Antipolis, Sophia-Antipolis, France
ABSTRACT
Intrinsic images aim at separating an image into its reflectance and illumination components to facilitate further analysis or manipulation. This separation is severely ill posed and the most successful methods rely on user indications or precise geometry to resolve the ambiguities inherent to this problem. In this paper, we propose a method to estimate intrinsic images from multiple views of an outdoor scene without the need for precise geometry and with a few manual steps to calibrate the input. We use multiview stereo to automatically reconstruct a 3D point cloud of the scene. Although this point cloud is sparse and incomplete, we show that it provides the necessary information to compute plausible sky and indirect illumination at each 3D point. We then introduce an optimization method to estimate sun visibility over the point cloud. This algorithm compensates for the lack of accurate geometry and allows the extraction of precise shadows in the final image. We finally propagate the information computed over the sparse point cloud to every pixel in the photograph using image-guided propagation. Our propagation not only separates reflectance from illumination, but also decomposes the illumination into a sun, sky, and indirect layer. This rich decomposition allows novel image manipulations as demonstrated by our results.
INDEX TERMS
optimisation, geometry, image processing, image manipulations, rich intrinsic image decomposition, outdoor scenes, image separation, illumination components, geometry, optimization, point cloud, image-guided propagation, Lighting, Sun, Image reconstruction, Three dimensional displays, Geometry, Materials, Image color analysis, mean-shift algorithm, Intrinsic images, image-guided propagation, multiview stereo
CITATION
P. Laffont, A. Bousseau, G. Drettakis, "Rich Intrinsic Image Decomposition of Outdoor Scenes from Multiple Views", IEEE Transactions on Visualization & Computer Graphics, vol.19, no. 2, pp. 210-224, Feb. 2013, doi:10.1109/TVCG.2012.112
REFERENCES
[1] H. Barrow and J. Tenenbaum, "Recovering Intrinsic Scene Characteristics from Images," Computer Vision Systems, Academic Press, 1978.
[2] E.H. Land and J.J. McCann, "Lightness and Retinex Theory," J. Optical Soc. of Am., vol. 61, no. 1, pp. 1-11, 1971.
[3] A. Bousseau, S. Paris, and F. Durand, "User-Assisted Intrinsic Images," ACM Trans. Graphics, vol. 28, no. 5,article 130, 2009.
[4] Y. Weiss, "Deriving Intrinsic Images from Image Sequences," Proc. IEEE Int'l Conf. Computer Comm. (ICCV), pp. 68-75, 2001.
[5] Y. Yu, P. Debevec, J. Malik, and T. Hawkins, "Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs," Proc. ACM SIGGRAPH, pp. 215-224, 1999.
[6] P. Debevec, C. Tchou, A. Gardner, T. Hawkins, C. Poullis, J. Stumpfel, A. Jones, N. Yun, P. Einarsson, T. Lundgren, M. Fajardo, and P. Martinez, "Estimating Surface Reflectance Properties of a Complex Scene under Captured Natural Illumination," technical report, USC Inst. for Creative Tech nologies, 2004.
[7] N. Snavely, S.M. Seitz, and R. Szeliski, "Photo Tourism: Exploring Photo Collections in 3D," ACM Trans. Graphics, vol. 25, no. 3, pp. 835-846, 2006.
[8] Y. Furukawa and J. Ponce, "Accurate, Dense, and Robust Multi-View Stereopsis," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 32, no. 8, pp. 1362-1376, Aug. 2010.
[9] A. Levin, D. Lischinski, and Y. Weiss, "Colorization Using Optimization," ACM Trans. Graphics, vol. 23, no. 3, pp. 689-694, 2004.
[10] D. Comaniciu and P. Meer, "Mean Shift: A Robust Approach toward Feature Space Analysis," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 5, pp. 603-619, May 2002.
[11] B.K. Horn, Robot Vision, first ed. McGraw-Hill Higher Education, 1986.
[12] L. Shen and C. Yeo, "Intrinsic Image Decomposition Using a Local and Global Sparse Representation of Reflectance," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2011.
[13] M.F. Tappen, W.T. Freeman, and E.H. Adelson, "Recovering Intrinsic Images from a Single Image," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 9, pp. 1459-1472, Sept. 2005.
[14] L. Shen, P. Tan, and S. Lin, "Intrinsic Image Decomposition with Non-Local Texture Cues," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2008.
[15] R. Grosse, M.K. Johnson, E.H. Adelson, and W.T. Freeman, "Ground-Truth Data Set and Baseline Evaluations for Intrinsic Image Algorithms," Proc. Int'l Conf. Computer Vision (ICCV), 2009.
[16] K. Sunkavalli, W. Matusik, H. Pfister, and S. Rusinkiewicz, "Factored Time-Lapse Video," ACM Trans. Graphics, vol. 26, no. 3,article 101, 2007.
[17] W. Matusik, M. Loper, and H. Pfister, "Progressively-Refined Reflectance Functions from Natural Illumination," Proc. Eurographics Symp. Rendering, pp. 299-308, 2004.
[18] F. Melendez, M. Glencross, G.J. Ward, and R.J. Hubbold, "Relightable Buildings from Images," Proc. Eurographics: Special Area on Cultural Heritage, pp. 33-40, 2011.
[19] J. Shen, X. Yang, Y. Jia, and X. Li, "Intrinsic Images Using Optimization," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2011.
[20] M. Okabe, G. Zeng, Y. Matsushita, T. Igarashi, L. Quan, and H.-Y. Shum, "Single-View Relighting with Normal Map Painting," Proc. Pacific Graphics, pp. 27-34, 2006.
[21] R. Carroll, R. Ramamoorthi, and M. Agrawala, "Illumination Decomposition for Material Recoloring with Consistent Interreflections," ACM Trans. Graphics, vol. 30, no. 4,article 43, 2011.
[22] K. Karsch, V. Hedau, D. Forsyth, and D. Hoiem, "Rendering Synthetic Objects into Legacy Photographs," ACM Trans. Graphics, vol. 30, no. 6,article 157, 2011.
[23] Y. Yu and J. Malik, "Recovering Photometric Properties of Architectural Scenes from Photographs," Proc. ACM SIGGRAPH, pp. 207-217, 1998.
[24] C. Loscos, G. Drettakis, and L. Robert, "Interactive Virtual Relighting of Real Scenes," IEEE Trans. Visualization and Computer Graphics, vol. 6, no. 4, pp. 289-305, Oct.-Dec. 2000.
[25] J.T. Kajiya, "The Rendering Equation," Computer Graphics, vol. 20, no. 4, pp. 143-150, 1986.
[26] D. Lischinski, Z. Farbman, M. Uyttendaele, and R. Szeliski, "Interactive Local Adjustment of Tonal Values," ACM Trans. Graphics, vol. 25, no. 3, pp. 646-653, 2006.
[27] A. Levin, D. Lischinski, and Y. Weiss, "A Closed-Form Solution to Natural Image Matting," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 2, pp. 228-242, Feb. 2008.
[28] E. Hsu, T. Mertens, S. Paris, S. Avidan, and F. Durand, "Light Mixture Estimation for Spatially Varying White Balance," ACM Trans. Graphics, vol. 27, no. 3,article 70, 2008.
[29] G.D. Finlayson, S.D. Hordley, and M.S. Drew, "Removing Shadows from Images," Proc. European Conf. Computer Vision (ECCV), 2002.
[30] G.D. Finlayson, M.S. Drew, and C. Lu, "Intrinsic Images by Entropy Minimization," Proc. European Conf. Computer Vision (ECCV), pp. 582-595, 2004.
[31] A. Mohan, J. Tumblin, and P. Choudhury, "Editing Soft Shadows in a Digital Photograph," IEEE Computer Graphics and Applications, vol. 27, no. 2, pp. 23-31, Mar. 2007.
[32] T.-P. Wu, C.-K. Tang, M.S. Brown, and H.-Y. Shum, "Natural Shadow Matting," ACM Trans. Graphics, vol. 26, no. 2,article 8, 2007.
[33] Y. Shor and D. Lischinski, "The Shadow Meets the Mask: Pyramid-Based Shadow Removal," Computer Graphics Forum, vol. 27, no. 2, pp. 577-586, 2008.
[34] P. Debevec and J. Malik, "Recovering High Dynamic Range Radiance Maps from Photographs," Proc. ACM SIGGRAPH, pp. 369-378, 1997.
[35] H. Hoppe, T. DeRose, T. Duchamp, J. McDonald, and W. Stuetzle, "Surface Reconstruction from Unorganized Points," Computer Graphics, vol. 26, no. 2, pp. 71-78, 1992.
[36] J. Digne, J.-M. Morel, C.-M. Souzani, and C. Lartigue, "Scale Space Meshing of Raw Data Point Sets," Computer Graphics Forum, vol. 6, no. 4, pp. 1630-1642, 2011.
[37] P. Cignoni, M. Corsini, and G. Ranzuglia, "Meshlab: An Open-Source 3D Mesh Processing System," ERCIM News, vol. 73, pp. 45-46, 2008.
[38] M. Kazhdan, M. Bolitho, and H. Hoppe, "Poisson Surface Reconstruction," Proc. Eurographics Symp. Geometry Processing, pp. 61-70, 2006.
[39] S. Sinha, D. Steedly, and R. Szeliski, "Piecewise Planar Stereo for Image-Based Rendering," Proc. Int'l Conf. Computer Vision (ICCV), pp. 1881-1888, 2009.
[40] Y. Furukawa, B. Curless, S.M. Seitz, and R. Szeliski, "Manhattan-World Stereo," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1422-1429, 2009.
[41] D. Gallup, J.-M. Frahm, and M. Pollefeys, "Piecewise Planar and Non-Planar Stereo for Urban Scene Reconstruction," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 1418-1425, 2010.
[42] M. Pharr and G. Humphreys, Physically Based Rendering: From Theory to Implementation, second ed. Morgan Kaufmann Publishers, Inc., 2010.
[43] K. Fukunaga and L. Hostetler, "The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition," IEEE Trans. Information Theory, vol. IT-21, no. 1, pp. 32-40, Jan. 1975.
[44] I. Omer and M. Werman, "Color Lines: Image Specific Color Representation," Proc. IEEE CS Conf. Computer Vision and Pattern Recognition (CVPR), pp. 946-953, 2004.
[45] D. Comaniciu, V. Ramesh, and P. Meer, "Real-Time Tracking of Non-Rigid Objects Using Mean Shift," Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pp. 142-149, 2000.
[46] C. Buehler, M. Bosse, L. McMillan, S. Gortler, and M. Cohen, "Unstructured Lumigraph Rendering," Proc. ACM SIGGRAPH, pp. 425-432, 2001.
39 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool