This Article 
 Bibliographic References 
 Add to: 
Interactive Virtual Relighting of Real Scenes
October-December 2000 (vol. 6 no. 4)
pp. 289-305

Abstract—Computer augmented reality (CAR) is a rapidly emerging field which enables users to mix real and virtual worlds. Our goal is to provide interactive tools to perform common illumination, i.e., light interactions between real and virtual objects, including shadows and relighting (real and virtual light source modification). In particular, we concentrate on virtually modifying real light source intensities and inserting virtual lights and objects into a real scene; such changes can be very useful for virtual lighting design and prototyping. To achieve this, we present a three-step method. We first reconstruct a simplified representation of real scene geometry using semiautomatic vision-based techniques. With the simplified geometry, and by adapting recent hierarchical radiosity algorithms, we construct an approximation of real scene light exchanges. We next perform a preprocessing step, based on the radiosity system, to create unoccluded illumination textures. These replace the original scene textures which contained real light effects such as shadows from real lights. This texture is then modulated by a ratio of the radiosity (which can be changed) over a display factor which corresponds to the radiosity for which occlusion has been ignored. Since our goal is to achieve a convincing relighting effect, rather than an accurate solution, we present a heuristic correction process which results in visually plausible renderings. Finally, we perform an interactive process to compute new illumination with modified real and virtual light intensities. Our results show that we are able to virtually relight real scenes interactively, including modifications and additions of virtual light sources and objects.

[1] TotalCalib/
[2] R.T. Azuma, “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments 6, pp. 355-385, Aug. 1997, earlier version in “Course Notes #9: Developing Advanced Virtual Reality Applications,” ACM SIGGRAPH, pp. 20-1-20-38, 1995.
[3] M.F. Cohen, S.E. Chen, J.R. Wallace, and D.P. Greenberg, "A Progressive Refinement Approach to Fast Radiosity Image Generation," Proc. SIGGRAPH 88, pp. 75-84, 1988.
[4] P.E. Debevec, "Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-based Graphics with Global Illumination and High Dynamic Range Photography," Proc. SIGGRAPH 98, pp. 189-198, July 1998.
[5] P.E. Debevec, G. Borshukov, and Y. Yu, “Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping,” Proc. Rendering Techniques '98 (Ninth EG Workshop Rendering), G. Drettakis and N. Max, eds., June 1998.
[6] P. Debevec and J. Malik, “Recovering High Dynamic Range Radiance Maps from Photographs,” Proc. SIGGRAPH Computer Graphics, vol. 31, pp. 369-378, 1997.
[7] P.E. Debevec, C.J. Taylor, and J. Malik, “Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach,” Proc. SIGGRAPH '96, pp. 11-20, Aug. 1996.
[8] G. Drettakis, L. Robert, and S. Bougnoux, "Interactive Common Illumination for Computer Augmented Reality," Proc. Eighth Eurographics Workshop Rendering, pp. 45-57, 1997.
[9] G. Drettakis and F.X. Sillion, “Interactive Update of Global Illumination Using a Line-Space Hierarchy,” Computer Graphics (ACM SIGGRAPH '97 Proc.), vol. 31, pp. 57-64, 1997.
[10] M.D. Fairchild, Color Appearance Models. Addison-Wesley, 1998.
[11] O. Faugeras, S. Laveau, L. Robert, G. Csurka, C. Zeller, C. Gauclin, and I. Zoghlami, “3-D Reconstruction of Urban Scenes from Image Sequences,” CVGIP: Image Understanding, 1997.
[12] A. Fournier, A.S. Gunawan, and C. Romanzin, “Common Illumination between Real and Computer Generated Scenes,” Proc. Graphics Interface '93, pp. 254-262, May 1993.
[13] S, Gibson and R.J. Hubbold, “Efficient Hierarchical Refinement and Clustering for Radiosity in Complex Environments,” Computer Graphics Forum, vol. 15, no. 5, pp. 297-310, 1996.
[14] E. Haines and J. Wallace, “Shaft Culling for Efficient Ray-Traced Radiosity,” Photorealistic Rendering Techniques in Computer Graphics (Proc. Second EG Workshop Rendering), P. Brunet and F.W. Jansen, eds., pp. 122-138, 1991.
[15] P. Hanrahan, D. Saltzman, and L. Aupperle, "A Rapid Hierarchical Radiosity Algorithm," Computer Graphics, SIGGRAPH '91,Las Vegas, vol. 25, no. 4, pp. 197-206, Aug. 1991.
[16] P. Jancène, F. Neyret, X. Provot, J.-P. Tarel, J.-M. Vézien, C. Meilhac, and A. Vérroust, “Res: Computing the Interactions between Real and Virtual Objects in Video Sequences,” Proc. Second IEEE Workshop Networked Realities, pp. 27-40, Oct. 1995. .
[17] D. Lischinski, F. Tampieri, and D.P. Greenberg, "Combining Hierarchical Radiosity and Discontinuity Meshing," Computer Graphics Proc., Ann. Conf. Series: SIGGRAPH '93,Anaheim, Calif., pp. 199-208. ACM, Aug. 1993.
[18] C. Loscos, M.-C. Frasson, G. Drettakis, B. Walter, X. Granier, and P. Poulin, “Interactive Virtual Relighting and Remodeling of Real Scenes,” Proc. Rendering Techniques '99 (10th EG Workshop Rendering), D. Lischinski and G. Ward-Larson, eds., June 1999.
[19] E. Nakamae, K. Harada, T. Ishizaki, and T. Nishita, “A Montage Method: The Overlaying of the Computer Generated Images onto a Background Photograph,” Computer Graphics (SIGGRAPH '86 Proc.), D.C. Evans and R.J. Athay, eds., vol. 20, pp. 207-214, Aug. 1986.
[20] P. Poulin, M. Ouimet, and M.-C. Frasson, “Interactively Modeling with Photogrammetry,” Proc. Rendering Techniques '98 (Ninth EG Workshop Rendering), G. Drettakis and N. Max, eds., pp. 93-104, June 1998.
[21] L. Robert, “Camera Calibration without Feature Extraction,” Computer Vision and Image Understanding, vol. 63, no. 2, pp. 314-325, 1996.
[22] I. Sato, Y. Sato, and K. Ikeuchi, “Acquiring a Radiance Distribution to Superimpose Virtual Objects onto a Real Scene,” IEEE Trans. Visualization and Computer Graphics, vol. 5, no. 1, pp. 1-12, 1999.
[23] F.X. Sillion, “A Unified Hierarchical Algorithm for Global Ilumination with Scattering Volumes and Object Clusters,” IEEE Trans. Visualization and Computer Graphics, vol. 1, no. 3, pp. 240-254, Sept. 1995.
[24] F.X. Sillion, G. Drettakis, and C. Soler, “A Clustering Algorithm for Radiance Calculation in General Environments,” Proc. Rendering Techniques '95 (Fifth EG Workshop Rendering), P.M. Hanrahan and W. Purgathofer, eds., pp. 196-205, June 1995.
[25] F.X. Sillion and C. Puech, Radiosity and Global Illumination. San Francisco: Morgan Kaufmann, 1994.
[26] B. Smits, J. Arvo, and D. Greenberg, “A Clustering Algorithm for Radiosity in Complex Environments,” Computer Graphics Proc., Ann. Conf. Series (ACM SIGGRAPH '94 Proc.), pp. 435-442, 1994.
[27] C. Soler and F. Sillion, “Automatic Calculation of Soft Shadow Textures for Fast, High-Quality Radiosity,” Proc. Rendering Techniques '98 (Ninthth EG Workshop Rendering), G. Drettakis and N. Max, eds., June 1998.
[28] A. State, G. Hirota, D.T. Chen, W.F. Garrett, and M.A. Livingston, “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking,” Proc. 23rd Int'l Conf. Computer Graphics and Interactive Techniques (SIGGRAPH '96), pp. 429-438, Aug. 1996.
[29] G.J. Ward, "The RADIANCE Lighting Simulation and Rendering System," Computer Graphics, pp. 459-472, July 1994.
[30] Y. Yu, P. Debevec, J. Malik, and T. Hawkins, “Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs,” Proc. SIGGRAPH '99, pp. 215-224, July 1998.
[31] Y. Yu and J. Malik, “Recovering Photometric Properties of Architectural Scenes from Photographs,” Proc. SIGGRAPH '98, pp. 207-217, July 1998.
[32] Z. Zhang, R. Deriche, O. Faugeras, and Q.T. Luong, “A Rubust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry,” Artificial Intelligence J., vol. 78, pp. 87-119, 1995.
[33] I. Zoghlami, O. Faugeras, and R. Deriche, “Using Geometric Corners to Build a 2D Mosaic from a Set of Images” Proc. Int'l Conf. Computer Vision and Pattern Recognition, June 1997.

Index Terms:
Hierarchical radiosity, global illumination, interactivity, computer augmented reality, common illumination, virtual relighting.
Céline Loscos, George Drettakis, Luc Robert, "Interactive Virtual Relighting of Real Scenes," IEEE Transactions on Visualization and Computer Graphics, vol. 6, no. 4, pp. 289-305, Oct.-Dec. 2000, doi:10.1109/2945.895874
Usage of this product signifies your acceptance of the Terms of Use.