This Article 
 Bibliographic References 
 Add to: 
Image-Based Interactive Exploration of Real-World Environments
May/June 2004 (vol. 24 no. 3)
pp. 52-63
Matthew Uyttendaele, Microsoft Research
Antonio Criminisi, Microsoft Research
Sing Bing Kang, Microsoft Research
Simon Winder, Microsoft Research
Richard Szeliski, Microsoft Research
Richard Hartley, Australian National University and National ICT Australia
This article describes an image-based acquisition and rendering system that enables users to explore remote real-world locations. The system is based on a lightweight, high-resolution, multisensor camera that can be used to quickly film a tour through a large environment such as a house or a garden. During the offline authoring phase, the raw video is processed to produce stabilized high-quality, high-dynamic-range video using a combination of novel techniques. These include a stitching algorithm to remove parallax in areas of image overlap, registration and blending algorithms to produce high dynamic range video from alternately exposed video frames, feature tracking to perform camera stabilization, and a video compression scheme that supports selective runtime decompression and random access. The final user experience is enhanced with multimedia elements such as overview maps, video textures, and sound.

The user controls the viewpoint and location through a standard game controller, and the resulting user experience feels much like a computer game. The combination of high-resolution, continuous imagery with real-time interactivity provides viewers with an unprecedented sense of presence in interesting real world environments.

1. F.P. Brooks, "Walkthrough—A Dynamic Graphics System for Simulating Virtual Buildings," Proc. Workshop Interactive 3D Graphics, ACM Press, 1986, pp. 9-21.
2. P.E. Debevec, C.J. Taylor, and J. Malik, "Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach," Proc. Siggraph, ACM Press, 1996, pp. 11-20.
3. A. Schödl et al., "Video Textures," Proc. Siggraph, ACM Press, 2000, pp. 489-498.
4. S. Teller et al., "Calibrated, Registered Images of an Extended Urban Area," Int'l J. Computer Vision, vol. 53, no. 1, June 2003, pp. 93-107.
5. E. Chang et al., "Color Filter Array Recovery Using a Threshold-Based Variable Number of Gradients," SPIE vol. 3650, Sensors, Cameras, and Applications for Digital Photography, SPIE, Mar. 1999, pp. 36-43.
6. R. Szeliski and H.-Y. Shum, "Creating Full View Panoramic Image Mosaics and Texture-Mapped Models," Proc. Siggraph 97, ACM Press, 1997, pp. 251-258.
7. M. Uyttendaele et al., High-Quality Image-Based Interactive Exploration of Real-World Environments, tech. report MSR-TR-2003-61, Microsoft Research, Oct. 2003.
8. C. Schmid, R. Mohr, and C. Bauckhage, "Comparing and Evaluating Interest Points," Proc. 6th Int'l Conf. Computer Vision (ICCV 98), IEEE CS Press, 1998, pp. 230-235.
9. C. Buehler, A. Bosse, and L. McMillan, "Non-metric Image-Based Rendering for Video Stabilization," Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition (CVPR 01), vol. 2, IEEE CS Press, 2001, pp. 609-614.
1. S.E. Chen, "QuickTime VR: An Image-Based Approach to Virtual Environment Navigation," Proc. Siggraph, ACM Press, 1995, pp. 29-38.
2. PanoramicVision: Sensors, Theory, and Applications, R. Benosman and S.B. Kang, eds., Springer, 2001.
3. A. Lippman, "Moviemaps: An Application of the Optical Videodisc to Computer Graphics," Proc. Siggraph, ACM Press, vol. 14, no. 3, 1980, pp. 32-43.
4. T.E. Boult, "Remote Reality via Omnidirectional Imaging," Proc. Siggraph 1998 Technical Sketch, ACM Press, 1998, p. 253.
5. S. Coorg and S. Teller, "Extracting Textured Vertical Facades from Controlled Close-Range Imagery," Proc. IEEE Computer Soc. Conf. Computer Vision and Pattern Recognition (CVPR 99), vol. 1, IEEE CS Press, 1999, pp. 625-632.
6. D.G. Aliaga and I. Carlbom, "Plenoptic Stitching: A Scalable Method for Reconstructing 3D Interactive Walkthroughs," Proc. Siggraph, ACM Press, 2001, pp. 443-450.
7. C.J. Taylor, "Videoplus: A Method for Capturing the Structure and Appearance of Immersive Environments," IEEE Trans. Visualization and Computer Graphics, vol. 8, no. 2, Apr.-June 2002, pp. 171-182.
1. P.E. Debevec and J. Malik, "Recovering High Dynamic Range Radiance Maps from Photographs," Proc. Siggraph, ACM Press, 1997, pp. 369-378.
2. G.W. Larson, H. Rushmeier, and C. Piatko, "A Visibility Matching Tone Reproduction Operator for High Dynamic Range Scenes," IEEE Trans. Visualization and Computer Graphics, vol. 3, no. 4, 1997, pp. 291-306.
3. E. Reinhard et al., "Photographic Tone Reproduction for Digital Images," ACM Trans. Graphics, vol. 21, no. 3, 2002, pp. 267-276.
4. S.B. Kang et al., "High Dynamic Range Video," ACM Trans. Graphics, vol. 22, no. 3, July 2003, pp. 319-325.

Matthew Uyttendaele, Antonio Criminisi, Sing Bing Kang, Simon Winder, Richard Szeliski, Richard Hartley, "Image-Based Interactive Exploration of Real-World Environments," IEEE Computer Graphics and Applications, vol. 24, no. 3, pp. 52-63, May-June 2004, doi:10.1109/MCG.2004.1297011
Usage of this product signifies your acceptance of the Terms of Use.