VR systems were traditionally used for tasks relying on high-quality graphics rendering where virtual environments were entirely made of user-defined objects. We are foreseeing that serious breakthroughs will emerge, where “augmented reality” environments will be created in a more dynamic fashion, using 2D and 3D data from computer vision sensors. Thus, virtual environments will not be made exclusively of user-defined objects, but also from real data. This data, after proper modeling to provide some behavior and abstraction levels, will then be used to feed high-performance imaging systems.This paper deals with 1) local surface modeling of the 3D data visible from each 2D viewpoint using a modified Marching Cubes algorithm and 2) region matching of the local models (from neighboring views in the sequence) using color region clustering information from the 2D snap-shots. Local models and the region matching information are used to build complete global models from a sequence of images.