We present an application for design and simulation of a building that fits into its surroundings in a heterogeneous collaborative virtual environment. Constructed heterogeneous collaborative VE consists of different VR display systems such as a virtual table and a projection wall. Since the virtual table provides a table-type workspace on a horizontal screen, it is efficient for interactive design of a building with a bird's-eye-view. However, it is not so efficient to view a designed building in the same proportions as that for a real building. On the other hand, as the projection wall is a vertical type VR environment having a large display, it enables simulation of the whole design process on a virtual table in frontal view. In addition, it also enables navigation around the building and in its surroundings. So we propose to use a projection wall as well as virtual table for constructing virtual environment so that design and simulation could be performed at the same time in each environment. Since the two VR systems are located close to one another, the user in the projection wall can participate in the design process that is performed in the virtual table indirectly through verbal communication. Guided 3D interaction techniques are also described for designing a building with ease in the 3D VR environment.