This paper discusses a model-based approach to the design of complex interaction environments like virtual worlds, mixed and augmented reality. The environment a user interacts with is seen as a virtual environment populated by virtual entities, created and maintained active by a program interpreted by the computer, which can be described by specifying the behavior of the population. The specification of the behavior occurs along three dimensions: 1) programming languages to specify system computations; 2) user activity languages to specify user activities; 3) perceptual languages to deal with the physical characteristics of the messages from the machine to the user. These dimensions define an interaction modeling space which constitutes the frame in which the virtual environment is specified.