In this paper we present a vision system that gives a natural language interpretation of activities where a person handles objects. The system integrates low-level image components such as hand and object tracking, detection and recognition, posture and gesture recognition, with highlevel processes such as spatio-temporal object relationship generation and activity reasoning. To achieve near realtime operation, a task-oriented approach focuses processing depending on the situation context. Vision components are dynamically coordinated and their interrelation and region of interest continuously adapted to the interpretation task. The task of putting a CD in a CD-player with one or two hands demonstrates the operation of the system.