In the field of image understanding, it is becoming popular to recognize human actions and objects in a scene to improve human-computer interaction. Recently, various methods for scene recognition have been proposed, and most of them focus on recognizing human actions and objects separately. We consider, however, it is important to recognize them complementarily, because human actions are closely related with objects. The relationship between actions and objects are represented by hierarchical models. First, movement of human head and hands are tracked by stereo vision. Features of actions such as position and direction of the head and hands are extracted and input to Dynamic Bayesian Networks to classify actions roughly. Then the actions and related objects are refined by using conceptual models of human actions and objects. Finally, detailed actions and objects are recognized by referring each models cooperatively.