In this paper, we propose a novel method for recognizing environment based on relationship between human actions and objects for a mobile robot. Most of previous works on environment recognition for robots focused on generating obstacle maps for path-planning. In addition, model-based object recognition techniques are also used for searching particular objects. It is, however, difficult in reality to prepare a lot of models in advance for recognizing various objects in unknown environments. On the other hand, human can often recognize objects not from their appearances but by watching other person taking actions on them. This is because the function and/or the usage of the objects are closely related with human actions. We have introduced conceptual models of human actions and objects for classifying objects by observing human activities in our previous work. In this paper, we apply this key idea to a mobile robot. We also demonstrate that the arrangement of objects can be recognized by analyzing human actions.