Based on human action recognition, this paper proposes a new description model to record human behavior. The paper makes use of action recognition result and regards time information accumulated in action recognition as features, records human actions and time spent in these actions, then identifies events through action combination and gives effective processing toward these identified events. In order to prove feasibility of human behavior description model, we take events produced when pedestrians pass through cross-road as example. Under cross-road context in the experiment, total 60 films are shot when five pedestrians are passing through cross-road, producing 191 events. 187 events are correctly detected in the experiment with correct rate of 98%.