This paper proposes an automatic annotation method of sports video for content-based retrieval. Conventional methods using position information of objects such as locus, relative positions, their transitions, etc. as indices, have drawbacks that tracking errors of a certain object due to occlusions cause recognition failures, and that representation by position information essentially has limited number of recognizable events in the retrieval. Our approach incorporates human behavior analysis and specific domain knowledge with conventional methods, to develop integrated reasoning module for richer expressiveness of events and robust recognition.Based on the proposed method, we implemented content-based retrieval system which can identify several actions on real tennis video. We select court and net lines, players' positions, ball positions, and players' actions, as indices. Court and net lines are extracted using court model and hough transforms. Players and ball positions are tracked by adaptive template matching and particular predictions against sudden changes of motion direction. Players' actions are analyzed by 2-d appearance based matching using the transition of players' silhouettes and hidden markov model. The results using two sets of tennis video is presented demonstrating the performance and the validity of our approach.