Motion is an important cue for video understanding and widely used in many semantic video analysis. In this paper, we present a new motion representation scheme in which motions in a video is represented by the responses of frames to a set of motion filters. Each of these filters is designed to be most responsive to a type of dominant motion. Then we employ Hidden Markov Models (HMMs) to characterize the motion patterns based on these features and thus classify basketball video into 16 events. The evaluation by human satisfaction rate to classification result is 75%, demonstrating effectiveness of the proposed approach to recognizing semantic events in video.