To effectively deal with the vast amount of videos, we need to construct a content-based representation for each video. As a step towards this goal, this paper proposes a method to automatically generate the semantical annotations for a sports video by integrating the text (closed-caption) and image stream. We first segment the text data and extract segments, which are meaningful to grasp the story of the video, and then extract the actors, the actions and the events of each scene, which are useful for information retrieval by using the linguistic cues and the domain knowledge. We also segment the image stream so that each segment can associate with each text segment extracted above by using the image cues. Finally, we can annotate the video by associating the text segments with the image segments. Some experimental results are presented and discussed in this paper.