Digital video databases have become more pervasive and finding video clips in video databases quickly becomes a major bottleneck. Indexing and annotation can help video system retrieve appropriate video clips fast. Currently, these techniques are based on computation on image low-level features and manual annotation, whose low success ratio and retrieval speed limit applications of video databases. In this paper we consider the semantics interpretation of the contents as the annotation for video clips. We give a self-adaptive high-level description model for application-oriented semantics extraction, which annotates and retrieve video flexibly and automatically.