In this paper, a new ontology-based composite concept detection method is proposed, which adopts Bayesian network to construct the ontology and uses the inference rules to perform the composite concept detection providing the concrete concepts in a phrase of video. Furthermore, the probability instead of binary value is gained through the inference pattern of Bayesian network, which can obtain more precise results. The main contribution of this paper is that semantic concept ontology is constructed using Bayesian network and the constructed ontology represents the hierarchical relationship between the concepts which can be used for inference. The method narrows the influence of "Semantic Gap" in some extent and achieves good performance in composite concept detection.