Quite recently, a few hundreds of semantic concepts are detected automatically with varied performance and subsequently, a new video retrieval paradigm of query-byconcept emerges. In this paper, we consider the problem of exploiting the potential of learned semantics concepts, together with the combination of traditional methods, for automatic and interactive retrieval. We argue that it is important, in both automatic and interactive retrieval scenarios, to find a few relevant concepts to search with, given a multimedia query. For automatic retrieval, we show that both text and image inputs are useful for solving this query-concept- mapping (QUCOM) problem. For interactive retrieval, searching with relevant concepts plus conventional feedback methods is quite effective and is robust to initial search results. Experimental evidence on the search task of TRECVID 2006 shows that by solving QUCOM with a large lexicon of 311 semantic concept detectors, the automatic retrieval performance increases 20% and the interactive retrieval performance has the potential to outperform the state-of-the-art systems.