Relevance feedback is a usually used technique to narrow the gap between high-level concepts and lowlevel visual features in the content-based image retrieval. In this paper, a novel long-term learning mechanism is proposed to grasp the retrieval intention as much as possible. With more retrieval sessions going on, an anamnesis semantic tree is constructed to record the semantic relationship between the query and the retrieved back images on the high level concepts. In the dynamic updating process of the anamnesis semantic tree, both the mean shift based query refining and clustering techniques are adopted. The final experimental results show that the proposed approach greatly improves the retrieval performance.