SVM based image retrieval suffers from the scarcity of labelled samples. In this paper, this problem is solved by incorporating prior knowledge into SVM. Firstly, some prior knowledge of image retrieval is discussed and constructed. After that, the knowledge is incorporated into SVM optimization as a constraint, and a new knowledge-based target function is formulated. Based on this, a framework of image retrieval with knowledge based SVM is proposed. Experimental results demonstrate that the proposed method can effectively improve the learning and retrieval performance of SVM, especially when the number of labelled samples is small.