Image retrieval has been widely used in many fields of science and engineering. The semantic concept of user interest is obtained by a learning process. Traditional techniques often assume the images are from certain distribution and all images from the same class are visually similar. Our study shows that those assumptions are inappropriate in many cases. To solve this problem we model the images as lying on non-linear subspaces embedded in the high dimensional space. We also find that a set of low-level feature subspaces may correspond to one high-level semantic concept. Unlike most unsupervised subspace learning techniques, we propose to intelligently use the semantic similarity and dissimilarity information provided by user in discovering the discriminant structure of image subspaces in respect to classification. Theoretical study shows that our methods converge to Linear Discriminant Analysis if certain criteria are met. Extensive experiments are designed to evaluate the performance of our method and compare it to other state-of-the-art techniques. The results show the superior performance of our proposed method.