In previous work, we developed a novel Relevance Feedback (RF) framework that learns One-class Support Vector Machines (1SVM) from retrieval experience to represent the set memberships of users' high level semantics. By doing a fuzzy classification of a query into the regions of support represented by the 1SVMs, past experience is merged with short-term (i.e., intra-query) learning. However, this led to the representation of long-term (i.e., inter-query) learning with a constantly growing number of 1SVMs in the feature space. We present an improved version of our earlier work that uses an incremental k-means algorithm to cluster 1SVMs. The main advantage of the improved approach is that it is scalable and can accelerate query processing by considering only a small number of cluster representatives, rather than the entire set of accumulated 1SVMs. Experimental results against real data sets demonstrate the effectiveness of the proposed method.