Abstract: The classification algorithm that is based on a support vector machine (SVM) is now attracting more attention, due to its perfect theoretical properties and good empirical results. In this paper, we first analyze the properties of the support vector (SV) set thoroughly, then introduce a new learning method, which extends the SVM classification algorithm to the incremental learning area. The theoretical basis of this algorithm is the classification equivalence of the SV set and the training set. In this algorithm, knowledge is accumulated in the process of incremental learning. In addition, unimportant samples are discarded optimally by a least-recently used (LRU) scheme. Theoretical analyses and experimental results showed that this algorithm could not only speed up the training process, but it could also reduce the storage costs, while the classification precision is also guaranteed.