Many data mining applications can benefit from adapt- ing existing classifiers to new data with shifted distribu- tions. In this paper, we present Adaptive Support Vector Machine (Adapt-SVM) as an efficient model for adapting a SVM classifier trained from one dataset to a new dataset where only limited labeled examples are available. By in- troducing a new regularizer into SVM's objective function, Adapt-SVM aims to minimize both the classification error over the training examples, and the discrepancy between the adapted and original classifier. We also propose a selective sampling strategy based on the loss minimization principle to seed the most informative examples for classifier adap- tation. Experiments on an artificial classification task and on a benchmark video classification task shows that Adapt- SVM outperforms several baseline methods in terms of ac- curacy and/or efficiency.