This paper presents a novel algorithm named Diverse AdaBoostSVM Tracking(DABSVT) for target tracking in infrared imagery. The tracker trains a Support Vector Machine(SVM) classifier per frame. All of the classifiers are combined into an ensemble classifier using AdaBoost. By proper parameter adjusting strategies, a set of effective SVM classifiers with moderate accuracy are obtained, and the dilemma problem between accuracy and diversity of AdaBoost is dealt with too. To cope with the changes in features of both foreground and background, the component classifier can be discarded or added at any time. The experiments performed on several sequences show the robustness of the proposed method.