In this paper an effective method of using SVM classifier for multiple feature classification is proposed. Compared with traditional combination methods where all needed base classifiers should be trained before the decision combination, the proposed approach is to train individual classifiers and combine the decisions of these base classifiers at the same time. Thus the complexity of the training can be reduced because our proposed method involves solving only one optimization problem while several optimization problems should be solved for traditional methods. Furthermore, during the combination, our proposed approach takes into account both a base classifier's performance on the training data and its generalization ability while traditional combination approaches consider only a base classifier's performance on the training data. The experiments proved the efficiency of our proposed approach.