In many application areas, as in medicine, knowledge is discovered from large databases using data mining methods. It is evident that quite often even with static data mining methods the accuracy of the results of data mining is better if fewer features are selected or if the classification task is divided into sub tasks guided by the heterogeneity of the feature space. On the other hand, one of the most important directions in the improvement of data mining and knowledge discovery is to integrate multiple classification techniques included in an ensemble of classifiers. We present two variations of an advanced dynamic integration technique, which first build an ensemble of classifiers containing base classifiers based on the subsets of the original feature set, then evaluate the competence areas of the base classifiers inside the application domain, and finally select a classifier to produce the final classification result dynamically. The technique is evaluated on two data sets taken from the UCI machine learning repository and a comparison of the results show that our dynamic integration technique with classifiers based on reduced feature sets is able to produce more accurate results than C4.5 with the whole set of features on those data sets and that our method outperforms the cross-validation majority technique and is comparable with weighted voting on some data sets.