Current electronic data repositories contain enormous amount of data, especially in medical domains, where data is often feature-space heterogeneous so that different features have different importance in different sub-areas of the whole space. In this paper, we suggest a technique that searches for a strategic splitting of the feature space identifying the best subsets of features for each instance. Our technique is based on the wrapper approach where a classification algorithm is used as the evaluation function to differentiate between several feature subsets. We apply the recently developed technique for dynamic integration of classifiers and use decision trees. For each test instance, we consider only those feature combinations that include features present in the path taken by the test instance in the decision tree. We evaluate our technique on medical datasets from the UCI machine-learning repository. The experiments show that the local feature selection is often advantageous in comparison with feature selection on the whole space.