Feature selection is an important consideration in several applications where one needs to choose a smaller subset of features from a complete set of raw measurements such that the improved subset generates as good or better classification performance compared to original data. In this paper, we describe a novel feature selection approach that is based on the estimation of classification complexity though data partitioning. This approach allows us to select the N best features from a given set in order of their ability to separate data from different classes. In this paper, we perform our experiments on the ORLface database that consists of 400 images. The results show that the proposed approach outperforms the probability distance approach and is a viable method for implementing more advanced search methods of feature selection.