Feature selection has become the cornerstone of many classification problems. It has been applied in many domains such as web mining, text categorization, gene expression microarray analysis, image analysis, and combinatorial chemistry. One type of well-studied feature selection methodology is filtering, which is typically divided into ranking and subset evaluation. This work provides an empirical study regarding one type of feature ranking for which very limited research exists, namely wrapper-based feature ranking. Nine performance metrics are evaluated, and while these metrics are commonly used in data mining to evaluate classifier performance, they are rarely used as feature ranking techniques. Moreover, five different learners, 5-Nearest Neighbors (5NN), Logistic Regression (LR), Multi Layer Perceptron (MLP), Na¨ıve Bayes (NB), and Support Vector Machines (SVM) in conjunction with two different methodologies, 3-Fold Cross-Validation (CV) and 3- Fold Cross-validation Risk Impact (CVR) are used in this study to evaluate feature relevancy and to determine ranking similarities among the different ranking techniques.