This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Ordering and Finding the Best of K>2 Supervised Learning Algorithms
March 2006 (vol. 28 no. 3)
pp. 392-402
ASCII Text
x 
 
Olcay Taner Yildiz, Ethem Alpaydin, "Ordering and Finding the Best of K>2 Supervised Learning Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 3, pp. 392-402, March, 2006.
 
 
BibTex
x
 
@article{ 10.1109/TPAMI.2006.61,
author = {Olcay Taner Yildiz and Ethem Alpaydin},
title = {Ordering and Finding the Best of K>2 Supervised Learning Algorithms},
journal ={IEEE Transactions on Pattern Analysis and Machine Intelligence},
volume = {28},
number = {3},
issn = {0162-8828},
year = {2006},
pages = {392-402},
doi = {http://doi.ieeecomputersociety.org/10.1109/TPAMI.2006.61},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Pattern Analysis and Machine Intelligence
TI - Ordering and Finding the Best of K>2 Supervised Learning Algorithms
IS - 3
SN - 0162-8828
SP392
EP402
EPD - 392-402
A1 - Olcay Taner Yildiz,
A1 - Ethem Alpaydin,
PY - 2006
KW - Index Terms- Machine learning
KW - classifier design and evaluation
KW - experimental design.
VL - 28
JA - IEEE Transactions on Pattern Analysis and Machine Intelligence
ER -
 
Given a data set and a number of supervised learning algorithms, we would like to find the algorithm with the smallest expected error. Existing pairwise tests allow a comparison of two algorithms only; range tests and ANOVA check whether multiple algorithms have the same expected error and cannot be used for finding the smallest. We propose a methodology, the MultiTest algorithm, whereby we order supervised learning algorithms taking into account 1) the result of pairwise statistical tests on expected error (what the data tells us), and 2) our prior preferences, e.g., due to complexity. We define the problem in graph-theoretic terms and propose an algorithm to find the "best” learning algorithm in terms of these two criteria, or in the more general case, order learning algorithms in terms of their "goodness.” Simulation results using five classification algorithms on 30 data sets indicate the utility of the method. Our proposed method can be generalized to regression and other loss functions by using a suitable pairwise test.

[1] T.G. Dietterich, “Approximate Statistical Tests for Comparing Supervised Classification Learning Classifiers,” Neural Computation, vol. 10, pp. 1895-1923, 1998.
[2] E. Alpaydin, Introduction to Machine Learning. The MIT Press, 2004.
[3] B. Efron, “Computers and the Theory of Statistics,” SIAM Rev., vol. 21, pp. 460-480, 1979.
[4] E. Alpaydin, “Combined $5\times2$ cv $f$ Test for Comparing Supervised Classification Learning Classifiers,” Neural Computation, vol. 11, pp. 1975-1982, 1999.
[5] S.M. Ross, Introduction to Probability and Statistics for Engineers and Scientists. New York: Wiley, 1987.
[6] R.G. Miller, Simultaneous Statistical Inference. New York: Springer Verlag, 1981.
[7] A. Dean and D. Voss, Design and Analysis of Experiments. New York: Springer Verlag, 1999.
[8] Y. Hochberg and A.C. Tamhane, Multiple Comparison Procedures. John Wiley and Sons, 1987.
[9] D.D. Jensen and P.R. Cohen, “Multiple Comparisons in Induction Algorithms,” Machine Learning, vol. 38, pp. 309-338, 2000.
[10] W.J. Conover, Practical Nonparametric Statistics. New York: John Wiley and Sons, 1999.
[11] B.S. Everitt, The Analysis of Contingency Tables. London: Chapman and Hall, 1977.
[12] S.W. Looney, “A Statistical Technique for Comparing the Accuracies of Several Classifiers,” Pattern Recognition Letters, vol. 8, pp. 5-9, 1988.
[13] R.R. Bouckaert, “Choosing between Two Learning Algorithms Based on Calibrated Tests,” Proc. 20th Int'l Conf. Machine Learning, 2003.
[14] F. Provost, T. Fawcett, and R. Kohavi, “The Case against Accuracy Estimation for Comparing Induction Algorithms,” Proc. 15th Int'l Conf. Machine Learning, 1998.
[15] F. Provost and T. Fawcett, “Robust Classification for Imprecise Environments,” Machine Learning, vol. 42, pp. 203-231, 2001.
[16] S.G. Alsing, K.W. Bauer, and J.O. Miller, “A Multinomial Selection Procedure for Evaluating Pattern Recognition Algorithms,” Pattern Recognition, vol. 35, pp. 2397-2412, 2002.
[17] P.D. Turney, “Types of Cost in Inductive Concept Learning,” Proc. Workshop Cost-Sensitive Learning, 17th Int'l Conf. Machine Learning, pp. 15-21, 2000.
[18] S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian J. Statistics, vol. 6, pp. 65-70, 1979.
[19] K.H. Rosen, Discrete Mathematics and Its Applications. New York: McGraw-Hill, 1995.
[20] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. New York: Springer Verlag, 2001.
[21] J.R. Quinlan, C4.5: Programs for Machine Learning. San Mateo, Calif.: Morgan Kaufmann, 1993.
[22] C.L. Blake and C.J. Merz, “UCI Repository of Machine Learning Databases,” http://www.ics.uci.edu/~mlearnMLRepository.html , 2000.
[23] O.T. Yildiz, “Tuning Model Complexity Using Cross-Validation for Supervised Learning,” PhD thesis, Dept. of Computer Eng., Boğaziçi Univ., 2005.

Index Terms:
Index Terms- Machine learning, classifier design and evaluation, experimental design.
Citation:
Olcay Taner Yildiz, Ethem Alpaydin, "Ordering and Finding the Best of K>2 Supervised Learning Algorithms," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 3, pp. 392-402, March 2006, doi:10.1109/TPAMI.2006.61
Usage of this product signifies your acceptance of the Terms of Use.