| | This Article | |
| |
| |
| | Share | |
| |
| |
| | Bibliographic References | |
| |
| |
| | Add to: | |
| |
 Digg
 Furl
 Spurl
 Blink
 Simpy
 Google
 Del.icio.us
 Y!MyWeb
| |
| | Search | |
| |
| |
| | |
A Comparison of Decision Tree Ensemble Creation Techniques
January 2007 (vol. 29 no. 1)
pp. 173-180
We experimentally evaluate bagging and seven other randomization-based approaches to creating an ensemble of decision tree classifiers. Statistical tests were performed on experimental results from 57 publicly available data sets. When cross-validation comparisons were tested for statistical significance, the best method was statistically more accurate than bagging on only eight of the 57 data sets. Alternatively, examining the average ranks of the algorithms across the group of data sets, we find that boosting, random forests, and randomized trees are statistically significantly better than bagging. Because our results suggest that using an appropriate ensemble size is important, we introduce an algorithm that decides when a sufficient number of classifiers has been created for an ensemble. Our algorithm uses the out-of-bag error estimate, and is shown to result in an accurate ensemble for those methods that incorporate bagging into the construction of the ensemble.
[1] L. Breiman, “Bagging Predictors,” Machine Learning, vol. 24, pp. 123-140, 1996.
[2] G. Eibl and K. Pfeiffer, “How to Make AdaBoost.M1 Work for Weak Base Classifiers by Changing Only One Line of the Code,” Proc. 13th European Conf. Machine Learning, pp. 72-83, 2002.
[3] Y. Freund and R. Schapire, “Experiments with a New Boosting Algorithm,” Proc. 13th Nat'l Conf. Machine Learning, pp. 148-156, 1996.
[4] R. Schapire, “The Strength of Weak Learnability,” Machine Learning, vol. 5, no. 2, pp. 197-227, 1990.
[5] T. Ho, “The Random Subspace Method for Constructing Decision Forests,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832-844, Aug. 1998.
[6] L. Breiman, “Random Forests,” Machine Learning, vol. 45, no. 1, pp. 5-32, 2001.
[7] T. Dietterich, “An Experimental Comparison of Three Methods for Constructing Ensembles of Decision Trees: Bagging, Boosting, and Randomization,” Machine Learning, vol. 40, no. 2, pp. 139-157, 2000.
[8] T.G. Dietterich, “Approximate Statistical Test for Comparing Supervised Classification Learning Algorithms,” Neural Computation, vol. 10, no. 7, pp.1895-1923, 1998.
[9] E. Alpaydin, “Combined $5 \times 2$ cv F Test for Comparing Supervised Classification Learning Algorithms,” Neural Computation, vol. 11, no. 8, pp.1885-1892, 1999.
[10] J. Demsar, “Statistical Comparisons of Classifiers over Multiple Data Sets,” J. Machine Learning Research, vol. 7, pp. 1-30, 2006.
[11] L. Hall, K. Bowyer, R. Banfield, D. Bhadoria, W. Kegelmeyer, and S. Eschrich, “Comparing Pure Parallel Ensemble Creation Techniques against Bagging,” Proc. Third IEEE Int'l Conf. Data Mining, pp. 533-536, 2003.
[12] R.E. Banfield, L.O. Hall, K.W. Bowyer, and W.P. Kegelmeyer, “A Statistical Comparison of Decision Tree Ensemble Creation Techniques,” Proc. 2006 Int'l Conf. Systems, Man, and Cybernetics, 2006, to appear.
[13] R. Banfield, OpenDT, http:/opendt.sourceforge.net/, 2005.
[14] J. Quinlan, C4.5: Programs for Machine Learning. Morgan Kaufmann, 1992.
[15] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools with Java Implementations. Morgan Kaufmann, 1999.
[16] E. Bauer and R. Kohavi, “An Empirical Comparison of Voting Classification Algorithms: Bagging, Boosting, and Variants,” Machine Learning, vol. 36, nos.1-2, pp. 105-139, 1999.
[17] L. Breiman, “Arcing Classifiers,” Annals of Statistics, vol. 26, no. 2, pp. 801-824, 1998.
[18] Y. Freund and R. Schapire, “Discussion of the Paper ‘Arcing Classifiers’ by Leo Breiman,” Annals of Statistics, vol. 26, no. 2, pp. 824-832, 1998.
[19] L. Breiman, “Rejoinder to the Paper ‘Arcing Classifiers’ by Leo Breiman,” Annals of Statistics, vol. 26, no. 2, pp. 841-849, 1998.
[20] R.E. Banfield, L.O. Hall, K.W. Bowyer, and W.P. Kegelmeyer, “A New Ensemble Diversity Measure Applied to Thinning Ensembles,” Proc. Fifth Int'l Workshop Multiple Classifier Systems, pp. 306-316, 2003.
[21] C. Merz and P. Murphy, UCI Repository of Machine Learning Databases, Dept. of CIS, Univ. of California, Irvine, http://www.ics.uci.edu/~mlearnMLRepository.html , 2006.
[22] R.E. Schapire, Y. Freund, P. Bartlett, and W.S. Lee, “Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods,” Proc. 14th Int'l Conf. Machine Learning, pp. 322-330, 1997.
[23] R. Johnson and D. Wichern, Applied Multivariate Statistical Analysis, third ed. Prentice-Hall, 1992.
Index Terms:
Classifier ensembles, bagging, boosting, random forests, random subspaces, performance evaluation.
Citation:
Robert E. Banfield, Lawrence O. Hall, Kevin W. Bowyer, W.P. Kegelmeyer, "A Comparison of Decision Tree Ensemble Creation Techniques," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 1, pp. 173-180, Jan. 2007, doi:10.1109/TPAMI.2007.2
