This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
The Random Subspace Method for Constructing Decision Forests
August 1998 (vol. 20 no. 8)
pp. 832-844

Abstract—Much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes. The dilemma between overfitting and achieving maximum accuracy is seldom resolved. A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity. The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces. The subspace method is compared to single-tree classifiers and other forest construction methods by experiments on publicly available datasets, where the method's superiority is demonstrated. We also discuss independence between trees in a forest and relate that to the combined classification accuracy.

[1] Y. Amit, D. Geman, and K. Wilder, “Joint Induction of Shape Features and Tree Classifiers,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 19, no. 11, pp. 1300-1305, Nov. 1997.
[2] R. Berlind, "An Alternative Method of Stochastic Discrimination With Applications to Pattern Recognition," doctoral dissertation, Dept. of Mathematics, State Univ. of New York at Buffalo, 1994.
[3] L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone, Classification and Regression Trees.Belmont, Calif.: Wadsworth, 1984.
[4] L. Breiman, "Bagging Predictors," Machine Learning, vol. 24, pp. 123-140, 1996.
[5] G.R. Dattatreya and L.N. Kanal, "Decision Trees in Pattern Recognition," L.N. Kanal and A. Rosenfeld, eds., Progress in Pattern Recognition 2.Amsterdam: Elsevier, 1985.
[6] Y. Freund and R.E. Schapire, "Experiments With a New Boosting Algorithm," Proc. 13th Int'l Conf. Machine Learning, pp. 148-156,Bari, Italy, July3-6 1996.
[7] D. Heath, S. Kasif, and S. Salzberg, "Induction of Oblique Decision Trees," Proc. 13th Int'l Joint Conf. Artificial Intelligence, vol. 2, pp. 1,002-1,007,Chambery, France, Aug. 28- Sept.3 1993.
[8] S. Murthy, S. Kasif, and S. Salzberg, "A System for Induction of Oblique Decision Trees," J. Artificial Intelligence Res., vol. 2, no. 1, pp. 1-32, 1994.
[9] D. Heath, S. Kasif, and S. Salzberg, "Committees of Decision Trees," B. Gorayska and J.L. Mey, eds., Cognitive Technology: In Search of a Humane Interface, pp. 305-317.New York: Elsevier Science, 1996.
[10] T.K. Ho, "Recognition of Handwritten Digits by Combining Independent Learning Vector Quantizations," Proc. Second Int'l Conf. Document Analysis and Recognition, pp. 818-821,Tsukuba Science City, Japan,20-22 Oct. 1993.
[11] T.K. Ho, “Random Decision Forests,” Proc. Third Int'l Conf. Document Analysis and Recognition, pp. 278-282, 1995.
[12] T.K. Ho, "C4.5 Decision Forests," Proc. 14th Int'l Conf. Pattern Recognition,Brisbane, Australia,17-20 Aug. 1998.
[13] T.K. Ho, H.S. Baird, "Perfect Metrics," Proc. Second Int'l Conf. Document Analysis and Recognition, pp. 593-597,Tsukuba Science City, Japan, Oct.20-22, 1993.
[14] T.K. Ho and H.S. Baird, "Pattern Classification With Compact Distribution Maps," Computer Vision and Image Understanding, vol. 70, no. 1, pp. 101-110, Apr. 1998.
[15] T.K. Ho, J.J. Hull, and S.N. Srihari, “Decision Combination in Multiple Classifiers Systems,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 16, no. 1, pp. 66-75, Jan. 1994.
[16] T.K. Ho and E.M. Kleinberg, "Building Projectable Classifiers of Arbitrary Complexity," Proc. 13th Int'l Conf. Pattern Recognition, pp. 880-885,Vienna, Aug.25-30 1996.
[17] E.M. Kleinberg, "Stochastic Discrimination," Annals of Mathematics and Artificial Intelligence, vol. 1, pp. 207-239, 1990.
[18] E.M. Kleinberg, "An Overtraining-Resistant Stochastic Modeling Method for Pattern Recognition," Annals of Statistics, vol. 4, no. 6, pp. 2,319-2,349, Dec. 1996.
[19] S.W. Kwok and C. Carter, "Multiple Decision Trees," R.D. Shachter, T.S. Levitt, L.N. Kanal, and J.F. Lemmer, eds., Uncertainty in Artificial Intelligence, vol. 4, pp. 327-335.New York: Elsevier Science Publishers, 1990.
[20] J. Mingers, "Expert Systems—Rule Induction With Statistical Data," J. Operational Res. Soc., vol. 38, pp. 39-47, 1987.
[21] J. Mingers, "An Empirical Comparison of Selection Measures for Decision-Tree Induction," Machine Learning, vol. 3, pp. 319-342, 1989.
[22] N.J. Nilsson, Learning Machines: Foundations of Trainable Pattern-Classifying Systems.New York: McGraw-Hill, 1965.
[23] Y. Park and J. Sklansky, "Automated Design of Multiple-Class Piecewise Linear Classifiers," J. Classification, vol. 6, pp. 195-222, 1989.
[24] Project StatLog, LIACC, Univ. of Porto, internet address:ftp.ncc.up.pt(directory: pub/statlog/datasets).
[25] J.R. Quinlan, "Induction of Decision Trees," Machine Learning, vol. 1, pp. 81-106, 1986.
[26] J.R. Quinlan, C4.5: Programs for Machine Learning.San Mateo, Calif.: Morgan Kaufmann, 1993.
[27] J.R. Quinlan, "Bagging, Boosting, and C4.5," Proc. 13th Nat'l Conf. Artificial Intelligence, pp. 725-730,Portland, Ore.,4-8 Aug. 1996.
[28] I.K. Sethi and G.P.R. Sarvarayudu, "Hierarchical Classifier Design Using Mutual Information," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 4, no. 4, pp. 441-445, July 1982.
[29] I.K. Sethi and J.H. Yoo, "Structure-Driven Induction of Decision Tree Classifiers Through Neural Learning," Pattern Recognition, vol. 30, no. 11, pp. 1,893-1,904, 1997.
[30] S. Shlien, "Multiple Binary Decision Tree Classifiers," Pattern Recognition, vol. 23, no. 7, pp. 757-763, 1990.
[31] S. Shlien, "Nonparametric Classification Using Matched Binary Decision Trees," Pattern Recognition Letters, vol. 13, pp. 83-87, Feb. 1992.
[32] P. Turney, "Technical Note: Bias and the Quantification of Stability," Machine Learning, vol. 20, pp. 23-33, 1995.
[33] V. Vapnik, The Nature of Statistical Learning Theory.New York: Springer-Verlag, 1995.

Index Terms:
Pattern recognition, decision tree, decision forest, stochastic discrimination, decision combination, classifier combination, multiple-classifier system, bootstrapping.
Citation:
Tin Kam Ho, "The Random Subspace Method for Constructing Decision Forests," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 8, pp. 832-844, Aug. 1998, doi:10.1109/34.709601
Usage of this product signifies your acceptance of the Terms of Use.