This Article 
 Bibliographic References 
 Add to: 
Orthogonal Decision Trees
August 2006 (vol. 18 no. 8)
pp. 1028-1042
This paper introduces orthogonal decision trees that offer an effective way to construct a redundancy-free, accurate, and meaningful representation of large decision-tree-ensembles often created by popular techniques such as Bagging, Boosting, Random Forests, and many distributed and data stream mining algorithms. Orthogonal decision trees are functionally orthogonal to each other and they correspond to the principal components of the underlying function space. This paper offers a technique to construct such trees based on the Fourier transformation of decision trees and eigen-analysis of the ensemble in the Fourier representation. It offers experimental results to document the performance of orthogonal trees on the grounds of accuracy and model complexity.

[1] J.R. Quinlan, “Induction of Decision Trees,” Machine Learning, vol. 1, no. 1, pp. 81-106, 1986.
[2] Y. Freund, “Boosting a Weak Learning Algorithm by Majority,” Information and Computation, vol. 121, no. 2, pp. 256-285, 1995.
[3] H. Drucker and C. Cortes, “Boosting Decision Trees,” Advances in Neural Information Processing Systems, vol. 8, pp. 479-485, 1996.
[4] L. Breiman, “Bagging Predictors,” Machine Learning, vol. 24, no. 2, pp. 123-140, 1996.
[5] D. Wolpert, “Stacked Generalization,” Neural Networks, vol. 5, pp. 241-259, 1992.
[6] L. Breiman, “Random Forests,” Machine Learning, vol. 45, no. 1, pp. 5-32, 2001.
[7] W. Fan, S. Stolfo, and J. Zhang, “The Application of Adaboost for Distributed, Scalable, and On-Line Learning,” Proc. Fifth ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 1999.
[8] W.N. Street and Y. Kim, “A Streaming Ensemble Algorithm (Sea) for Large-Scale Classificaiton,” Proc. Seventh ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, 2001.
[9] H. Kargupta and B. Park, “A Fourier Spectrum-Based Approach to Represent Decision Trees for Mining Data Streams in Mobile Environments,” IEEE Trans. Knowledge and Data Eng., vol. 16, no. 2, pp. 216-229, 2002.
[10] B. Park, A. R, and H. Kargupta, “A Fourier Analysis-Based Approach to Learn Classifier from Distributed Heterogeneous Data,” Proc. First SIAM Int'l Conf. Data Mining, 2001.
[11] B.H. Park and H. Kargupta, “Constructing Simpler Decision Trees from Ensemble Models Using Fourier Analysis,” Proc. Seventh Workshop Research Issues in Data Mining and Knowledge Discovery, pp. 18-23, 2002.
[12] F. Chung, Spectral Graph Theory. Providence, R.I.: Am. Math. Soc., 1994.
[13] H. Kargupta and B. Park, “Mining Time-Critical Data Stream Using the Fourier Spectrum of Decision Trees,” Proc. IEEE Int'l Conf. Data Mining, pp. 281-288, 2001.
[14] H. Kargupta, B. Park, S. Pittie, L. Liu, D. Kushraj, and K. Sarkar, “Mobimine: Monitoring the Stock Market from a PDA,” ACM SIGKDD Explorations, vol. 3, no. 2, pp. 37-46, Jan. 2002.
[15] N. Linial, Y. Mansour, and N. Nisan, “Constant Depth Circuits, Fourier Transform, and Learnability,” J. ACM, vol. 40, pp. 607-620, 1993.
[16] E. Kushilevitz and Y. Mansour, “Learning Decision Trees Using the Fourier Spectrum,” SIAM J. Computing, vol. 22, no. 6, pp. 1331-1348, 1993.
[17] D. Goldberg, “Genetic Algorithms and Walsh Functions: Part I, a Gentle Introduction,” Complex Systems, vol. 3, no. 2, pp. 129-152, 1989.
[18] C.J. Merz and M.J. Pazzani, “A Principal Components Approach to Combining Regression Estimates,” Machine Learning, vol. 36, nos. 1-2, pp. 9-32, 1999.
[19] C. Merz and M. Pazzani, “A Principal Components Approach to Combining Regression Estimates,” Machine Learning, vol. 36, pp. 9-32, 1999.
[20] H. Kargupta, B. Park, D. Hershberger, and E. Johnson, “Collective Data Mining: A New Perspective towards Distributed Data Mining,” Advances in Distributed and Parallel Knowledge Discovery, H. Kargupta and P. Chan, eds., AAAI/MIT Press, 2000.

Index Terms:
Orthogonal decision trees, redundancy free trees, principle component analysis, Fourier transform.
Hillol Kargupta, Byung-Hoon Park, Haimonti Dutta, "Orthogonal Decision Trees," IEEE Transactions on Knowledge and Data Engineering, vol. 18, no. 8, pp. 1028-1042, Aug. 2006, doi:10.1109/TKDE.2006.127
Usage of this product signifies your acceptance of the Terms of Use.