Issue No.01 - Jan. (2013 vol.25)
pp: 206-219
Shuo Wang , The University of Birmingham, Birmingham
Xin Yao , The University of Birmingham, Birmingham
In class imbalance learning problems, how to better recognize examples from the minority class is the key focus, since it is usually more important and expensive than the majority class. Quite a few ensemble solutions have been proposed in the literature with varying degrees of success. It is generally believed that diversity in an ensemble could help to improve the performance of class imbalance learning. However, no study has actually investigated diversity in depth in terms of its definitions and effects in the context of class imbalance learning. It is unclear whether diversity will have a similar or different impact on the performance of minority and majority classes. In this paper, we aim to gain a deeper understanding of if and when ensemble diversity has a positive impact on the classification of imbalanced data sets. First, we explain when and why diversity measured by Q-statistic can bring improved overall accuracy based on two classification patterns proposed by Kuncheva et al. We define and give insights into good and bad patterns in imbalanced scenarios. Then, the pattern analysis is extended to single-class performance measures, including recall, precision, and F-measure, which are widely used in class imbalance learning. Six different situations of diversity's impact on these measures are obtained through theoretical analysis. Finally, to further understand how diversity affects the single class performance and overall performance in class imbalance problems, we carry out extensive experimental studies on both artificial data sets and real-world benchmarks with highly skewed class distributions. We find strong correlations between diversity and discussed performance measures. Diversity shows a positive impact on the minority class in general. It is also beneficial to the overall performance in terms of AUC and G-mean.
Accuracy, Diversity reception, Correlation, Context, Pattern analysis, Training, Boosting, data mining, Class imbalance learning, ensemble learning, diversity, single-class performance measures
Shuo Wang, Xin Yao, "Relationships between Diversity of Classification Ensembles and Single-Class Performance Measures", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 1, pp. 206-219, Jan. 2013, doi:10.1109/TKDE.2011.207
[1] R.M. Valdovinos and J.S. Sanchez, "Class-Dependant Resampling for Medical Applications," Proc. Fourth Int'l Conf. Machine Learning and Applications (ICMLA '05), pp. 351-356, 2005.
[2] T. Fawcett and F. Provost, "Adaptive Fraud Detection," Data Mining and Knowledge Discovery, vol. 1, no. 3, pp. 291-316, 1997.
[3] K.J. Ezawa, M. Singh, and S.W. Norton, "Learning Goal Oriented Bayesian Networks for Telecommunications Risk Management," Proc. 13th Int'l Conf. Machine Learning, pp. 139-147, 1996.
[4] C. Cardie and N. Howe, "Improving Minority Class Prediction Using Casespecific Feature Weights," Proc. 14th Int'l Conf. Machine Learning, pp. 57-65, 1997.
[5] G.M. Weiss, "Mining with Rarity: A Unifying Framework," ACM SIGKDD Explorations Newsletters, vol. 6, no. 1, pp. 7-19, 2004.
[6] S. Visa and A. Ralescu, "Issues in Mining Imbalanced Data Sets - A Review Paper," Proc. 16th Midwest Artificial Intelligence and Cognitive Science Conf., pp. 67-73, 2005.
[7] N. Japkowicz and S. Stephen, "The Class Imbalance Problem: A Systematic Study," Intelligent Data Analysis, vol. 6, no. 5, pp. 429-449, 2002.
[8] C. Li, "Classifying Imbalanced Data Using a Bagging Ensemble Variation," Proc. 45th Ann. Southeast Regional Conf. (AVM-SE 45), pp. 203-208, 2007.
[9] X.-Y. Liu, J. Wu, and Z.-H. Zhou, "Exploratory Undersampling for Class Imbalance Learning," IEEE Trans. Systems, Man, and Cybernetics, vol. 39, no. 2, pp. 539-550, Apr. 2009.
[10] N.V. Chawla, A. Lazarevic, L.O. Hall, and K.W. Bowyer, "SMOTEBoost: Improving Prediction of the Minority Class in Boosting," Knowledge Discovery in Databases: PKDD 2003, vol. 2838, pp. 107-119, 2003.
[11] M.V. Joshi, V. Kumar, and R.C. Agarwal, "Evaluating Boosting Algorithms to Classify Rare Classes: Comparison and Improvements," Proc. IEEE Int'l Conf. Data Mining, pp. 257-264, 2001.
[12] N.V. Chawla and J. Sylvester, "Exploiting Diversity in Ensembles: Improving the Performance on Unbalanced Datasets," Multiple Classifier Systems, vol. 4472, pp. 397-406, 2007.
[13] H. Guo and H.L. Viktor, "Learning from Imbalanced Data Sets with Boosting and Data Generation: The Databoost-im Approach," SIGKDD Explorations Newsletter, vol. 6, no. 1, pp. 30-39, 2004.
[14] W. Fan, S.J. Stolfo, J. Zhang, and P.K. Chan, "Adacost: Misclassification Cost-Sensitive Boosting," Proc. 16th Int'l Conf. Machine Learning, pp. 97-105, 1999.
[15] M.P. Perrone and L.N. Cooper, "When Networks Disagree: Ensemble Methods for Hybrid Neural Networks," Neural Networks for Speech and Image Processing, Chapman-Hill, 1993.
[16] A. Estabrooks, T. Jo, and N. Japkowicz, "A Multiple Resampling Method for Learning from Imbalanced Data Sets," Computational Intelligence, vol. 20, no. 1, pp. 18-36, 2004.
[17] S.B. Kotsiantis and P.E. Pintelas, "Mixture of Expert Agents for Handling Imbalanced Data Sets," Ann. Math., Computing and Teleinformatics, vol. 1, no. 1, pp. 46-55, 2003.
[18] L.I. Kuncheva, M. Skurichina, and R.P. Duin, "An Experimental Study on Diversity for Bagging and Boosting with Linear Classifiers," Information Fusion, vol. 3, no. 4, pp. 245-258, 2002.
[19] R. Polikar, "Ensemble Based Systems in Decision Making," IEEE Circuits and Systems Magazine, vol. 6, no. 3, pp. 21-45, July-Sept. 2006.
[20] G. Brown, J.L. Wyatt, and P. Tino, "Managing Diversity in Regression Ensembles," J. Machine Learning Research, vol. 6, pp. 1621-1650, 2005.
[21] S.B. Kotsiantis, D. Kanellopoulos, and P. Pintelas, "Handling Imbalanced Datasets: A Review," GESTS Int'l Trans. Computer Science and Eng., vol. 30, no. 1, pp. 25-36, 2006.
[22] C.V. Rijsbergen, Information Retrieval. Butterworths, 1979.
[23] G.U. Yule, "On the Association of Attributes in Statistics," Philosophical Trans. Royal Soc. of London, vol. A194, pp. 257-319, 1900.
[24] L.I. Kuncheva, C.J. Whitaker, C.A. Shipp, and R.P. Duin, "Limits on the Majority Vote Accuracy in Classifier Fusion," Pattern Analysis and Applications, vol. 6, no. 1, pp. 22-31, 2003.
[25] G. Brown, J. Wyatt, R. Harris, and X. Yao, "Diversity Creation Methods: A Survey and Categorisation," Information Fusion, vol. 6, no. 1, pp. 5-20, 2004.
[26] K.M. Ali and M.J. Pazzani, "On the Link Between Error Correlation and Error Reduction in Decision Tree Ensembles," Technical Report ICS-TR-95-38, Univ. of California, Irvine, Department of Information and Computer Science, 1995.
[27] A.H.-R. Ko and R. Sabourin, "Compound Diversity Functions for Ensemble Selection," Int'l J. Pattern Recognition and Artificial Intelligence, vol. 23, no. 4, pp. 659-686, 2009.
[28] L.I. Kuncheva and C.J. Whitaker, "Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy," Machine Learning, vol. 51, pp. 181-207, 2003.
[29] L.I. Kuncheva and C.J. Whitaker, "Ten Measures of Diversity in Classifier Ensembles: Limits for Two Classifiers," Proc. A DERA/IEE Workshop Intelligent Sensor Processing (Ref. No. 2001/050), pp. 1-10, 2001.
[30] K. Tang, P.N. Suganthan, and X. Yao, "An Analysis of Diversity Measures," Machine Learning, vol. 65, pp. 247-271, 2006.
[31] G. Giacinto and F. Roli, "Design of Effective Neural Network Ensembles for Image Classification Purposes," Image and Vision Computing, vol. 19, nos. 9/10, pp. 699-707, 2000.
[32] P. Cunningham and J. Carney, "Diversity versus Quality in Classification Ensembles Based on Feature Selection," Proc. 11th European Conf. Machine Learning (ECML '00), pp. 109-116, 2000.
[33] D. Partridge and W. Krzanowski, "Software Diversity: Practical Statistics for Its Measurement and Exploitation," Information and Software Technology, vol. 39, no. 10, pp. 707-717, 1997.
[34] K. Tumer and J. Ghosh, "Linear and Order Statistics Combiners for Pattern Classification," Combining Artificial Neural Networks, pp. 127-162, 1999.
[35] K. Tumer and J. Ghosh, "Analysis of Decision Boundaries in Linearly Combined Neural Classifiers," Pattern Recognition, vol. 29, no. 2, pp. 341-348, 1996.
[36] Q. Gu, Z. Cai, L. Zhu, and B. Huang, "Data Mining on Imbalanced Data Sets," Proc. Int'l Conf. Advanced Computer Theory and Eng., pp. 1020-1024, 2008.
[37] X. Guo, Y. Yin, C. Dong, G. Yang, and G. Zhou, "On the Class Imbalance Problem," Proc. Fourth Int'l Conf. Natural Computation (ICNC '08), pp. 192-201, 2008.
[38] F. Provost, "Machine Learning from Imbalanced Data Sets 101," Proc. AAAI '00 Workshop Imbalanced Data Sets, 2000.
[39] H. He and E.A. Garcia, "Learning from Imbalanced Data," IEEE Trans. Knowledge and Data Eng., vol. 21, no. 9, pp. 1263-1284, Sept. 2009.
[40] Y. Sun, M.S. Kamel, A.K. Wong, and Y. Wang, "Cost-Sensitive Boosting for Classification of Imbalanced Data," Pattern Recognition, vol. 40, no. 12, pp. 3358-3378, 2007.
[41] P.K. Chan and S.J. Stolfo, "Toward Scalable Learning with Non-Uniform Class and Cost Distributions: A Case Study in Credit Card Fraud Detection," Proc. Fourth Int'l Conf. Knowledge Discovery and Data Mining, pp. 164-168, 1998.
[42] L. Breiman, "Bagging Predictors," Machine Learning, vol. 24, no. 2, pp. 123-140, 1996.
[43] Y. Freund and R.E. Schapire, "Experiments with a New Boosting Algorithm," Proc. 13th Int'l Conf. Machine Learning, pp. 148-156, 1996.
[44] K.M. Ting, "A Comparative Study of Cost-Sensitive Boosting Algorithms," Proc. 17th Int'l Conf. Machine Learning, pp. 983-990, 2000.
[45] C. Drummond and R.C. Holte, "C4.5, Class Imbalance, and Cost Sensitivity: Why Under-Sampling Beats Over-Sampling," Proc. Int'l Conf. Machine Learning (ICML '03) Workshop Learning from Imbalanced Data Sets, pp. 1-8, 2003.
[46] M.V. Joshi, "On Evaluating Performance of Classifiers for Rare Classes," Proc. IEEE Int'l Conf. Data Mining, pp. 641-661, 2002.
[47] Y. Sun, M.S. Kamel, and Y. Wang, "Boosting for Learning Multiple Classes with Imbalanced Class Distribution," Proc. Sixth Int'l Conf. Data Mining (ICDM '06), pp. 592-602, 2006.
[48] M. Kubat, R. Holte, and S. Matwin, "Learning When Negative Examples Abound," Proc. Ninth European Conf. Machine Learning, vol. 1224, pp. 146-153, 1997.
[49] M. Kubat and S. Matwin, "Addressing the Curse of Imbalanced Training Sets: One-Sided Selection," Proc. 14th Int'l Conf. Machine Learning, pp. 179-186, 1997.
[50] A.P. Bradley, "The Use of the Area Under the Roc Curve in the Evaluation of Machine Learning Algorithms," Pattern Recognition, vol. 30, no. 7, pp. 1145-1159, 1997.
[51] C.X. Ling, J. Huang, and H. Zhang, "Auc: A Statistically Consistent and More Discriminating Measure Than Accuracy," Proc. 18th Int'l Conf. Artificial Intelligence (IJCAI '03), pp. 329-341, 2003.
[52] J.A. Swets, "Measuring the Accuracy of Diagnostic Systems," Science, vol. 240, no. 4857, pp. 1285-1293, 1988.
[53] M. Skurichina, L.I. Kuncheva, and R.P. Duin, "Bagging and Boosting for the Nearest Mean Classifier - Effects of Sample Size on Diversity and Accuracy," Proc. Third Int'l Workshop Multiple Classifier Systems, pp. 307-311, 2002.
[54] R.E. Schapire, Y. Freund, P. Bartlett, and W.S. Lee, "Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods," The Ann. Statistics, vol. 26, no. 5, pp. 1651-1686, 1998.
[55] G. Boetticher, T. Menzies, and T.J. Ostrand Promise Repository of Empirical Software Eng. Data. repository, 2007.
[56] P.V.D. Putten, and M.V. Someren, "A Bias-Variance Analysis of a Real World Learning Problem: The Coil Challenge 2000," Machine Learning, vol. 57, nos. 1/2, pp. 177-195, 2004.
[57] A. Frank and A. Asuncion UCI Machine Learning Repository: http://archive.ics.uci.eduml, 2010.
[58] N. Garcia-Pedrajas, C. Hervas-Martinez, and D. Ortiz-Boyer, "Cooperative Coevolution of Artificial Neural Network Ensembles for Pattern Classification," IEEE Trans. Evolutionary Computation, vol. 9, no. 3, pp. 271-302, June 2005.