The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.10 - Oct. (2013 vol.25)
pp: 2283-2301
Gregory Ditzler , Drexel University, Philadelphia
Robi Polikar , Rowan University, Glassboro
ABSTRACT
Learning in nonstationary environments, also known as learning concept drift, is concerned with learning from data whose statistical characteristics change over time. Concept drift is further complicated if the data set is class imbalanced. While these two issues have been independently addressed, their joint treatment has been mostly underexplored. We describe two ensemble-based approaches for learning concept drift from imbalanced data. Our first approach is a logical combination of our previously introduced Learn++.NSE algorithm for concept drift, with the well-established SMOTE for learning from imbalanced data. Our second approach makes two major modifications to Learn++.NSE-SMOTE integration by replacing SMOTE with a subensemble that makes strategic use of minority class data; and replacing Learn++.NSE and its class-independent error weighting mechanism with a penalty constraint that forces the algorithm to balance accuracy on all classes. The primary novelty of this approach is in determining the voting weights for combining ensemble members, based on each classifier's time and imbalance-adjusted accuracy on current and past environments. Favorable results in comparison to other approaches indicate that both approaches are able to address this challenging problem, each with its own specific areas of strength. We also release all experimental data as a resource and benchmark for future research.
INDEX TERMS
Classification algorithms, Heuristic algorithms, Joints, Electronic mail, Knowledge engineering, Data models, Algorithm design and analysis, class imbalance, Classification algorithms, Heuristic algorithms, Joints, Electronic mail, Knowledge engineering, Data models, Algorithm design and analysis, multiple classifier systems, Incremental learning, concept drift
CITATION
Gregory Ditzler, Robi Polikar, "Incremental Learning of Concept Drift from Streaming Imbalanced Data", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 10, pp. 2283-2301, Oct. 2013, doi:10.1109/TKDE.2012.136
REFERENCES
[1] C. Giraud-Carrier, "A Note on the Utility of Incremental Learning," Artificial Intelligence Comm., vol. 13, no. 4, pp. 215-223, 2000.
[2] S. Lange and S. Zilles, "Formal Models of Incremental Learning and Their Analysis," Proc. Int'l Joint Conf. Neural Networks, vol. 4, pp. 2691-2696, 2003.
[3] M.D. Muhlbaier, A. Topalis, and R. Polikar, "Learn++.NC: Combining Ensemble of Classifiers With Dynamically Weighted Consult-and-Vote for Efficient Incremental Learning of New Classes," IEEE Trans. Neural Networks, vol. 20, no. 1, pp. 152-168, Jan. 2009.
[4] L.I. Kuncheva, "Classifier Ensembles for Changing Environments," Proc. Int'l Workshop Multiple Classifier Systems, pp. 1-15, 2004.
[5] A. Bifet, "Adaptive Learning and Mining for Data Streams and Frequent Patterns." PhD dissertation, Universitat Politècnica de Catalunya, 2009.
[6] S. Grossberg, "Nonlinear Neural Networks: Principles, Mechanisms, and Architectures," Neural Networks, vol. 1, no. 1, pp. 17-61, 1988.
[7] R. Elwell and R. Polikar, "Incremental Learning of Concept Drift in Nonstationary Environments," IEEE Trans. Neural Networks, vol. 22, no. 10, pp. 1517-1531, Oct. 2011.
[8] D.P. Helmbold and P.M. Long, "Tracking Drifting Concepts by Minimizing Disagreements," Machine Learning, vol. 14, no. 1, pp. 27-45, 1994.
[9] J.C. Schlimmer and R.H. Granger, "Incremental Learning from Noisy Data," Machine Learning, vol. 1, no. 3, pp. 317-354, Sept. 1986.
[10] G. Widmer and M. Kubat, "Learning in the Presence of Concept Drift and Hidden Contexts," Machine Learning, vol. 23, no. 1, pp. 69-101, 1996.
[11] J. Case, S. Jain, S. Kaufmann, A. Sharma, and F. Stephan, "Predictive Learning Models for Concept Drift," Theoretical Computer Science, vol. 268, no. 2, pp. 323-349, Oct. 2001.
[12] F.H. Hamker, "Life-Long Learning Cell Structures-Continuously Learning without Catastrophic Interference," Neural Networks, vol. 14, no. 4/5, pp. 551-573, May 2001.
[13] C. Alippi and M. Roveri, "Just-in-Time Adaptive Classifiers - Part I: Detecting Nonstationary Changes," IEEE Trans. Neural Networks, vol. 19, no. 7, pp. 1145-1153, July 2008.
[14] C. Alippi and M. Roveri, "Just-in-Time Adaptive Classifiers - Part II: Designing the Classifier," IEEE Trans. Neural Networks, vol. 19, no. 12, pp. 2053-2064, Dec. 2008.
[15] C. Alippi, G. Boracchi, and M. Roveri, "Just in Time Classifiers: Managing the Slow Drift Case," Proc. Int'l Joint Conf. Neural Networks (IJCNN '09), pp. 114-120, 2009.
[16] C. Alippi, G. Boracchi, and M. Roveri, "Change Detection Tests Using the ICI Rule," Proc. World Congress Computational Intelligence (WCCI '10) - Int'l Joint Conf. Neural Networks (IJCNN '10), pp. 1190-1196, 2010.
[17] P. Vorburger and A. Bernstein, "Entropy-Based Concept Shift Detection," Proc. Int'l Conf. Data Mining (ICDM '06), pp. 1113-1118, 2006.
[18] S. Hoeglinger and R. Pears, "Use of Hoeffding Trees in Concept Based Data Stream Mining," Proc. Int'l Conf. Information and Automation for Sustainability (CIAFS '07), pp. 57-62, 2007.
[19] C.J. Tsai, C.I. Lee, and W.P. Yang, "Mining Decision Rules on Data Streams in the Presence of Concept Drifts," Expert Systems with Applications, vol. 36, no. 2, pp. 1164-1178, Mar. 2009.
[20] G. Hulten, L. Spencer, and P. Domingos, "Mining Time-Changing Data Streams," Proc. Conf. Knowledge Discovery in Data, pp. 97-106, 2001.
[21] L. Cohen, G. Avrahami, M. Last, and A. Kandel, "Info-Fuzzy Algorithms for Mining Dynamic Data Streams," Applied Soft Computing, vol. 8, no. 4, pp. 1283-1294, Sept. 2008.
[22] L. Cohen, G. Avrahami-Bakish, M. Last, A. Kandel, and O. Kipersztok, "Real-Time Data Mining of Non-Stationary Data Streams from Sensor Networks," Information Fusion, vol. 9, no. 3, pp. 344-353, 2008.
[23] M. Baena-Garcia, J.del Campo-Avila, R. Fidalgo, A. Bifet, R. Gavalda, and R. Bueno-Morales, "Early Drift Detection Method," Proc. ECML PKDD Workshop Knowledge Discovery from Data Streams, pp. 77-86, 2006.
[24] L.I. Kuncheva, "Classifier Ensembles for Detecting Concept Change in Streaming Data: Overview and Perspectives," Proc. European Conf. Artificial Intelligence (ECAI), pp. 5-10, 2008.
[25] W.N. Street and Y. Kim, "A Streaming Ensemble Algorithm (SEA) for Large-Scale Classification," Proc. Int'l Conf. Knowledge Discovery and Data Mining, pp. 377-382, 2001.
[26] A. Bifet, G. Holmes, B. Pfahringer, R. Kirkby, and R. Gavalda, "New Ensemble Methods For Evolving Data Streams," Proc. 15th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD 09), pp. 139-148, 2009.
[27] A. Tsymbal, M. Pechenizkiy, P. Cunningham, and S. Puuronen, "Dynamic Integration of Classifiers for Handling Concept Drift," Information Fusion, vol. 9, no. 1, pp. 56-68, Jan. 2008.
[28] J.Z. Kolter and M.A. Maloof, "Dynamic Weighted Majority: An Ensemble Method for Drifting Concepts," J. Machine Learning Research, vol. 8, pp. 2755-2790, 2007.
[29] J. Gao, W. Fan, and J. Han, "On Appropriate Assumptions to Mine Data Streams: Analysis and Practice," Proc. Int'l Conf. Data Mining (ICDM '07), pp. 143-152, 2007.
[30] K. Nishida and K. Yamauchi, "Adaptive Classifiers-Ensemble System for Tracking Concept Drift," Proc. Int'l Conf. Machine Learning and Cybernetics, K. Yamauchi, ed., vol. 6, pp. 3607-3612, 2007.
[31] H. He and S. Chen, "IMORL: Incremental Multiple-Object Recognition and Localization," IEEE Trans. Neural Networks, vol. 19, no. 10, pp. 1727-1738, Oct. 2008.
[32] J. Gao, B. Ding, F. Wei, H. Jiawei, and P.S. Yu, "Classifying Data Streams with Skewed Class Distributions and Concept Drifts," IEEE Internet Computing, vol. 12, no. 6, pp. 37-49, Nov./Dec. 2008.
[33] H. Abdulsalam, D. Skillicorn, and P. Martin, "Classification Using Streaming Random Forests," IEEE Trans. Knowledge and Data Eng., vol. 23, no. 1, pp. 22-36, Jan. 2011.
[34] A. Bifet, E. Frank, G. Holmes, and B. Pfahringer, "Accurate Ensembles for Data Streams: Combining Restricted Hoeffding Trees Using Stacking," Proc. Second Asian Conf. Machine Learning, Tokyo. JMLR: Workshop and Conf. Proc., vol. 13, pp. 225-240, 2010.
[35] A. Bifet, MOA : Massive Online Analysis, http:/moa.cs.waikato. ac.nz, 2011.
[36] T.R. Hoens, N.V. Chawla, and R. Polikar, "Heuristic Updatable Weighted Random Subspaces for Non-Stationary Environments," Proc. IEEE 11th Int'l Conf. Data Mining (ICDM), pp. 241-250, 2011.
[37] M. Muhlbaier and R. Polikar, "An Ensemble Approach for Incremental Learning in Nonstationary Environments," Proc. Seventh Int'l Conf. Multiple Classifier Systems, pp. 490-500, 2007.
[38] M. Karnick, M. Ahiskali, M. Muhlbaier, and R. Polikar, "Learning Concept Drift in Nonstationary Environments Using an Ensemble of Classifiers Based Approach," Proc. IEEE Int'l Joint Conf. Neural Networks, pp. 3455-3462, 2008.
[39] R. Elwell and R. Polikar, "Incremental Learning in Nonstationary Environments with Controlled Forgetting," Proc. Int'l Joint Conf. Neural Networks (IJCNN '09), pp. 771-778, 2009.
[40] R. Elwell and R. Polikar, "Incremental Learning of Variable Rate Concept Drift," Proc. Int'l Workshop Multiple Classifier Systems (MCS '09), pp. 142-151, 2009.
[41] R. Polikar, L. Upda, S.S. Upda, and V. Honavar, "Learn++: An Incremental Learning Algorithm for Supervised Neural Networks," IEEE Trans. Systems, Man, Cybernetics A, Systems, Humans, vol. 31, no. 4, pp. 497-508, Nov. 2001.
[42] M. Kubat, R. Holte, and S. Matwin, "Machine Learning for the Detection of Oil Spills in Satellite Radar Images," Machine Learning, vol. 30, pp. 195-215, 1998.
[43] H. He and E.A. Garcia, "Learning from Imbalanced Data," IEEE Trans. Knowledge and Data Eng., vol. 21, no. 9, pp. 1263-1284, Sept. 2009.
[44] P.E. Hart, "The Condensed Nearest Neighbor Rule," IEEE Trans. Information Theory, vol. IT-14, no. 3, pp. 515-516, May 1968.
[45] I. Tomek, "Two Modifications of CNN," IEEE Trans. Systems, Man and Cybernetics, Part A, vol. SMC-6, no. 11, pp. 769-772, Nov. 1976.
[46] N.V. Chawla, K.W. Bowyer, L.O. Hall, and W.P. Kegelmeyer, "SMOTE: Synthetic Minority Oversampling Technique," J. Artificial Intelligence, vol. 16, pp. 321-357, 2002.
[47] N.V. Chawla, A. Lazarevic, L.O. Hall, and K.W. Bowyer, "SMOTEBoost: Improving Prediction of the Minority Class in Boosting," Proc. Seventh European Conf. Principles and Practice of Knowledge Discovery in Databases (PKDD), pp. 107-119, 2003.
[48] C. Li, "Classifying Imbalanced Data using a Bagging Ensemble Variation (BEV)," Proc. ACM Southeast Regional Conf., pp. 203-208, 2007.
[49] G. Ditzler, M. Muhlbaier, and R. Polikar, "Incremental Learning of New Classes in Unbalanced Data Sets: Learn++.UDNC," Proc. Int'l Workshop Multiple Classifier Systems (MCS '10), pp. 33-42, 2010.
[50] H. Guo and H. Viktor, "Learning form Imbalanced Data sets with Boosting and Data Generation: The DataBoost-IM Approach," ACM SIGKDD Explorations Newsletter, vol. 6, no. 1, pp. 30-39, 2004.
[51] T. Yuchun, Z. Yan-Qing, N.V. Chawla, and S. Krasser, "SVMs Modeling for Highly Imbalanced Classification," IEEE Trans. Systems, Man, and Cybernetics, Part B: Cybernetics, vol. 39, no. 1, pp. 281-288, Feb. 2009.
[52] R. Batuwita and V. Palade, "FSVM-CIL: Fuzzy Support Vector Machines for Class Imbalance Learning," IEEE Trans. Fuzzy Systems, vol. 18, no. 3, pp. 558-571, June 2010.
[53] J. Gao, W. Fan, J. Han, and P.S. Yu, "A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions," Proc. SIAM Int'l Conf. Data Mining, vol. 7, 2007.
[54] S. Chen and H. He, "SERA: Selectively Recursive Approach Towards Nonstationary Imbalanced Stream Data Mining," Proc. Int'l Joint Conf. Neural Networks (IJCNN '09), pp. 522-529, 2009.
[55] S. Chen, H. He, L. Kang, and S. Desai, "MuSeRA: Multiple Selectively Recursive Approach Towards Imbalanced Stream Data Mining," Proc. World Congress Computer Intelligence (WCCI '10) - Int'l Joint Conf. Neural Networks (IJCNN '10), pp. 1-8, 2010.
[56] S. Chen and H. He, "Towards Incremental Learning of Nonstationary Imbalanced Data Stream: A Multiple Selectively Recursive Approach," Evolving Systems, vol. 2, no. 1, pp. 35-50, 2011.
[57] E.S. Xioufis, M. Spiliopoulou, G. Tsoumakas, and I. Vlahavas, "Dealing with Concept Drift and Class Imbalance in Multi-Label Stream Classification," Proc. Int'l Joint Conf. Artificial Intelligence (IJCAI '11), pp. 1583-1588, 2011.
[58] G. Ditzler, R. Polikar, and N. Chawla, "An Incremental Learning Algorithm for Non-Stationary Environments and Class Imbalance," Proc. 20th Int'l Conf. Pattern Recognition (ICPR '10), pp. 2997-3000, 2010.
[59] Y. Freund and R.E. Schapire, "Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting," J. Computer and System Sciences, vol. 55, no. 1, pp. 119-139, 1997.
[60] G. Ditzler and R. Polikar, "An ensemble Based Incremental Learning Framework for Concept Drift and Class Imbalance," Proc. World Congress Computational Intelligence (WCCI '10) - Int'l Joint Conf. Neural Networks (IJCNN '10), pp. 1-8, 2010.
[61] R. Polikar and G. Ditzler, Benchmark Data Sets for Evaluating Concept Drift / Imbalanced Data Algorithms, http://users.rowan. edu/~polikar/research NIE_data, 2012.
[62] M.B. Harries, SPLICE-2 Comparative Evaluation: Electricity Pricing, ftp://ftp.cse.unsw.edu.au/pub/doc/papers/UNSW/9905.pdf, 2012.
[63] "U.S.National Oceanic and Atmospheric Administration (NOAA)," Fed. Climate Complex Global Surface Summary of Day Data - Version 7 - USAF Datsav3 Station # 725540, ftp://ftp. ncdc.noaa.gov/pub/datagsod, 2012.
[64] J. Demsar, "Statistical Comparisons of Classifiers over Multiple Data Sets," J. Machine Learning Research, vol. 7, pp. 1-30, 2006.
45 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool