The Community for Technology Leaders
RSS Icon
Issue No.05 - May (2013 vol.25)
pp: 961-973
Thomas Verbraken , Katholieke Universiteit Leuven, Leuven
Wouter Verbeke , University of Edinburgh, Edinburgh
Bart Baesens , Katholieke Universiteit Leuven, Leuven and University of Southampton, Highfield Southampton
The interest for data mining techniques has increased tremendously during the past decades, and numerous classification techniques have been applied in a wide range of business applications. Hence, the need for adequate performance measures has become more important than ever. In this paper, a cost-benefit analysis framework is formalized in order to define performance measures which are aligned with the main objectives of the end users, i.e., profit maximization. A new performance measure is defined, the expected maximum profit criterion. This general framework is then applied to the customer churn problem with its particular cost-benefit structure. The advantage of this approach is that it assists companies with selecting the classifier which maximizes the profit. Moreover, it aids with the practical implementation in the sense that it provides guidance about the fraction of the customer base to be included in the retention campaign.
Business, Knowledge engineering, Area measurement, Receivers, Data engineering, Educational institutions, performance measures, Data mining, classification
Thomas Verbraken, Wouter Verbeke, Bart Baesens, "A Novel Profit Maximizing Metric for Measuring Classification Performance of Customer Churn Prediction Models", IEEE Transactions on Knowledge & Data Engineering, vol.25, no. 5, pp. 961-973, May 2013, doi:10.1109/TKDE.2012.50
[1] T. Fawcett, "An Introduction to ROC Analysis," Pattern Recognition Letters, vol. 27, no. 8, pp. 861-874, 2006.
[2] D. Hand, "Measuring Classifier Performance: A Coherent Alternative to the Area under the ROC Curve," Machine Learning, vol. 77, no. 1, pp. 103-123, 2009.
[3] W. Verbeke, D. Martens, C. Mues, and B. Baesens, "Building Comprehensible Customer Churn Prediction Models with Advanced Rule Induction Techniques," Expert Systems with Applications, vol. 38, pp. 2354-2364, 2011.
[4] A.A. Nanavati, R. Singh, D. Chakraborty, K. Dasgupta, S. Mukherjea, G. Das, S. Gurumurthy, and A. Joshi, "Analyzing the Structure and Evolution of Massive Telecom Graphs," IEEE Trans. Knowledge and Data Eng., vol. 20, no. 5, pp. 703-718, May 2008.
[5] S. Ali and K. Smith, "On Learning Algorithm Selection for Classification," Applied Soft Computing, vol. 6, no. 2, pp. 119-138, 2006.
[6] P. Baldi, S. Brunak, Y. Chauvin, C. Andersen, and H. Nielsen, "Assessing the Accuracy of Prediction Algorithms for Classification: An Overview," Bioinformatics, vol. 16, no. 5, pp. 412-424, 2000.
[7] N. Chawla, "Data Mining for Imbalanced Datasets: An Overview," Data Mining and Knowledge Discovery Handbook, pp. 875-886, Springer, 2010.
[8] P. Domingos, "Metacost: A General Method for Making Classifiers Cost-Sensitive," Proc. Fifth ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining, pp. 155-164, 1999.
[9] F. Provost and T. Fawcett, "Robust Classification for Imprecise Environments," Machine Learning, vol. 42, no. 3, pp. 203-231, 2001.
[10] A. Bernstein, F. Provost, and S. Hill, "Toward Intelligent Assistance for a Data Mining Process: An Ontology-Based Approach for Cost-Sensitive Classification," IEEE Trans. Knowledge and Data Eng., vol. 17, no. 4, pp. 503-518, Apr. 2005.
[11] Z. Zhou and X. Liu, "Training Cost-Sensitive Neural Networks with Methods Addressing the Class Imbalance Problem," IEEE Trans. Knowledge and Data Eng., vol. 18, no. 1, pp. 63-77, Jan. 2006.
[12] C. Elkan, "The Foundations of Cost-Sensitive Learning," Proc. Int'l Joint Conf. Artificial Intelligence, vol. 17, no. 1, pp. 973-978, 2001.
[13] W. Verbeke, K. Dejaeger, D. Martens, J. Hur, and B. Baesens, "New Insights into Churn Prediction in the Telecommunication Sector: A Profit Driven Data Mining Approach," European J. Operational Research, vol. 218, no. 1, pp. 211-229, 2012.
[14] A. Bradley, "The Use of the Area under the ROC Curve in the Evaluation of Machine Learning Algorithms," Pattern Recognition, vol. 30, no. 7, pp. 1145-1159, 1997.
[15] R. Prati, G. Batista, and M. Monard, "A Survey on Graphical Methods for Classification Predictive Performance Evaluation," IEEE Trans. Knowledge and Data Eng., vol. 23, no 11, pp. 1601-1618, Nov. 2011.
[16] S. Neslin, S. Gupta, W. Kamakura, J. Lu, and C. Mason, "Detection Defection: Measuring and Understanding the Predictive Accuracy of Customer Churn Models," J. Marketing Research, vol. 43, no. 2, pp. 204-211, 2006.
[17] J. Burez and D. Van den Poel, "CRM at a Pay-TV Company: Using Analytical Models to Reduce Customer Attrition by Targeted Marketing for Subscription Services," Expert Systems with Applications, vol. 32, pp. 277-288, 2007.
[18] M. Mozer, R. Wolniewicz, D. Grimes, E. Johnson, and H. Kaushansky, "Predicting Subscriber Dissatisfaction and Improving Retention in the Wireless Telecommunications Industry," IEEE Trans. Neural Networks, vol. 11, no. 3, pp. 690-696, May 2000.
[19] T. Verbraken, W. Verbeke, D. Martens, and B. Baesens, "Profit Optimizing Customer Churn Prediction with Bayesian Network Classifiers," accepted for publication in Intelligent Data Analysis, 2013.
[20] J. Hur and J. Kim, "A Hybrid Classification Method Using Error Pattern Modeling," Expert Systems with Applications, vol. 34, no. 1, pp. 231-241, 2008.
[21] A. Lemmens and C. Croux, "Bagging and Boosting Classification Trees to Predict Churn," J. Marketing Research, vol. 43, no. 2, pp. 276-286, 2006.
[22] E. Lima, C. Mues, and B. Baesens, "Domain Knowledge Integration in Data Mining Using Decision Tables: Case Studies in Churn Prediction," J. Operational Research Soc., vol. 60, no. 8, pp. 1096-1106, 2009.
[23] A. Van der Vaart, Asymptotic Statistics. Cambridge Univ Press, 2000.
[24] P. Tan, M. Steinbach, and V. Kumar, Introduction to Data Mining. Pearson Education, 2006.
[25] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations. Morgan Kaufmann Publishers, Inc., 2000.
[26] B. Baesens, R. Setiono, C. Mues, and J. Vanthienen, "Using Neural Network Rule Extraction and Decision Tables for Credit-Risk Evaluation," Management Science, vol. 49, no. 3, pp. 312-329, 2003.
[27] G. Cui, M. Wong, and H. Lui, "Machine Learning for Direct Marketing Response Models: Bayesian Networks with Evolutionary Programming," Management Science, vol. 52, no. 4, pp. 597-612, 2006.
[28] S. Viaene, R.A. Derrig, B. Baesens, and G. Dedene, "A Comparison of State-of-the-Art Classification Techniques for Expert Automobile Insurance Claim Fraud Detection," J. Risk and Insurance, vol. 69, no. 3, pp. pp. 373-421, 2002.
[29] P. Domingos, "Mining Social Networks for Viral Marketing," IEEE Intelligent Systems, vol. 20, no. 1, pp. 80-82, 2005.
34 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool