The Community for Technology Leaders
RSS Icon
Issue No.06 - June (2010 vol.22)
pp: 812-825
Jun Du , The University of Western Ontario, London
Charles X. Ling , The University of Western Ontario, London
With the assistance of a domain expert, active learning can often select or construct fewer examples to request their labels to build an accurate classifier. However, previous works of active learning can only generate and ask specific queries. In real-world applications, the domain experts (or oracles) are often more readily to answer “generalized queries” with don't-care attributes. The power of such generalized queries is that one generalized query is often equivalent to many specific ones. However, overly general queries are not good as answers from the domain experts (or oracles) can be highly uncertain, and this makes learning difficult. In this paper, we propose a novel active learning algorithm that asks good generalized queries. We, then, extend our algorithm to construct new, hierarchical features for both nominal and numeric attributes. We demonstrate experimentally that our new method asks significantly fewer queries compared with the previous works of active learning, even when the initial labeled data set is very small, and the oracle is inaccurate in class probability estimations. Our method can be readily deployed in real-world data mining tasks where obtaining labeled examples is costly.
Active learning, domain expert, generalized query.
Jun Du, Charles X. Ling, "Asking Generalized Queries to Domain Experts to Improve Learning", IEEE Transactions on Knowledge & Data Engineering, vol.22, no. 6, pp. 812-825, June 2010, doi:10.1109/TKDE.2010.33
[1] N. Abe and H. Mamitsuka, "Query Learning Strategies Using Boosting and Bagging," Proc. 15th Int'l Conf. Machine Learning (ICML '98), pp. 1-9, 1998.
[2] C.C. Aggarwal, "Towards Effective and Interpretable Data Mining by Visual Interaction," SIGKDD Explorations, vol. 3, pp. 11-22, 2002.
[3] R. Agrawal, T. Imielinski, and A.N. Swami, "Mining Association Rules between Sets of Items in Large Databases," Proc. 1993 ACM SIGMOD Int'l Conf. Management of Data, P. Buneman and S. Jajodia, eds., pp. 207-216, 1993.
[4] D. Angluin, "Queries and Concept Learning," Machine Learning, vol. 2, no. 4, pp. 319-342, Apr. 1988.
[5] M. Ankerst, "Report on the SIGKDD-2002 Panel the Perfect Data Mining Tool: Interactive or Automated?" SIGKDD Explorations Newsletter, vol. 4, no. 2, pp. 110-111, 2002.
[6] A. Asuncion and D.J. Newman, UCI Machine Learning Repository, , 2007.
[7] Y. Baram, R. El-Yaniv, and K. Luz, "Online Choice of Active Learning Algorithms," J. Machine Learning Research, vol. 5, pp. 255-291, 2004.
[8] L. Breiman, "Bagging Predictors," Machine Learning, vol. 24, no. 2, pp. 123-140, 1996.
[9] L. Cao, "Introduction to Domain Driven Data Mining," Data Mining for Business Applications, pp. 3-10, Springer, 2009.
[10] L. Cao and C. Zhang, "Domain-Driven Data Mining: A Practical Methodology," Int'l J. Data Warehousing and Mining, vol. 2, no. 4, pp. 49-65, 2006.
[11] D.A. Cohn, L. Atlas, and R.E. Ladner, "Improving Generalization with Active Learning," Machine Learning, vol. 15, no. 2, pp. 201-221, 1994.
[12] D.A. Cohn, Z. Ghahramani, and M.I. Jordan, "Active Learning with Statistical Models," J. Artificial Intelligence Research, vol. 4, pp. 129-145, 1996.
[13] W. Dai, Q. Yang, G.R. Xue, and Y. Yu, "Boosting for Transfer Learning," Proc. 24th Int'l Conf. Machine Learning (ICML '07), pp. 193-200, 2007.
[14] L.S. Dasgupta, "Coarse Sample Complexity Bounds for Active Learning," Proc. Neural Information Processing Systems, 2005.
[15] G. Druck, G.S. Mann, and A. McCallum, "Learning from Labeled Features Using Generalized Expectation Criteria," S.H. Myaeng, D.W. Oard, F. Sebastiani, T.S. Chua, and M.K. Leong, eds., Proc. ACM SIGIR, pp. 595-602, 2008.
[16] J. Du and C.X. Ling, "Active Learning with Generalized Queries," Proc. Ninth IEEE Int'l Conf. Data Mining, pp. 120-128, 2009.
[17] C. Elkan, "The Foundations of Cost-Sensitive Learning," Proc. 17th Int'l Joint Conf. Artificial Intelligence, pp. 973-978, 2001.
[18] Y. Guo and D. Schuurmans, "Discriminative Batch Mode Active Learning," Proc. Advances in Neural Information Processing Systems (NIPS), pp. 593-600, 2008.
[19] I. Guyon and A. Elisseeff, "An Introduction to Variable and Feature Selection," J. Machine Learning Research, vol. 3, pp. 1157-1182, Mar. 2003.
[20] S.C.H. Hoi, R. Jin, J. Zhu, and M.R. Lyu, "Batch Mode Active Learning and its Application to Medical Image Classification," Proc. 23rd Int'l Conf. Machine Learning (ICML '06), pp. 417-424, 2006.
[21] D.D. Lewis and J. Catlett, "Heterogeneous Uncertainty Sampling for Supervised Learning," Proc. 11th Int'l Conf. Machine Learning (ICML '94), W.W. Cohen and H. Hirsh, eds., pp. 148-156, 1994.
[22] M. Lindenbaum, S. Markovitch, and D. Rusakov, "Selective Sampling for Nearest Neighbor Classifiers," Machine Learning, vol. 54, no. 2, pp. 125-152, Feb. 2004.
[23] C.X. Ling and J. Du, "Active Learning with Direct Query Construction," Proc. 14th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD '08), pp. 480-487, 2008.
[24] B. Liu, W. Hsu, and Y. Ma, "Integrating Classification and Association Rule Mining," Proc. Knowledge Discovery and Data Mining, pp. 80-86, 1998.
[25] H. Liu and H. Motoda, Feature Selection for Knowledge Discovery and Data Mining. Kluwer Academic Publishers, 1998.
[26] Computational Methods of Feature Selection, H. Liu and H. Motoda, eds., first ed. Chapman & Hall/CRC, Oct. 2007.
[27] O. Longbing Ca and G. Chengqi Zhan, "The Evolution of KDD: Towards Domain-Driven Data Mining," Int'l J. Pattern Recognition and Artificial Intelligence, vol. 21, no. 4, pp. 677-692, 2007.
[28] D.D. Margineantu, "Active Cost-Sensitive Learning," Proc. 19th Int'l Joint Conf. Artificial Intelligence, 2005.
[29] F. Provost and P. Domingos, "Tree Induction for Probability-Based Ranking," Machine Learning, vol. 52, no. 3, pp. 199-215, Sept. 2003.
[30] R. Raina, A. Battle, H. Lee, B. Packer, and A.Y. Ng, "Self-Taught Learning: Transfer Learning from Unlabeled Data," Proc. 24th Int'l Conf. Machine Learning (ICML '07), pp. 759-766, 2007.
[31] N. Roy and A. Mccallum, "Toward Optimal Active Learning through Sampling Estimation of Error Reduction," Proc. 18th Int'l Conf. Machine Learning, pp. 441-448, 2001.
[32] M. Saar-Tsechansky and F. Provost, "Active Sampling for Class Probability Estimation and Ranking," Machine Learning, vol. 54, no. 2, pp. 153-178, Feb. 2004.
[33] H.S. Seung, M. Opper, and H. Sompolinsky, "Query by Committee," Proc. Fifth Ann. Workshop Computational Learning Theory (COLT '92), pp. 287-294, 1992.
[34] V.S. Sheng and C.X. Ling, "Thresholding for Making Classifiers Cost-Sensitive," Proc. 21t Nat'l Conf. Artificial Intelligence (AAAI '06), 2006.
[35] J.W. Smith, J.E. Everhart, W.C. Dickson, W.C. Knowler, and R.S. Johannes, "Using the ADAP Learning Algorithm to Forecast the Onset of Diabetes Mellitus," Proc. Symp. Computer Applications and Medical Care, pp. 261-265, 1988.
[36] S. Tong and D. Koller, "Support Vector Machine Active Learning with Applications to Text Classification," J. Machine Learning Research, vol. 2, pp. 45-66, 2002.
[37] I.H. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques. second ed., Morgan Kaufmann, June 2005.
[38] Proc. 2007 Int'l Workshop Domain Driven Data Mining (DDDM '07), P. Yu, ed., 2007.
[39] T. Zhang and F.J. Oles, "A Probability Analysis on the Value of Unlabeled Data for Classification Problems," Proc. 17th Int'l Conf. Machine Learning, pp. 1191-1198, 2000.
375 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool