Issue No. 09 - Sept. (2016 vol. 28)
Bo Tang , Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI
Steven Kay , Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI
Haibo He , Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI
Automated feature selection is important for text categorization to reduce feature size and to speed up learning process of classifiers. In this paper, we present a novel and efficient feature selection framework based on the Information Theory, which aims to rank the features with their discriminative capacity for classification. We first revisit two information measures: Kullback-Leibler divergence and Jeffreys divergence for binary hypothesis testing, and analyze their asymptotic properties relating to type I and type II errors of a Bayesian classifier. We then introduce a new divergence measure, called Jeffreys-Multi-Hypothesis (JMH) divergence, to measure multi-distribution divergence for multi-class classification. Based on the JMH-divergence, we develop two efficient feature selection methods, termed maximum discrimination (
) and methods, for text categorization. The promising results of extensive experiments demonstrate the effectiveness of the proposed approaches. $_$MD$_$
Text categorization, Vocabulary, Bayes methods, Biomedical measurement, Computational efficiency, Frequency measurement, Support vector machines
B. Tang, S. Kay and H. He, "Toward Optimal Feature Selection in Naive Bayes for Text Categorization," in IEEE Transactions on Knowledge & Data Engineering, vol. 28, no. 9, pp. 2508-2521, 2016.