Issue No. 04 - April (2009 vol. 31)
Man Lan , East China Normal University, Shanghai
Chew Lim Tan , National University of Singapore, Singapore
Jian Su , Institute for Infocomm Research, Singapore
Yue Lu , East China Normal University, Shanghai
In vector space model (VSM), text representation is the task of transforming the content of a textual document into a vector in the term space so that the document could be recognized and classified by a computer or a classifier. Different terms (i.e. words, phrases, or any other indexing units used to identify the contents of a text) have different importance in a text. The term weighting methods assign appropriate weights to the terms to improve the performance of text categorization. In this study, we investigate several widely-used unsupervised (traditional) and supervised term weighting methods on benchmark data collections in combination with SVM and kNN algorithms. In consideration of the distribution of relevant documents in the collection, we propose a new simple supervised term weighting method, i.e. tf.rf, to improve the terms' discriminating power for text categorization task. From the controlled experimental results, these supervised term weighting methods have mixed performance. Specifically, our proposed supervised term weighting method, tf.rf, has a consistently better performance than other term weighting methods while other supervised term weighting methods based on information theory or statistical metric perform the worst in all experiments. On the other hand, the popularly used tf.idf method has not shown a uniformly good performance in terms of different data sets.
Knowledge and data engineering tools and techniques, Clustering, classification, and association rules, Text mining, Database Applications, Database Management, Information Technology and Systems, Indexing methods, Content Analysis and Indexing, Information Storage and Retrieval, Information Technolog, Text analysis, Natural Language Processing, Artificial Intelligence, Computing Methodologies
Y. Lu, M. Lan, J. Su and C. L. Tan, "Supervised and Traditional Term Weighting Methods for Automatic Text Categorization," in IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 31, no. , pp. 721-735, 2008.