The Community for Technology Leaders
Green Image
Issue No. 09 - September (2008 vol. 20)
ISSN: 1041-4347
pp: 1217-1229
Xiaotie Deng , City University of Hong Kong
Hung Chim , City University of Hong Kong
In this paper, we propose a phrase-based document similarity to compute the pair-wise similarities of documents based on the Suffix Tree Document (STD) model. By mapping each node in the suffix tree of STD model into a unique feature term in the Vector Space Document (VSD) model, the phrase-based document similarity naturally inherits the term tf-idf weighting scheme in computing the document similarity with phrases. We apply the phrase-based document similarity to the group-average Hierarchical Agglomerative Clustering (HAC) algorithm and develop a new document clustering approach. Our evaluation experiments indicate that, the new clustering approach is very effective on clustering the documents of two standard document benchmark corpora OHSUMED and RCV1. The quality of the clustering results significantly surpass the results of traditional single-word \textit{tf-idf} similarity measure in the same HAC algorithm, especially in large document data sets. Furthermore, by studying the property of STD model, we conclude that the feature vector of phrase terms in the STD model can be considered as an expanded feature vector of the traditional single-word terms in the VSD model. This conclusion sufficiently explains why the phrase-based document similarity works much better than the single-word tf-idf similarity measure.
Clustering, Linguistic processing, Trees
Xiaotie Deng, Hung Chim, "Efficient Phrase-Based Document Similarity for Clustering", IEEE Transactions on Knowledge & Data Engineering, vol. 20, no. , pp. 1217-1229, September 2008, doi:10.1109/TKDE.2008.50
107 ms
(Ver )