Issue No. 07 - July (2007 vol. 19)
Wee-Keong Ng , IEEE Computer Society
Ying Liu , IEEE Computer Society
For many clustering algorithms, such as K-Means, EM, and CLOPE, there is usually a requirement to set some parameters. Often, these parameters directly or indirectly control the number of clusters, that is, k, to return. In the presence of different data characteristics and analysis contexts, it is often difficult for the user to estimate the number of clusters in the data set. This is especially true in text collections such as Web documents, images, or biological data. In an effort to improve the effectiveness of clustering, we seek the answer to a fundamental question: How can we effectively estimate the number of clusters in a given data set? We propose an efficient method based on spectra analysis of eigenvalues (not eigenvectors) of the data set as the solution to the above. We first present the relationship between a data set and its underlying spectra with theoretical and experimental results. We then show how our method is capable of suggesting a range of k that is well suited to different analysis contexts. Finally, we conclude with further empirical results to show how the answer to this fundamental question enhances the clustering process for large text collections.
Clustering, spectral methods, eigenvalues, eigenvectors.
K. Ong, W. Li, Y. Liu and W. Ng, "Enhancing the Effectiveness of Clustering with Spectra Analysis," in IEEE Transactions on Knowledge & Data Engineering, vol. 19, no. , pp. 887-902, 2007.