This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Evaluation of Stability of k-Means Cluster Ensembles with Respect to Random Initialization
November 2006 (vol. 28 no. 11)
pp. 1798-1808
L.I. Kuncheva, Sch. of Informatics, Univ. of Wales, Bangor
Many clustering algorithms, including cluster ensembles, rely on a random component. Stability of the results across different runs is considered to be an asset of the algorithm. The cluster ensembles considered here are based on k-means clusterers. Each clusterer is assigned a random target number of clusters, k and is started from a random initialization. Here, we use 10 artificial and 10 real data sets to study ensemble stability with respect to random k, and random initialization. The data sets were chosen to have a small number of clusters (two to seven) and a moderate number of data points (up to a few hundred). Pairwise stability is defined as the adjusted Rand index between pairs of clusterers in the ensemble, averaged across all pairs. Nonpairwise stability is defined as the entropy of the consensus matrix of the ensemble. An experimental comparison with the stability of the standard k-means algorithm was carried out for k from 2 to 20. The results revealed that ensembles are generally more stable, markedly so for larger k. To establish whether stability can serve as a cluster validity index, we first looked at the relationship between stability and accuracy with respect to the number of clusters, k. We found that such a relationship strongly depends on the data set, varying from almost perfect positive correlation (0.97, for the glass data) to almost perfect negative correlation (-0.93, for the crabs data). We propose a new combined stability index to be the sum of the pairwise individual and ensemble stabilities. This index was found to correlate better with the ensemble accuracy. Following the hypothesis that a point of stability of a clustering algorithm corresponds to a structure found in the data, we used the stability measures to pick the number of clusters. The combined stability index gave best results

[1] H. Ayad and M. Kamel, “Finding Natural Clusters Using Multiclusterer Combiner Based on Shared Nearest Neighbors,” Proc. Fourth Int'l Workshop Multiple Classifier Systems, 2003.
[2] C.L. Blake and C.J. Merz, “UCI Repository of Machine Learning Databases,” 1998, http://www.ics.uci.edu/~mlearnMLRepository.html .
[3] O. Bousquet and A. Elisseeff, “Stability and Generalization,” J.Machine Learning Research, vol. 2, no. 3, pp. 499-526, 2002.
[4] R.O. Duda and P.E. Hart, Pattern Classification and Scene Analysis. John Wiley and Sons, 1973.
[5] S. Dudoit and J. Fridlyand, “Bagging to Improve the Accuracy of a Clustering Procedure,” Bioinformatics, vol. 19, no. 9, pp. 1090-1099, 2003.
[6] A. Elisseeff, T. Evgeniou, and M. Pontil, “Stability of Randomized Learning Algorithms,” J. Machine Learning Research, vol. 6, no. 1, pp. 55-79, 2005.
[7] A. Ben-Hur, A. Elisseeff, and I. Guyon, “A Stability Based Method for Discovering Structure in Clustered Data,” Proc. Pacific Symp. Biocomputing, pp. 6-17, 2002.
[8] X.Z. Fern and C.E. Brodley, “Random Projection for High Dimensional Data Clustering: A Cluster Ensemble Approach,” Proc. 20th Int'l Conf. Machine Learning, pp. 186-193, 2003.
[9] B. Fischer and J.M. Buhmann, “Bagging for Path-Based Clustering,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 11, pp. 1411-1415, Nov. 2003.
[10] A. Fred, “Finding Consistent Clusters in Data Partitions,” Proc. Second Int'l Workshop Multiple Classifier Systems, 2001.
[11] A. Fred and A.K. Jain, “Data Clustering Using Evidence Accumulation,” Proc. 16th Int'l Conf. Pattern Recognition, pp. 276-280, 2002.
[12] A. Fred and A.K. Jain, “Combining Multiple Clusterungs Using Evidence Accumulation,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 6, pp. 835-850, June 2005.
[13] A. Fred and A.K. Jain, “Robust Data Clustering,” Proc. IEEE CS Conf. Computer Vision and Pattern Recognition, 2003.
[14] J. Ghosh, “Multiclassifier Systems: Back to the Future,” Proc. Third Int'l Workshop Multiple Classifier Systems, 2002.
[15] A.D. Gordon, Classification. Boca Raton, Fla.: Chapman and Hall/CRC, 1999.
[16] D. Greene, A. Tsymbal, N. Bolshakova, and P. Cunningham, “Ensemble Clustering in Medical Diagnostics,” Technical Report TCD-CS-2004-12, Dept. of Computer Science, Trinity College, Dublin, Ireland, 2004.
[17] S.T. Hadjitodorov, L.I. Kuncheva, and L.P. Todorova, “Moderate Diversity for Better Cluster Ensembles,” Information Fusion, 2005.
[18] X. Hu and I. Yoo, “Cluster Ensemble and Its Applications in Gene Expression Analysis,” Proc. Second Asia-Pacific Bioinformatics Conf., 2004.
[19] L. Hubert and P. Arabie, “Comparing Partitions,” J. Classification, vol. 2, pp. 193-218, 1985.
[20] A.K. Jain and R.C. Dubes, Algorithms for Clustering Data. Prentice Hall, 1988.
[21] L.I. Kuncheva and S.T. Hadjitodorov, “Using Diversity in Cluster Ensembles,” Proc. IEEE Int'l Conf. Systems, Man, and Cybernetics, 2004.
[22] T. Lange, V. Roth, M.L. Braun, and J.M. Buhmann, “Stability-Based Validation of Clustering Solutions,” Neural Computation, vol. 16, pp. 1299-1323, 2004.
[23] M.H. Law and A.K. Jain, “Cluster Validity by Boostrapping Partitions,” Technical Report MSU-CSE-03-5, Michigan State Univ., 2003.
[24] E. Levine and E. Domany, “Resampling Method for Unsupervised Estimation of Cluster Validity,” Neural Computation, vol. 13, pp.2573-2593, 2001.
[25] U. Maulik and S. Bandyopadhyay, “Performance Evaluation of Some Clustering Algorithms and Validity Indices,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 24, no. 12, pp. 1650-1654, Dec. 2002.
[26] B. Minaei, A. Topchy, and W. Punch, “Ensembles of Partitions via Data Resampling,” Proc. Int'l Conf. Information Technology: Coding and Computing, 2004.
[27] S. Monti, P. Tamayo, J. Mesirov, and T. Golub, “Consensus Clustering: A Resampling Based Method for Class Discovery and Visualization of Gene Expression Microarray Data,” Machine Learning, vol. 52, pp. 91-118, 2003.
[28] G. BelMufti, P. Bertrand, and L. ElMoubarki, “Determining the Number of Groups from Measures of Cluster Validity,” Proc. Int'l Symp. Applied Stochastic Models and Data Analysis, pp. 404-414, 2005.
[29] W.M. Rand, “Objective Criteria for the Evaluation of Clustering Methods,” J. Am. Statistical Assoc., vol. 66, pp. 846-850, 1971.
[30] B.D. Ripley, Pattern Recognition and Neural Networks. Cambridge: Univ. Press, 1996.
[31] V. Roth, T. Lange, M. Braun, and J. Buhmann, “A Resampling Approach to Cluster Validation,” Proc. Conf. Computational Statistics, pp. 123-128, 2002.
[32] A. Strehl and J. Ghosh, “Cluster Ensembles—A Knowledge Reuse Framework for Combining Multiple Partitions,” J. Machine Learning Research, vol. 3, pp. 583-618, 2002.
[33] R. Tibshirani, G. Walther, and T. Hastie, “Estimating the Number of Clusters in a Data Set via Gap Statistic,” J. Royal Statistical Soc. B, vol. 63, pp. 411-423, 2001.
[34] A. Topchy, A.K. Jain, and W. Punch, “Combining Multiple Weak Clusterings,” Proc. IEEE Int'l Conf. Data Mining, pp. 331-338, 2003.
[35] A. Topchy, A.K. Jain, and W. Punch, “A Mixture Model for Clustering Ensembles,” Proc. SIAM Conf. Data Mining, pp. 379-390, 2004.
[36] F. Vilarino, L.I. Kuncheva, and P.I. Radeva, “ROC Curves in Video Analysis Optimization in Intestinal Capsule Endoscopy,” Pattern Recognition Letters, 2005.
[37] A. Weingessel, E. Dimitriadou, and K. Hornik, “An Ensemble Method for Clustering,” 2003, working paper, http://www.ci.tuwien.ac.at/ConferencesDSC-2003 .

Index Terms:
Stability,Clustering algorithms,Entropy,Glass,Shape,Voting,Clustering methods,Protocols,Partitioning algorithms,cluster validity.,Clustering,cluster ensembles,stability and diversity
Citation:
L.I. Kuncheva, D.P. Vetrov, "Evaluation of Stability of k-Means Cluster Ensembles with Respect to Random Initialization," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1798-1808, Nov. 2006, doi:10.1109/TPAMI.2006.226
Usage of this product signifies your acceptance of the Terms of Use.