CSDL Home IEEE Transactions on Pattern Analysis & Machine Intelligence 2011 vol.33 Issue No.02 - February

Subscribe

Issue No.02 - February (2011 vol.33)

pp: 279-293

Ioannis T. Christou , Athens Information Technology, Paiania and Carnegie-Mellon University, Pittsburgh

DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/TPAMI.2010.85

ABSTRACT

We present a novel optimization-based method for the combination of cluster ensembles for the class of problems with intracluster criteria, such as Minimum-Sum-of-Squares-Clustering (MSSC). We propose a simple and efficient algorithm—called EXAMCE—for this class of problems that is inspired from a Set-Partitioning formulation of the original clustering problem. We prove some theoretical properties of the solutions produced by our algorithm, and in particular that, under general assumptions, though the algorithm recombines solution fragments so as to find the solution of a Set-Covering relaxation of the original formulation, it is guaranteed to find better solutions than the ones in the ensemble. For the MSSC problem in particular, a prototype implementation of our algorithm found a new better solution than the previously best known for 21 of the test instances of the 40-instance TSPLIB benchmark data sets used in [CHECK END OF SENTENCE], [CHECK END OF SENTENCE], and [CHECK END OF SENTENCE], and found a worse-quality solution than the best known only five times. For other published benchmark data sets where the optimal MSSC solution is known, we match them. The algorithm is particularly effective when the number of clusters is large, in which case it is able to escape the local minima found by K-means type algorithms by recombining the solutions in a Set-Covering context. We also establish the stability of the algorithm with extensive computational experiments, by showing that multiple runs of EXAMCE for the same clustering problem instance produce high-quality solutions whose Adjusted Rand Index is consistently above 0.95. Finally, in experiments utilizing external criteria to compute the validity of clustering, EXAMCE is capable of producing high-quality results that are comparable in quality to those of the best known clustering algorithms.

INDEX TERMS

Clustering, machine learning, constrained optimization, combinatorial algorithms.

CITATION

Ioannis T. Christou, "Coordination of Cluster Ensembles via Exact Methods",

*IEEE Transactions on Pattern Analysis & Machine Intelligence*, vol.33, no. 2, pp. 279-293, February 2011, doi:10.1109/TPAMI.2010.85REFERENCES

- [1] P. Hansen and N. Mladenovic, "J-Means: A New Local Search Heuristic for Minimum Sum-of-Squares Clustering,"
Pattern Recognition, vol. 34, no. 2, pp. 405-413, Feb. 2001.- [2] M. Laszlo and S. Mukherjee, "A Genetic Algorithm Using Hyper-Quadtrees for Low-Dimensional K-Means Clustering,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 533-543, Apr. 2006.- [3] J. Pacheco, "A Scatter-Search Approach for the Minimum-Sum-of-Squares Clustering Problem,"
Computers and Operations Research, vol. 32, no. 5, pp. 1325-1335, May 2005.- [4] T. Lange and J.M. Buhmann, "Combining Partitions by Probabilistic Label Aggregation,"
Proc. Int'l Conf. Knowledge Discovery in Databases, 2005.- [5] A.K. Jain and A. Fred, "Evidence Accumulation Clustering Based on the K-Means Algorithm,"
Structural, Syntactic, and Statistical Pattern Recognition, pp. 442-451, Springer, 2002.- [6] B. Fischer and J.M. Buhmann, "Bagging for Path-Based Clustering,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 25, no. 11, pp. 1411-1415, Nov. 2003.- [7] H. Ayad and M. Kamel, "Cumulative Voting Consensus Method for Partitions with Variable Number of Clusters,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 30, no. 1, pp. 160-173, Jan. 2008.- [8] E. Dimitriadou, A. Weingessel, and K. Hornik, "A Combination Scheme for Fuzzy Clustering,"
Int'l J. Pattern Recognition and Artificial Intelligence, vol. 16, no. 7, pp. 901-912, 2002.- [9] L. Breiman, "Bagging Predictors," Technical Report 421, Dept. of Statistics, Univ. of California at Berkeley, 1994.
- [10] L.I. Kuncheva,
Combining Pattern Classifiers. Wiley, 2004.- [11] A. Strehl and J. Ghosh, "Cluster Ensembles—A Knowledge Re-Use Framework for Combining Multiple Partitions,"
J. Machine Learning Research, vol. 3, pp. 583-618, 2002.- [12] A. Topchy, A.K. Jain, and W. Punch, "Clustering Ensembles: Models of Consensus and Weak Partitions,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 12, pp. 1866-1881, Dec. 2005.- [13] A. Asuncion and D.J. Newman, "UCI Machine Learning Repository," School of Information and Computer Science, Univ. of California, http://www.ics.uci.edu/~mlearnMLRepository. html , 2007.
- [14] L.I. Kuncheva and D.P. Vetrov, "Evaluation of Stability of K-Means Cluster Ensembles with Respect to Random Initializations,"
IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 28, no. 11, pp. 1798-1808, Nov. 2006.- [15] M. Meila, "Comparing Clusterings: An Axiomatic View,"
Proc. 22nd Int'l Conf. Machine Learning, pp. 577-584, 2005.- [16] V. Singh, L. Mukherjee, J. Peng, and J. Xu, "Ensemble Clustering Using Semidefinite Programming,"
Advances in Neural Information Processing Systems, J.C. Platt, D. Koller, Y. Singer, and S. Roweis, eds., pp. 1353-1360, MIT Press, 2008.- [17] S. Theodoridis and K. Koutroumbas,
Pattern Recognition, third ed. Academic Press, 2006.- [18] H. Li, K. Zhang, and T. Jiang, "Minimum Entropy Clustering and Applications to Gene Expression Analysis,"
Proc. IEEE Conf. Computational Systems Bioinformatics, pp. 142-151, 2004.- [19] P. Hansen and N. Mladenovic, "Variable Neighborhood Search for the P-Median,"
Location Science, vol. 5, no. 4, pp. 207-226, 1997.- [20] M.G.C. Resende and R.F. Werneck, "A Hybrid Heuristic for the P-Median Problem,"
J. Heuristics, vol. 10, pp. 59-88, 2004.- [21] D. Pelleg and A. Moore, "X-Means: Extending K-Means with Efficient Estimation of the Number of Clusters,"
Proc. 17th Int'l Conf. Machine Learning, pp. 727-734, 2000.- [22] O. du Merle, P. Hansen, B. Jaumard, and N. Mladenovich, "An Interior Point Algorithm for Minimum Sum of Squares Clustering,"
SIAM J. Scientific Computing, vol. 21, no. 4, pp. 1484-1505, Mar. 2000.- [23] P. Hansen and B. Jaumard, "Cluster Analysis and Mathematical Programming,"
Math. Programming, vol. 79, pp. 191-215, 1997.- [24] G.L. Nemhauser and L.A. Wolsey,
Integer and Combinatorial Optimization, first ed. Wiley Interscience, 1988.- [25] G.B. Dantzig and P. Wolfe, "Decomposition Principle for Linear Programs,"
Operations Research, vol. 8, no. 1, pp. 101-111, 1960.- [26] P.S. Bradley, K.P. Bennett, and A. Demiriz, "Constrained K-Means Clustering," Microsoft Research Technical Report MSR-TR-2000-65, May 2000.
- [27] T. Achterberg, "Constraint Integer Programming," PhD thesis, Technische Univ. Berlin, 2007.
- [28] J. Peng and Y. Xia, "A New Theoretical Framework for K-Means-Type Clustering,"
Foundations and Advances in Data Mining, W. Chu and T.Y. Lin, eds., pp. 79-95, Springer, 2005.- [29] K. Rose, "Deterministic Annealing for Clustering, Compression, Classification, Regression and Related Optimization Problems,"
Proc. IEEE, vol. 86, no. 11, pp. 2210-2239, Aug. 1998.- [30] B. Akteke-Ozturk, G.-W. Weber, and E. Kropat, "Continuous Optimization Approaches for Clustering via Minimum Sum of Squares,"
Proc. 20th Mini-EURO Conf. Continuous Optimization and Knowledge-Based Technologies, May 2007.- [31] E.B. Baum, "Toward Practical 'Neural' Computation for Combinatorial Optimization Problems,"
Neural Networks for Computing, J. Denker, ed., Am. Inst. of Physics, 1986.- [32] G. Karypis and V. Kumar, "Multi-Level k-Way Hyper-Graph Partitioning,"
VLSI Design, vol. 11, no. 3, pp. 285-300, 2000.- [33] I.T. Christou and R.R. Meyer, "Decomposition Algorithms for Communication Minimization in Parallel Computing,"
Non-Linear Assignment Problems Theory and Practice, P.M. Pardalos and L. Pitsoulis, eds., Kluwer Academic Publishers, 2000. |