2013 IEEE 5th International Conference on Cloud Computing Technology and Science (2013)

Bristol, United Kingdom United Kingdom

Dec. 2, 2013 to Dec. 5, 2013

pp: 17-24

ABSTRACT

The tremendous growth in data volumes has created a need for new tools and algorithms to quickly analyze large datasets. Cluster analysis techniques, such as K-means can be used for large datasets distributed across several machines. The accuracy of K-means depends on the selection of seed centroids during initialization. K-means++ improves on the K-means seeder, but suffers from problems when it is applied to large datasets: (a) the random algorithm it employs can produce inconsistent results across several analysis runs under the same initial conditions, and (b) it scales poorly for large datasets. In this paper we describe a new Competitive K-means algorithm we developed that addresses both of these problems. We describe an efficient MapReduce implementation of our new Competitive K-means algorithm that we found scales well with large datasets. We compared the performance of our new algorithm with three existing cluster analysis algorithms and found that our new algorithm improves cluster analysis accuracy and decreases variance. Our results show that our new algorithm produced a speedup of 76 &177; 9 times compared with the serial K-means++ and is as fast as the Streaming K-means. Our work provides a method to select a good initial seeding in less time, facilitating accurate cluster analysis over large datasets in shorter time.

INDEX TERMS

Clustering algorithms, Algorithm design and analysis, Partitioning algorithms, Integrated circuits, Google, Computers, Approximation algorithms,MapReduce, K-means, K-means++, Streaming K-means

CITATION

Rui Maximo Esteves,
Thomas Hacker,
Chunming Rong,
"Competitive K-Means, a New Accurate and Distributed K-Means Algorithm for Large Datasets",

*2013 IEEE 5th International Conference on Cloud Computing Technology and Science*, vol. 01, no. , pp. 17-24, 2013, doi:10.1109/CloudCom.2013.89