2016 IEEE International Conference on Cloud Engineering (IC2E) (2016)
April 4, 2016 to April 8, 2016
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/IC2E.2016.31
While distributed graph processing engines have become popular for processing large graphs, these engines are typically configured with a static set of servers in the cluster. In other words, they lack the flexibility to scale-out or scale-in the number of servers, when requested to do so by the user. In this paper, we propose the first techniques to make distributed graph processing truly elastic. While supporting on-demand scale-out/in operations, we meet three goals: i) perform scale-out/in without interrupting the graph computation, ii) minimize the background network overhead involved in the scale-out/in, and iii) mitigate stragglers by maintaining load balance across servers. We present and analyze two techniques called Contiguous Vertex Repartitioning (CVR) and Ring-based Vertex Repartitioning (RVR) to address these goals. We implement our techniques in the LFGraph distributed graph processing system, and incorporate several systems optimizations. Experiments performed with multiple graph benchmark applications on a real graph indicate that our techniques perform within 9% and 21% of the optimum for scale-out and scale-in operations, respectively.
Servers, Engines, Elasticity, Twitter, Clustering algorithms, Partitioning algorithms, Cloud computing
M. Pundir, M. Kumar, L. M. Leslie, I. Gupta and R. H. Campbell, "Supporting On-demand Elasticity in Distributed Graph Processing," 2016 IEEE International Conference on Cloud Engineering (IC2E), Berlin, Germany, 2016, pp. 12-21.