The Community for Technology Leaders
2015 Third International Conference on Advanced Cloud and Big Data (CBD) (2015)
Yangzhou, Jiangsu, China
Oct. 30, 2015 to Nov. 1, 2015
ISBN: 978-1-4673-8537-4
pp: 167-174
ABSTRACT
The emergence of large-scale graph data has posed essential challenges for processing them efficiently. The fundamental step for effectively processing the graph is to partition the graph and distribute the relevant parts on multiple workers for parallel computing. The existing partition strategies may suffer from the following problems: 1) They ignore the certain application features, making the partition not satisfy the application needs, which may cause performance degradation, 2) Because of the ignorance of the applications features, current partition strategies are not dynamic to meet the needs from different applications. In this paper, the Superblock partition strategy, an application-aware dynamic partition strategy for large-scale data is proposed to solve the above problems. It pre-partitions the graph into blocks and then extracts the application features and combines the blocks into Superblocks. The Superblock will be re-constructed when new application arrives as well. Experiments are performed using some common graph algorithms to confirm that the Superblock partition strategy can boost the performance of various data processing application on large-scale graph data and be dynamic enough to alter the partitions for different applications.
INDEX TERMS
Partitioning algorithms, Metadata, Data processing, Load modeling, Feature extraction, Distributed databases, Computational modeling,application-aware, graph computation, graph partition, Pregel
CITATION
Junxue Zhang, Fang Dong, Dian Shen, Jiahui Jin, Junzhou Luo, "Superblock: An Application-Aware Dynamic Partition Strategy for Large-Scale Graph", 2015 Third International Conference on Advanced Cloud and Big Data (CBD), vol. 00, no. , pp. 167-174, 2015, doi:10.1109/CBD.2015.35
87 ms
(Ver 3.3 (11022016))