The Community for Technology Leaders
2014 23rd International Conference on Parallel Architecture and Compilation (PACT) (2014)
Edmonton, Canada
Aug. 23, 2014 to Aug. 27, 2014
ISBN: 978-1-5090-6607-0
pp: 517-518
Harshvardhan , Parasol Laboratory, Department of Computer Science and Engineering, Texas A&M University
Nancy M. Amato , Parasol Laboratory, Department of Computer Science and Engineering, Texas A&M University
Lawrence Rauchwerger , Parasol Laboratory, Department of Computer Science and Engineering, Texas A&M University
ABSTRACT
With the advent of big-data, processing large graphs quickly has become increasingly important. Most existing approaches either utilize in-memory processing techniques, which can only process graphs that fit completely in RAM, or disk-based techniques that sacrifice performance. Contribution. In this work, we propose a novel RAM-Disk hybrid approach to graph processing that can scale well from a single shared-memory node to large distributed-memory systems. It works by partitioning the graph into subgraphs that fit in RAM and uses a paging-like technique to load subgraphs. We show that without modifying the algorithms, this approach can scale from small memory-constrained systems (such as tablets) to large-scale distributed machines with 16, 000+ cores.
INDEX TERMS
Random access memory, Libraries, Parallel processing, Twitter, Software algorithms, Programming, Optimization,Distributed Computing, Parallel Graph Processing, Out-of-Core Graph Algorithms, Graph Analytics, Big Data
CITATION
Harshvardhan, Nancy M. Amato, Lawrence Rauchwerger, "Processing big data graphs on memory-restricted systems", 2014 23rd International Conference on Parallel Architecture and Compilation (PACT), vol. 00, no. , pp. 517-518, 2014, doi:10.1145/2628071.2671429
98 ms
(Ver 3.3 (11022016))