Issue No. 02 - April (2017 vol. 25)
Qiaomin Xie , Electrical and Computer Engineering Department, University of Illinois at Urbana–Champaign, Urbana, IL, USA
Mayank Pundir , Computer Science Department, University of Illinois at Urbana–Champaign, Urbana, IL, USA
Yi Lu , Electrical and Computer Engineering Department, University of Illinois at Urbana–Champaign, Urbana, IL, USA
Cristina L. Abad , Computer Science Department, Escuela Superior Politécnica del Litoral, Guayaquil, Ecuador
Roy H. Campbell , Computer Science Department, University of Illinois at Urbana–Champaign, Urbana, IL, USA
Data locality is a fundamental problem to data-parallel applications where data-processing tasks consume different amounts of time and resources at different locations. The problem is especially prominent under stressed conditions such as hot spots. While replication based on data popularity relieves hot spots due to contention for a single file, hot spots caused by skewed
node popularity, due to contention for files co-located with each other, are more complex, unpredictable, hence more difficult to deal with. We propose Pandas, a light-weight acceleration engine for data-processing tasks that is robust to changes in load and skewness in node popularity. Pandas is a stochastic delay-optimal algorithm. Trace-driven experiments on Hadoop show that Pandas accelerates the data-processing phase of jobs by 11 times with hot spots and 2.4 times without hot spots over existing schedulers. When the difference in processing times due to location is large, such as applicable to the case of memory-locality, the acceleration by Pandas is 22 times.
Delays, Peer-to-peer computing, Hafnium, Robustness, Stochastic processes, Acceleration, Scheduling
Q. Xie, M. Pundir, Y. Lu, C. L. Abad and R. H. Campbell, "Pandas: Robust Locality-Aware Scheduling With Stochastic Delay Optimality," in IEEE/ACM Transactions on Networking, vol. 25, no. 2, pp. 662-675, 2017.