2010 IEEE Second International Conference on Cloud Computing Technology and Science (2010)
Indianapolis, Indiana USA
Nov. 30, 2010 to Dec. 3, 2010
This paper investigates the problem of Partitioning Skew in MapReduce-based system. Our studies with Hadoop, a widely used MapReduce implementation, demonstrate that the presence of partitioning skew causes a huge amount of data transfer during the shuffle phase and leads to significant unfairness on the reduce input among different data nodes. As a result, the applications experience performance degradation due to the long data transfer during the shuffle phase along with the computation skew, particularly in reduce phase. We develop a novel algorithm named LEEN for locality-aware and fairness-aware key partitioning in MapReduce. LEEN embraces an asynchronous map and reduce scheme. All buffered intermediate keys are partitioned according to their frequencies and the fairness of the expected data distribution after the shuffle phase. We have integrated LEEN into Hadoop-0.18.0. Our experiments demonstrate that LEEN can efficiently achieve higher locality and reduce the amount of shuffled data. More importantly, LEEN guarantees fair distribution of the reduce inputs. As a result, LEEN achieves a performance improvement of up to 40% on different workloads.
MapReduce, Hadoop, partationing skew, Cloud Computing
S. Wu, B. He, L. Lu, H. Jin, S. Ibrahim and L. Qi, "LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud," 2010 IEEE Second International Conference on Cloud Computing Technology and Science(CLOUDCOM), Indianapolis, Indiana USA, 2010, pp. 17-24.