This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
2012 IEEE Fifth International Conference on Cloud Computing
Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic
Honolulu, HI, USA USA
June 24-June 29
ISBN: 978-1-4673-2892-0
MapReduce is by far one of the most successful realizations of large-scale data-intensive cloud computing platforms. MapReduce automatically parallelizes computation by running multiple map and/or reduce tasks over distributed data across multiple machines. Hadoop is an open source implementation of MapReduce. When Hadoop schedules reduce tasks, it neither exploits data locality nor addresses partitioning skew present in some MapReduce applications. This might lead to increased cluster network traffic. In this paper we investigate the problems of data locality and partitioning skew in Hadoop. We propose Center-of-Gravity Reduce Scheduler (CoGRS), a locality-aware skew-aware reduce task scheduler for saving MapReduce network traffic. In an attempt to exploit data locality, CoGRS schedules each reduce task at its center-of-gravity node, which is computed after considering partitioning skew as well. We implemented CoGRS in Hadoop-0.20.2 and tested it on a private cloud as well as on Amazon EC2. As compared to native Hadoop, our results show that CoGRS minimizes off-rack network traffic by averages of 9.6% and 38.6% on our private cloud and on an Amazon EC2 cluster, respectively. This reflects on job execution times and provides an improvement of up to 23.8%.
Index Terms:
Schedules,Bandwidth,Cloud computing,Benchmark testing,Scheduling,Distributed databases,Network topology,Reduce Task Scheduling,Hadoop,MapReduce
Citation:
Mohammad Hammoud, M. Suhail Rehman, Majd F. Sakr, "Center-of-Gravity Reduce Task Scheduling to Lower MapReduce Network Traffic," cloud, pp.49-58, 2012 IEEE Fifth International Conference on Cloud Computing, 2012
Usage of this product signifies your acceptance of the Terms of Use.