The Community for Technology Leaders
Parallel Architectures, Algorithms and Programming, International Symposium on (2011)
Tianjin, China
Dec. 9, 2011 to Dec. 11, 2011
ISBN: 978-0-7695-4575-2
pp: 213-217
ABSTRACT
A shared MapReduce cluster is beneficial to build data warehouse which can be used by multiple users. FAIR scheduler gives each user the illusion of owning a private cluster. Moreover, it can dynamic redistribute capacity unused by some users to other users. However, when reassigning the slots, FAIR picks the most recently launched tasks to kill without considering the job character and data locality, which increases the network traffic while rescheduling the killed Map/Reduce tasks. The paper, based on FAIR scheduling, proposes an improved FAIR scheduling algorithm, which take into account the job character and data locality while killing tasks to make slots for new users. Performance evaluation results demonstrate that the improved FAIR decreases the data movement, speeds the execution of jobs, consequently improving the system performance.
INDEX TERMS
MapReduce, job Scheduling, HDFS, Hadoop
CITATION
Yongcai Tao, Qing Zhang, Lei Shi, Pinhua Chen, "Job Scheduling Optimization for Multi-user MapReduce Clusters", Parallel Architectures, Algorithms and Programming, International Symposium on, vol. 00, no. , pp. 213-217, 2011, doi:10.1109/PAAP.2011.33
163 ms
(Ver 3.3 (11022016))