The Community for Technology Leaders
2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA) (2017)
Taipei, Taiwan
March 27, 2017 to March 29, 2017
ISSN: 1550-445X
ISBN: 978-1-5090-6029-0
pp: 1090-1097
ABSTRACT
MapReduce model was designed for distributed large volume of data processing. Time constraints are very important for user productivity but, in a shared cluster, the need for job starvation avoidance also arises. In this paper we propose an extension to the well-known FairScheduler algorithm from Hadoop which takes into consideration soft deadlines for jobs in homogeneous clusters, aiming to improve productivity by better satisfying the user time needs. Our model is as follows: when a job is launched, a deadline is provided. When the job starts running, it has a default priority assigned by which FairScheduler splits resources. The job gets allocated resources and at a certain moment in time it has a current execution speed. Based on speed, it is computed how many more resources the job needs to finish in time. Given this, the job priority is dynamically adjusted so the FairScheduler's resource division policy can meet the deadlines. We validated our algorithm by execution in a real Hadoop environment and we obtained increased performance under deadline constraints while also avoiding job starvation.
INDEX TERMS
Clustering algorithms, Algorithm design and analysis, Processor scheduling, Job shop scheduling, Delays, Dynamic scheduling, Heuristic algorithms
CITATION

A. Farcasanu, F. Pop, M. Nita and C. Dobre, "Starvation Avoidance with Deadline Constraints in Hadoop Environments," 2017 IEEE 31st International Conference on Advanced Information Networking and Applications (AINA), Taipei, Taiwan, 2017, pp. 1090-1097.
doi:10.1109/AINA.2017.134
94 ms
(Ver 3.3 (11022016))