The Community for Technology Leaders
Green Image
Issue No. 01 - Jan.-March (2017 vol. 5)
ISSN: 2168-7161
pp: 43-56
Mario Pastorelli , Teralytics, AG, Switzerland
Damiano Carra , Department of Computer Science, University of Verona, Verona, Italy
Matteo DellAmico , Symantec Research Labs, France
Pietro Michiardi , Department of Networking and Security, EURECOM, France
ABSTRACT
Size-based scheduling with aging has been recognized as an effective approach to guarantee fairness and near-optimal system response times. We present HFSP, a scheduler introducing this technique to a real, multi-server, complex, and widely used system such as Hadoop. Size-based scheduling requires a priori job size information, which is not available in Hadoop: HFSP builds such knowledge by estimating it on-line during job execution. Our experiments, which are based on realistic workloads generated via a standard benchmarking suite, pinpoint at a significant decrease in system response times with respect to the widely used Hadoop Fair scheduler, without impacting the fairness of the scheduler, and show that HFSP is largely tolerant to job size estimation errors.
INDEX TERMS
Training, Aging, Estimation, Time factors, Processor scheduling, Silicon, Cloud computing
CITATION

M. Pastorelli, D. Carra, M. DellAmico and P. Michiardi, "HFSP: Bringing Size-Based Scheduling To Hadoop," in IEEE Transactions on Cloud Computing, vol. 5, no. 1, pp. 43-56, 2017.
doi:10.1109/TCC.2015.2396056
237 ms
(Ver 3.3 (11022016))