The Community for Technology Leaders
2014 Sixth International Symposium on Parallel Architectures, Algorithms and Programming (PAAP) (2014)
Beijing, China
July 13, 2014 to July 15, 2014
ISSN: 2168-3034
ISBN: 978-1-4799-3844-5
pp: 268-273
MapReduce is a distributed parallel computing framework for large-scale data processing with extensive applications. Hadoop MapReduce is the most widely employed open-source implementation of MapReduce framework for its flexible customization and simple usage. To avoid the relatively slow running task, called a straggler task, slowing down the job, MapReduce speculatively backups the straggler task on another node to execute aiming to reduce the job's finish time. Although there have been many speculative execution strate-gies in heterogeneous environments, they all do not consider the impact of dynamic system load on the running time of tasks. They may make mistakes in determining stragglers. In our paper, we propose a novel speculative execution strategy in heterogeneous environments, ERUL, to im-prove the estimation of tasks' rest time. ERUL also overcomes some drawbacks of LATE that mislead the speculative execution in some cases. The experimental result indicates that, our Hadoop-ERUL strategy not only works more accurately in the estimation of running tasks' remaining execution time, but also reduces 26% job's running time compared to Hadoop-LATE.
Load modeling, Estimation, Distributed databases, Data models, Heuristic algorithms, Educational institutions, Open source software,dynamic loading, hadoop, cloud computing, MapReduce, speculative execution
Huicheng Wu, Kenli Li, Zhuo Tang, Longxin Zhang, "A Heuristic Speculative Execution Strategy in Heterogeneous Distributed Environments", 2014 Sixth International Symposium on Parallel Architectures, Algorithms and Programming (PAAP), vol. 00, no. , pp. 268-273, 2014, doi:10.1109/PAAP.2014.29
251 ms
(Ver 3.3 (11022016))