The Community for Technology Leaders
2015 IEEE International Conference on Autonomic Computing (ICAC) (2015)
Grenoble, France
July 7, 2015 to July 10, 2015
ISBN: 978-1-4673-6970-1
pp: 149-150
ABSTRACT
There is a trade-off between the number of concurrently running MapReduce jobs and their corresponding map and reduce tasks within a node in a Hadoop cluster. Leaving this trade-off statically configured to a single value can significantly reduce job response times leaving only sub optimal resource usage. To overcome this problem, we propose a feedback control loop based approach that dynamically adjusts the Hadoop resource manager configuration based on the current state of the cluster. The preliminary assessment based on workloads synthesized from real-world traces shows that the system performance can be improved by about 30% compared to default Hadoop setup.
INDEX TERMS
Time factors, Random access memory, Yarn, Feedback control, Analytical models, Containers, Memory management
CITATION

B. Zhang, F. Krikava, R. Rouvoy and L. Seinturier, "Self-Configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster," 2015 IEEE International Conference on Autonomic Computing (ICAC), Grenoble, France, 2015, pp. 149-150.
doi:10.1109/ICAC.2015.54
80 ms
(Ver 3.3 (11022016))