2015 IEEE International Conference on Autonomic Computing (ICAC) (2015)
July 7, 2015 to July 10, 2015
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/ICAC.2015.54
There is a trade-off between the number of concurrently running MapReduce jobs and their corresponding map and reduce tasks within a node in a Hadoop cluster. Leaving this trade-off statically configured to a single value can significantly reduce job response times leaving only sub optimal resource usage. To overcome this problem, we propose a feedback control loop based approach that dynamically adjusts the Hadoop resource manager configuration based on the current state of the cluster. The preliminary assessment based on workloads synthesized from real-world traces shows that the system performance can be improved by about 30% compared to default Hadoop setup.
Time factors, Random access memory, Yarn, Feedback control, Analytical models, Containers, Memory management
B. Zhang, F. Krikava, R. Rouvoy and L. Seinturier, "Self-Configuration of the Number of Concurrently Running MapReduce Jobs in a Hadoop Cluster," 2015 IEEE International Conference on Autonomic Computing (ICAC), Grenoble, France, 2015, pp. 149-150.