The Community for Technology Leaders
Green Image
Issue No. 02 - April-June (2014 vol. 7)
ISSN: 1939-1374
pp: 307-321
Zujie Ren , School of Computer, Science and Technology, Hangzhou Dianzi University, China
Jian Wan , School of Computer, Science and Technology, Hangzhou Dianzi University, China
Weisong Shi , Department of Computer Science, Wayne State, University, Detroit , USA
Xianghua Xu , School of Computer, Science and Technology, Hangzhou Dianzi University, China,
Min Zhou , Taobao, Inc. Hangzhou, Zhejiang Province, China
Understanding the characteristics of MapReduce workloads in a Hadoop cluster is the key to making optimal configuration decisions and improving the system efficiency and throughput. However, workload analysis on a Hadoop cluster, particularly in a large-scale e-commerce production environment, has not been well studied yet. In this paper, we performed a comprehensive workload analysis using the trace collected from a 2000-node Hadoop cluster at Taobao, which is the biggest online e-commerce enterprise in Asia, ranked 10th in the world as reported by Alexa. The results of the workload analysis are representative and generally consistent with the data warehouses for e-commerce web sites, which can help researchers and engineers understand the workload characteristics of Hadoop in their production environments. Based on the observations and implications derived from the trace, we designed a workload generator Ankus, to expedite the performance evaluation and debugging of new mechanisms. Ankus supports synthesizing an e-commerce style MapReduce workload at a low cost. Furthermore, we proposed and implemented a job scheduling algorithm, Fair4S , which is designed to be biased towards small jobs. Small jobs account for the majority of the workload, and most of them require instant and interactive responses, which is an important phenomenon at production Hadoop systems. The inefficiency of Hadoop fair scheduler for handling small jobs motivates us to design the Fair4S, which introduces pool weights and extends job priorities to guarantee the rapid responses for small jobs. Experimental evaluation verified that the Fair4S accelerates the average waiting times of small jobs by a factor of 7 compared with the fair scheduler.
Production, Log-normal distribution, Generators, Resource management, Standards, Distribution functions,job scheduler, Hadoop, MapReduce, workload analysis, workload synthesis
Zujie Ren, Jian Wan, Weisong Shi, Xianghua Xu, Min Zhou, "Workload Analysis, Implications, and Optimization on a Production Hadoop Cluster: A Case Study on Taobao", IEEE Transactions on Services Computing, vol. 7, no. , pp. 307-321, April-June 2014, doi:10.1109/TSC.2013.40
210 ms
(Ver )