2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum (2012)
Shanghai, China China
May 21, 2012 to May 25, 2012
The previous works about MapReduce task scheduling with deadline constraints neither take the diffenences of Map and Reduce task, nor the cluster's heterogeneity into account. This paper proposes an extensional MapReduce Task Scheduling algorithm for Deadline constraints in Hadoop platform: MTSD. It allows user specify a job's deadline and tries to make the job be finished before the deadline. Through measuring the node's computing capacity, a node classification algorithm is proposed in MTSD. This algorithm classifies the nodes into several levels in heterogeneous clusters. Under this algorithm, we firstly illuminate a novel data distribution model which distributes data according to the node's capacity level respectively. The experiments show that the data locality is improved about 57%. Secondly, we calculate the task's average completion time which is based on the node level. It improves the precision of task's remaining time evaluation. Finally, MTSD provides a mechanism to decide which job's task should be scheduled by calculating the Map and Reduce task slot requirements.
Classification algorithms, Scheduling algorithms, Clustering algorithms, Computational modeling, Data models, Scheduling, Data processing, Hadoop, MapReduce, scheduling algorithm, data locality, deadline constraints
Z. Tang, J. Zhou, K. Li and R. Li, "MTSD: A Task Scheduling Algorithm for MapReduce Base on Deadline Constraints," 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum(IPDPSW), Shanghai, China China, 2012, pp. 2012-2018.