High-Performance Distributed Computing, International Symposium on (2006)
June 19, 2006 to June 23, 2006
E. Yom-Tov , IBM Haifa Res. Lab.
Y. Aridor , IBM Haifa Res. Lab.
Heterogeneous clusters and grid infrastructures are becoming increasingly popular. In these computing infrastructures, machines have different resources (e.g., memory sizes, disk space, and installed software packages). These differences give rise to a problem of over-provisioning, that is, sub-optimal utilization of a cluster due to users requesting resource capacities greater than what their jobs actually need. Our analysis of a real workload file (LANL CM 5) revealed differences of up to two orders of magnitude between requested memory capacity and actual memory usage. The problem of over-provisioning has received very little attention so far. We discuss different approaches for applying machine learning methods to estimate the actual resource capacities used by jobs. These approaches are independent of the scheduling policies and the dynamic resource-matching schemes used. Our simulations show that these methods can yield an improvement of over 50% in utilization (throughput) of heterogeneous clusters
scheduling policy, dynamic resource-matching scheme, job requirements estimation, heterogeneous cluster sub-optimal utilization, grid infrastructure, over-provisioning problem, workload file analysis, machine learning method
Y. Aridor and E. Yom-Tov, "Improving Resource Matching Through Estimation of Actual Job Requirements," High-Performance Distributed Computing, International Symposium on(HPDC), Paris, 2006, pp. 367-368.