The Community for Technology Leaders
Cluster Computing and the Grid, IEEE International Symposium on (2007)
Rio De Janeiro, Brazil
May 14, 2007 to May 17, 2007
ISBN: 0-7695-2833-3
pp: 401-409
David Meyers , Northrop Grumman Information Technology, Pasadena
Arun Ramakrishnan , University of Southern California, Los Angeles, USA
Kent Blackburn , California Institute of Technology
Henan Zhao , University of Manchester, Manchester M13 9PL, UK
Rizos Sakellariou , University of Manchester, Manchester M13 9PL, UK
Michael Samidi , California Institute of Technology
ABSTRACT
In this paper we examine the issue of optimizing disk usage and of scheduling large-scale scientific workflows onto distributed resources where the workflows are dataintensive, requiring large amounts of data storage, and where the resources have limited storage resources. Our approach is two-fold: we minimize the amount of space a workflow requires during execution by removing data files at runtime when they are no longer required and we schedule the workflows in a way that assures that the amount of data required and generated by the workflow fits onto the individual resources. For a workflow used by gravitationalwave physicists, we were able to improve the amount of storage required by the workflow by up to 57 %. We also designed an algorithm that can not only find feasible solutions for workflow task assignment to resources in diskspace constrained environments, but can also improve the overall workflow performance.
INDEX TERMS
null
CITATION
David Meyers, Arun Ramakrishnan, Kent Blackburn, Henan Zhao, Ewa Deelman, Gurmeet Singh, Rizos Sakellariou, Michael Samidi, Karan Vahi, "Scheduling Data-IntensiveWorkflows onto Storage-Constrained Distributed Resources", Cluster Computing and the Grid, IEEE International Symposium on, vol. 00, no. , pp. 401-409, 2007, doi:10.1109/CCGRID.2007.101
84 ms
(Ver )