The Community for Technology Leaders
RSS Icon
Issue No.11 - Nov. (2013 vol.62)
pp: 2210-2223
Yang Wang , University of New Brunswick, Fredericton
Paul Lu , University of Alberta, Edmonton
Workflow-based workloads usually consist of multiple instances of the same workflow, which are jobs with control or data dependencies to carry out a well-defined scientific computation task, with each instance acting on its own input data. To maximize the performance, a high degree of concurrency is always achieved by running multiple instances simultaneously. However, since the amount of storage is limited on most systems, deadlock due to oversubscribed storage requests is a potential problem. To address this problem, we integrate two novel concepts with the traditional problem of deadlock avoidance by proposing two algorithms that can maximize active (not just allocated) resource utilization and minimize makespan. Our approach is based on the well-known banker's algorithm, but our algorithms make the important distinction between active and inactive resources, which is not a part of previous approaches. The central idea is to leverage the data-flow information to dynamically approximate localized maximum claim (i.e., the resource requirements of the remaining jobs of the instance) to improve either interinstance or intrainstance concurrency and still avoid deadlock. Through simulation-based studies, we show how our proposed algorithms are better than the classic banker's algorithm and the more recent Lang's algorithm in terms of makespan and active storage resource utilization.
System recovery, Heuristic algorithms, Resource management, Approximation algorithms, Processor scheduling, Scheduling, Concurrent computing,storage constraints, Storage aware, workflow scheduling, deadlock, data dependency, data flow, active storage resource
Yang Wang, Paul Lu, "Maximizing Active Storage Resources with Deadlock Avoidance in Workflow-Based Computations", IEEE Transactions on Computers, vol.62, no. 11, pp. 2210-2223, Nov. 2013, doi:10.1109/TC.2012.217
[1] T. Werner, "Target Gene Identification from Expression Array Data by Promoter Analysis," Biomolecular Eng., vol. 17, pp. 87-94, 2001.
[2] D. Szafron, P. Lu, R. Greiner, D. Wishart, B. Poulin, R. Eisner, Z. Lu, J. Anvik, C. Macdonell, A. Fyshe, and D. Meeuwis, "Proteome Analyst: Custom Predictions with Explanations in a Web-Based Tool for High-Throughput Proteome Annotations," Nucleic Acids Research, vol. 32, pp. W365-W371, 2004.
[3] Gromacs, "Homepage," http:/, 2013.
[4] M. Schmidt, K. Baldridge, J. Boatz, S. Elbert, M. Gordon, J. Jensen, S. Koseki, N. Matsunaga, and J. Montgomery, "The General Atomic and Molecular Electronic Structure System," J. Computational Chemistry, vol. 14, pp. 1347-1363, 1993.
[5] B. Ludascher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger, M. Jones, E. Lee, J. Tao, and Y. Zhao, "Scientific Workflow Management and the Kepler System," Concurrency and Computation: Practice & Experience, Special Issue on Scientific Workflows, vol. 18, pp. 1039-1065, 2006.
[6] E. Deelman, D. Gannon, M. Shields, and I. Taylor, "Workflows and E-Science: An Overview of Workflow System Features and Capabilities," Future Generation Computer Systems, vol. 25, no. 5, pp. 528-540, May 2009.
[7] K. Srimanotham and V. Muangsin, "Scheduling Workflow-Based Parameter-Sweep Applications with Best-Intermediate-Result-First Heuristic," Proc. IEEE Int'l Conf. Cluster Computing, pp. 1-6, 2006.
[8] S. Smanchat, M. Indrawan, S. Ling, C. Enticott, and D. Abramson, "Scheduling Multiple Parameter Sweep Workflow Instances on the Grid," Proc. IEEE Fifth Int'l Conf. e-Science, pp. 300-306, 2009.
[9] D. Abramson, B. Bethwaite, C. Enticott, S. Garic, and T. Peachey, "Parameter Exploration in Science and Engineering Using Many-Task Computing," IEEE Trans. Parallel and Distributed Systems, vol. 22, no. 6, pp. 960-973, June 2011.
[10] A. Mandal, K. Kennedy, C. Koelbel, B. Liu, and L. Johnsson, "Scheduling Strategies for Mapping Application Workflows onto the Grid," Proc. IEEE 14th Int'l Symp. High Performance Distributed Computing (HPDC '05), pp. 125-134, July 2005.
[11] Y.-K. Kwok and I. Ahmad, "Static Scheduling Algorithms for Allocating Directed Task Graphs to Multiprocessors," ACM Computing Survey, vol. 31, no. 4, pp. 406-471, Sept. 1999.
[12] M. Siddiqui, A. Villazon, and T. Fahringer, "Grid Capacity Planning with Negotiation-Based Advance Reservation for Optimized QoS," Proc. ACM/IEEE SuperComputing Conf. (SC '06), pp. 21-36, Nov. 2006.
[13] M. Wieczorek, M. Siddiqui, A. Villazon, R. Prodan, and T. Fahringer, "Applying Advance Reservation to Increase Predictability of Workflow Execution on the Grid," Proc. IEEE Second Int'l Conf. E-Science and Grid Computing, p. 82, Dec. 2006.
[14] L. do Nascimento, R. Ferreira, W.M. Jr, and D. Guedes, "Scheduling Data Flow Applications Using Linear Programming," Proc. Int'l Conf. Parallel Processing (ICPP '05), pp. 638-645, June 2005.
[15] A. Giersch, Y. Robert, and F. Vivien, "Scheduling Tasks Sharing Files on Heterogeneous Master-Slave Platforms," Proc. 12th Euromicro Conf. Parallel, Distributed and Network-Based Processing, pp. 364-371, Feb. 2004.
[16] E. Bertin, "Sextractor - Astronomical Source Extractor," http:/, 2011.
[17] S. Djorgovski, R. Gal, S. Odewahn, R. de Carvalho, R. Brunner, G. Longo, and R. Scaramella, "The Digital Palomar Sky Survey (DPOSS)," Wide Field Surveys in Cosmology, 1998.
[18] B. Barish and R. Weiss, "LIGO and the Detection of Gravitational Waves," Physics Today, vol. 52, p. 44, 1999.
[19] C. Vecchiola, S. Pandey, and R. Buyya, "High-Performance Cloud Computing: A View of Scientific Applications," Proc. 10th Int'l Symp. Pervasive Systems, Algorithms and Networks, Dec. 2009.
[20] L. Ramakrishnan, P.T. Zbiegel, S. Campbell, R. Bradshaw, R.S. Canon, S. Coghlan, I. Sakrejda, N. Desai, T. Declerck, and A. Liu, "Magellan: Experiences from a Science Cloud," Proc. Second Int'l Workshop Scientific Cloud Computing (ScienceCloud '11), pp. 49-58, 2011.
[21] L. Wang, J. Tao, M. Kunze, A. Castellanos, D. Kramer, and W. Karl, "Scientific Cloud Computing: Early Definition and Experience," Proc. IEEE 10th Int'l Conf. High Performance Computing and Comm., Sept. 2008.
[22] J. Gray, D. Liu, M. Nieto-Santisteban, A.S. Szalay, D. DeWitt, and G. Heber, "Scientific Data Management in the Coming Decade," Technical Report MSR-TR-2005-10, Microsoft Corporation, 2005.
[23] S. Pandey and R. Buyya, "Scheduling of Scientific Workflows on Data Grids," Proc. IEEE Eighth Int'l Symp. Cluster Computing and Grid, pp. 548-553, 2008.
[24] A. Ramakrishnan, G. Singh, H. Zhao, E. Deelman, R. Sakellariou, K. Vahi, K. Blackburn, D. Mayers, and M. Samidi, "Scheduling Data-Intensive Workflows onto Storage-Constrained Distributed Resources," Proc. IEEE Seventh Int'l Symp. Cluster Computing and the Grid, pp. 401-409, 2007.
[25] A. Tanenbaum, Modern Operating Systems. Prentice Hall, 2001.
[26] S.-D. Lang, "An Extended Banker's Algorithm for Deadlock Avoidance," IEEE Trans. Software Eng., vol. 25, no. 3, pp. 428-432, May/June 1999.
[27] J. Bent, D. Thain, A. Arpaci-Dusseau, R.H. Arpaci-Dusseau, and M. Livny, "Explicit Control in a Batch-Aware Distributed File System," Proc. First Conf. Symp. Networked Systems Design and Implementation (NSDI '04), pp. 365-378, 2004.
[28] W. Chen and E. Deelman, "Integration of Workflow Partitioning and Resource Provisioning," Proc. IEEE/ACM 12th Int'l Symp. Cluster, Cloud and Grid Computing, pp. 764-768, 2012.
[29] W. Zhang, J. Cao, Y. Zhong, L. Liu, and C. Wu, "An Integrated Resource Management and Scheduling System for Grid Data Streaming Applications," Proc. IEEE/ACM Ninth Int'l Conf. Grid Computing, pp. 258-265, 2008.
[30] W. Zhang, J. Cao, Y. Zhong, L. Liu, and C. Wu, "Block-Based Concurrent and Storage-Aware Data Streaming for Grid Applications with Lots of Small Files," Proc. IEEE/ACM Ninth Int'l Symp. Cluster Computing and Grid (GRID '08), pp. 538-543, 2009.
[31] J. Bent, "Data-Driven Batch Scheduling," PhD thesis, Univ. of Wisconsin-Madison, 2005.
[32] A. Shoshani and E. Coffman, "Sequencing Tasks in Multi-Process, Multiple Resource Systems to Avoid Deadlocks," Proc. 11th Ann. Symp. Switching and Automata Theory, pp. 225-233, Oct. 1970.
[33] S. Reveliotis and M.A. Lawley, "Efficient Implementations of Banker's Algorithm for Deadlock Avoidance in Flexible Manufacturing Systems," Proc. Sixth Int'l Conf. Emerging Technologies and Factory Automation, Mar. 1997.
[34] R. Sethi, "Complete Register Allocation Problem," SIAM J. Computing, vol. 3, no. 3, pp. 226-248, 1975.
[35] W. Dobosiewicz and P. Gburzynski, "SMURPH: An Object-Oriented Simulator for Communication Networks and Protocols," Proc. Int'l Workshop Modeling, Analysis, and Simulation on Computer and Telecomm. Systems (MASCOTS '93), pp. 351-352, 1993.
[36] P. Gburzynski SMURPH, PAPERSside.pdf , 2013.
[37] E. Coffman, Computer and Job-Shop Scheduling Theory. John Wiley & Sons, 1976.
[38] T. Glatard, J. Montagnat, and X. Pennec, "Grid-Enabled Workflows for Data Intensive Medical Applications," Proc. IEEE 18th Symp. Computer-Based Medical Systems, pp. 537-542, 2005.
[39] J. Wang, H. Kuehl, and M. Sacchi, "Least-Squares Wave-Equation AVP Imaging of 3D Common Azimuth Data," Proc. 73rd Ann. Int'l Meeting Soc. of Exploration Geophysicists, 2003.
[40] P. Blaha, K. Schwarz, G. Madsen, D. Kvasnicka, and J. Luitz, "WIEN2k: An Augmented Plane Wave Plus Local Orbitals Program for Calculating Crystal Properties," technical report, Inst. of Physical and Theoretical Chemistry, Vienna Univ. of Tech nology, 2001.
[41] R. Lake, J. Schaeffer, and P. Lu, "Solving Large Retrograde Analysis Problems Using a Network of Workstations," Advances in Computer Chess, vol. 7, pp. 135-162, 1994.
[42] J. Schaeffer and R. Lake, "Solving the Game of Checkers," Games of No Chance, R.J. Nowakowski, ed., vol. 20, Cambridge Univ. Press, 1996.
[43] A. Rosenberg, "On Scheduling Mesh-Structured Computations for Internet-Based Computing," IEEE Trans. Computers, vol. 53, no. 9, pp. 1176-1186, Sept. 2004.
[44] P. Crandall, R. Aydt, A. Chien, and D. Reed, "Input/Output Characteristics of Scalable Parallel Applications," Proc. IEEE/ACM Conf. Supercomputing, pp. 59-89, 1995.
[45] P. Hulith, "The AMANDA Experiment," Proc. 17th Int'l Conf. Neutrino Physics and Astrophysics, June 1996.
[46] A. Sum and J. de Pablo, "Nautilus: Molecular Simulation Code," technical report, Dept. of Chemical Eng., Univ. of Wisconsin-Madison, 2002.
[47] M. Aftosmis, M. Berger, M. Nemec, and J. Melton, "Cart3D v1.4," http://people. cart3Dhome.html, 2013.
[48] S. Venugopal and R. Buyya, "A Set Coverage-Based Mapping Heuristic for Scheduling Distributed Data-Intensive Applications on Global Grids," Proc. IEEE/ACM Seventh Int'l Conf. Grid Computing (GRID '06), 2006.
[49] Z. Yu and W. Shi, "An Adaptive Rescheduling Strategy for Grid Workflow Applications," Proc. IEEE Int'l Parallel & Distributed Processing Symp., pp. 214-220, Sept. 2007.
[50] J. Blythe, Y. Gil, and E. Deelman, "Coordinating Workflows in Shared Grid Environments," Proc. 14th Int'l Conf. Automated Planning and Scheduling, 2004.
[51] Y. Zhang, C. Koelbel, and K. Kennedy, "Relative Performance of Scheduling Algorithms in Grid Environment," Proc. IEEE Seventh Int'l Symp. Cluster Computing and Grid, 2007.
[52] Y. Wang and P. Lu, "DDS: A Deadlock Detection-Based Scheduling Algorithm for Workflow Computations in HPC Systems with Storage Constraints," Parallel Computing, vol. 39, no. 8, pp. 291-305, Aug. 2013.
[53] L. Devroye, Non-Uniform Random Variate Generation. Springer-Verlag, 1986.
[54] M. Lehmann, Data Access in Workflow Management Systems. IOS Press, 2006.
[55] Y. Wang and P. Lu, "Dataflow Detection and Applications to Workflow Scheduling," Concurrency and Computation: Practice and Experience, vol. 23, no. 11, pp. 1261-1283, 2011.
305 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool