The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.01 - January (2008 vol.19)
pp: 24-34
ABSTRACT
In many e-science applications, there exists an important need to aggregate information from data repositories distributed around the world. In an effort to better link these resources in a unified manner, many lambda-grid networks, which provide end-to-end dedicated optical circuit-switched connections, have been investigated. In this context, we consider the problem of aggregating files from distributed databases at a (grid) computing node over a lambda grid. The challenge is (i) to identify routes (i.e., circuits) in the lambda-grid network along which files should be transmitted and (ii) to schedule the transfers of these files over their respective circuits. To address this challenge, we propose a hybrid approach that combines off-line and on-line scheduling. We define the Time-Path Scheduling Problem (TPSP) for off-line scheduling. We prove that TPSP is NP-complete, develop a Mixed Integer Linear Program (MILP) formulation for TPSP, and then propose a greedy approach to solve TPSP because the MILP does not scale well. We compare the erformance of the greedy approach on a few representative lambda-grid network topologies. One key input to the off-line schedule is the file transfer time. Due to dynamics at the receiving end host which is hard to model precisely, the actual file transfer time may vary. We first propose a model for the estimating the file transfer time. Then, we propose on-line reconfiguration algorithms, so that, as files are transferred, the off-line schedule may be modified on-line depending on the amount of time that it actually took to transfer the file. This helps to reduce the total time to transfer all the files, which is an important metric. To demonstrate the effectiveness of our approach, we present results on an emulated lambda-grid network testbed.
INDEX TERMS
lambda grid, circuit switching, routing, scheduling, large scale data transfers
CITATION
Amitabha Banerjee, Wu-chun Feng, Dipak Ghosal, Biswanath Mukherjee, "Algorithms for Integrated Routing and Scheduling for Aggregating Data from Distributed Resources on a Lambda Grid", IEEE Transactions on Parallel & Distributed Systems, vol.19, no. 1, pp. 24-34, January 2008, doi:10.1109/TPDS.2007.1112
REFERENCES
[1] CANARIE CA*net4, http:/www.canarie.ca, 2007.
[2] DoE UltraScience Net Testbed, http://www.csm.ornl.govultranet/, 2007.
[3] “Genomes to Life Requires New Life from Networks,” US Dept. of Energy (DoE) Workshop, http://www.csm.ornl.gov/ghpngenome_wk2003.pdf , 2003.
[4] ILOG CPLEX version 9.0, http://www.ilog.com/products/cplex/product suite.cfm, 2007.
[5] Nat'l LambdaRail Inc., http:/www.nlr.net, 2007.
[6] Optiputer, http:/www.optiputer.net/, 2007.
[7] QUANTA 1.0, EVL, http:/www.evl.uic.edu, 2007.
[8] TeraGrid, http:/www.teragrid.org/, 2007.
[9] A. Banerjee et al., “A Time-Path Scheduling Problem (TPSP) for Aggregating Large Data Files from Distributed Databases Using an Optical Burst-Switched Network,” Proc. IEEE Int'l Conf. Comm. (ICC '04), June 2004.
[10] A. Banerjee, W. Feng, B. Mukherjee, and D. Ghosal, “RAPID: An End-System Aware Protocol for Intelligent Data Transfer over Lambda Grids,” Proc. 20th IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS '06), Apr. 2006.
[11] E. Coffman, M. Garey, D. Johnson, and S. Lapaugh, “Scheduling File Transfers,” SIAM J. Computing, vol. 14, no. 3, pp. 744-780, Aug. 1985.
[12] T. Cormen, C. Leiserson, R. Rivest, and C. Stein, Introduction to Algorithms, second ed. MIT Press, 2001.
[13] M. Garey and D. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness. W.H. Freeman, 1979.
[14] S. Gorinsky and N.S.V. Rao, “Dedicated Channels as an Optimal Network Support for Effective Transfer of Massive Data,” Proc. IEEE INFOCOM High-Speed Networking Workshop: The Terabits Challenge, 2006.
[15] Y. Gu, X. Hong, and R.L. Grossman, “Experiences in Design and Implementation of a High Performance Transport Protocol,” Proc. ACM/IEEE Conf. Supercomputing (SC '04),
[16] N. Hall and C. Sriskandarajah, “A Survey of Machine Scheduling Problems with Blocking and No-Wait in Process,” Operations Research, vol. 44, no. 3, pp. 510-525, May-June 1996.
[17] E. He, J. Leigh, O. Yu, and T.A. DeFanti, “Reliable Blast UDP: Predictable High Performance Bulk Data Transfer,” Proc. IEEE Int'l Conf. Cluster Computing (CLUSTER '02), 2002.
[18] B. Mukherjee, Optical Communication Networks. Springer, 2006.
[19] M. Pinedo, Scheduling: Theory, Algorithms, and Systems, second ed. Prentice Hall, 2002.
[20] N.S.V. Rao, W.R. Wing, S.M. Carter, and Q. Wu, “UltraScience Net: Network Testbed for Large-Scale Science Applications,” IEEE Comm. Magazine, vol. 43, no. 11, pp. S12-S17, Nov. 2005.
[21] N.S.V. Rao, W. Grimmell, Y. Bang, and S. Radhakrishnan, “On Algorithms for Quickest Paths under Different Routing Modes,” IEICE Trans. Comm., vol. E87-B, no. 4, 2004.
[22] N.S.V. Rao et al., “Control Plane for Advanced Bandwidth Scheduling in Ultra High-Speed Networks,” Proc. IEEE INFOCOM High-Speed Networking Workshop: The Terabits Challenge, Apr. 2006.
[23] L. Rizzo, “Dummynet: A Simple Approach to the Evaluation of Network Protocols,” ACM Computer Comm. Rev., vol. 27, no. 1, pp.31-41, 1997.
[24] S. Vazhkudai, J. Schopf, and I. Foster, “Predicting the Performance of Wide Area Data Transfers,” Proc. 16th Int'l Symp. Parallel and Distributed Processing (IPDPS '02), 2002.
[25] M. Veeraraghavan et al., “Scheduling and Transport for File Transfers on High-Speed Optical Circuits,” J. Grid Computing, vol. 1, no. 4, pp. 395-405, 2003.
[26] Y. Wang et al., “Checkpointing and Its Applications,” Proc. 25th IEEE Int'l Symp. Fault-Tolerant Computing (FTCS '95), 1995.
[27] R. Wu and A. Chien, “GTP: Group Transport Protocol for Lambda Grids,” Proc. Fourth IEEE/ACM Int'l Symp. Cluster Computing and the Grid (CCGrid '04), 2004.
18 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool