The Community for Technology Leaders
RSS Icon
Subscribe
Issue No.08 - August (2011 vol.22)
pp: 1365-1373
Jinoh Kim , University of Minnesota, Minneapolis
Abhishek Chandra , University of Minnesota, Minneapolis
Jon B. Weissman , University of Minnesota, Minneapolis
ABSTRACT
Distributed computing applications are increasingly utilizing distributed data sources. However, the unpredictable cost of data access in large-scale computing infrastructures can lead to severe performance bottlenecks. Providing predictability in data access is, thus, essential to accommodate the large set of newly emerging large-scale, data-intensive computing applications. In this regard, accurate estimation of network performance is crucial to meeting the performance goals of such applications. Passive estimation based on past measurements is attractive for its relatively small overhead compared to relying on explicit probing. In this paper, we take a passive approach for network performance estimation. Our approach is different from existing passive techniques that rely either on past direct measurements of pairs of nodes or on topological similarities. Instead, we exploit secondhand measurements collected by other nodes without any topological restrictions. In this paper, we present Overlay Passive Estimation of Network performance (OPEN), a scalable framework providing end-to-end network performance estimation based on secondhand measurements, and discuss how OPEN achieves cost-effective estimation in a large-scale infrastructure. Our extensive experimental results show that OPEN estimation can be applicable for replica and resource selections commonly used in distributed computing.
INDEX TERMS
Network performance estimation, secondhand estimation, data-intensive computing, replica selection, resource selection.
CITATION
Jinoh Kim, Abhishek Chandra, Jon B. Weissman, "Passive Network Performance Estimation for Large-Scale, Data-Intensive Computing", IEEE Transactions on Parallel & Distributed Systems, vol.22, no. 8, pp. 1365-1373, August 2011, doi:10.1109/TPDS.2010.201
REFERENCES
[1] "LHC: Large Hadron Collider," http://lhc.web.cern.chlhc/, 2011.
[2] "Sloan Digital Sky Survey/Skyserver," http://cas.sdss.org/dr7en/, 2011.
[3] S. Bharathi, A. Chervenak, E. Deelman, G. Mehta, and M.-H. Su, "Characterization of Scientific Workflows," Proc. Third Workshop Workflows in Support of Large-Scale Science (WORKS '08), 2008.
[4] E. Deelman and A. Chervenak, "Data Management Challenges of Data-Intensive Scientific Workflows," Proc. Eighth IEEE Int'l Symp. Cluster Computing and the Grid (CCGRID '08), pp. 687-692, 2008.
[5] D. Thain, T. Tannenbaum, and M. Livny, "Distributed Computing in Practice: The Condor Experience," Concurrency - Practice and Experience, vol. 17, nos. 2-4, pp. 323-356, 2005.
[6] "The Globus Alliance," http:/www.globus.org/, 2011.
[7] D. Kondo, A.A. Chien, and H. Casanova, "Resource Management for Rapid Application Turnaround on Enterprise Desktop Grids," Proc. ACM/IEEE Conf. Supercomputing (SC '04), 2004.
[8] V. Lo, D. Zappala, D. Zhou, Y. Liu, and S. Zhao, "Cluster Computing on the Fly: P2p Scheduling of Idle Cycles in the Internet," Proc. IEEE Fourth Int'l Conf. Peer-to-Peer Systems, pp. 227-236, 2004.
[9] D.P. Anderson and G. Fedak, "The Computational and Storage Potential of Volunteer Computing," Proc. Sixth IEEE Int'l Symp. Cluster Computing and the Grid (CCGRID '06), pp. 73-80, 2006.
[10] T.S.E. Ng, Y. hua Chu, S.G. Rao, K. Sripanidkulchai, and H. Zhang, "Measurement-Based Optimization Techniques for Bandwidth-Demanding Peer-to-Peer Systems," Proc. IEEE INFOCOM, pp. 2199-2209, 2003.
[11] S. Seshan, M. Stemm, and R.H. Katz, "SPAND: Shared Passive Network Performance Discovery," Proc. USENIX Symp. Internet Technologies and Systems, pp. 135-146, Dec. 1997.
[12] M. Andrews, B. Shepherd, A. Srinivasan, P. Winkler, and F. Zane, "Clustering and Server Selection Using Passive Monitoring," Proc. IEEE INFOCOM, pp. 1717-1725, 2002.
[13] J. Kim, A. Chandra, and J.B. Weissman, "Using Data Accessibility for Resource Selection in Large-Scale Distributed Systems," IEEE Trans. Parallel and Distributed Systems, vol. 20, no. 6, pp. 788-801, June 2009.
[14] "PlanetLab," http:/www.planet-lab.org, 2011.
[15] N. Kourtellis, L. Prieto, A. Iamnitchi, G. Zarrate, and D. Fraser, "Data Transfers in the Grid: Workload Analysis of Globus Gridftp," Proc. Int'l Workshop Data-Aware Distributed Computing (DADC '08), pp. 29-38, 2008.
[16] P. Yalagandula, P. Sharma, S. Banerjee, S. Basu, and S.-J. Lee, "S3: A Scalable Sensing Service for Monitoring Large Networked Systems," Proc. SIGCOMM Workshop Internet Network Management (INM '06), pp. 71-76, 2006.
[17] G.B. Berriman, A.C. Laity, J.C. Good, J.C. Jacob, D.S. Katz, E. Deelman, G. Singh, M.-H. Su, and T.A. Prince, "Montage: The Architecture and Scientific Applications of a National Virtual Observatory Service for Computing Astronomical Image Mosaics," Proc. Earth Sciences Technology Conf., 2006.
[18] A.B. Downey, "Using Pathchar to Estimate Internet Link Characteristics," Proc. ACM SIGCOMM '99, pp. 241-250, 1999.
[19] R. Wolski, N. Spring, and J. Hayes, "The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing," J. Future Generation Computing Systems, vol. 15, pp. 757-768, 1999.
[20] S. Keshav, "Packet-Pair Flow Control," IEEE/ACM Trans. Networking, 1995.
[21] Q. He, C. Dovrolis, and M. Ammar, "On the Predictability of Large Transfer Tcp Throughput," Proc. ACM SIGCOMM '05, pp. 145-156, 2005.
[22] R.L. Carter and M. Crovella, "Server Selection Using Dynamic Path Characterization in Wide-Area Networks," Proc. IEEE INFOCOM, pp. 1014-1021, 1997.
[23] A.-M. Kermarrec, L. Massoulié, and A.J. Ganesh, "Probabilistic Reliable Dissemination in Large-Scale Systems," IEEE Trans. Parallel and Distributed Systems, vol. 14, no. 3, pp. 248-258, Mar. 2003.
[24] S. Voulgaris and M. van Steen, "Hybrid Dissemination: Adding Determinism to Probabilistic Multicasting in Large-Scale p2p Systems," Proc. ACM/IFIP/USENIX Int'l Conf. Middleware, pp. 389-409, 2007.
[25] M. Deshpande, B. Xing, I. Lazardis, B. Hore, N. Venkatasubramanian, and S. Mehrotra, "Crew: A Gossip-Based Flash-Dissemination System," Proc. 26th IEEE Int'l Conf. Distributed Computing Systems (ICDCS '06), p. 45, 2006.
[26] Z.J. Haas, J.Y. Halpern, and L. Li, "Gossip-Based Ad Hoc Routing," IEEE/ACM Trans. Networking, vol. 14, no. 3, pp. 479-491, June 2006.
[27] P. Kyasanur, R. Choudhury, and I. Gupta, "Smart Gossip: An Adaptive Gossip-Based Broadcasting Service for Sensor Networks," Proc. IEEE Int'l Conf. Mobile Adhoc and Sensor Systems, pp. 91-100, 2006.
[28] J. Strauss, D. Katabi, and F. Kaashoek, "A Measurement Study of Available Bandwidth Estimation Tools," Proc. Third ACM SIGCOMM Conf. Internet Measurement (IMC '03), pp. 39-44, 2003.
[29] R. Wolski, "Experiences with Predicting Resource Performance Online in Computational Grid Settings," SIGMETRICS Performance Evaluation Rev., vol. 30, no. 4, pp. 41-49, 2003.
[30] H.V. Madhyastha, T. Isdal, M. Piatek, C. Dixon, T. Anderson, A. Krishnamurthy, and A. Venkataramani, "iPlane: An Information Plane for Distributed Services," Proc. Seventh USENIX Symp. Operating Systems Design and Implementation (OSDI '06), 2006.
[31] H.V. Madhyastha, E. Katz-Bassett, T. Anderson, A. Krishnamurthy, and A. Venkataramani, "Iplane Nano: Path Prediction for Peer-to-Peer Applications," Proc. Sixth USENIX Symp. Networked Systems Design and Implementation (NSDI '09), pp. 137-152, 2009.
[32] I. Gupta, A.-M. Kermarrec, and A.J. Ganesh, "Efficient and Adaptive Epidemic-Style Protocols for Reliable and Scalable Multicast," IEEE Trans. Parallel and Distributed Systems, vol. 17, no. 7, pp. 593-605, July 2006.
[33] M. jang Lin and K. Marzullo, "Directional Gossip: Gossip in a Wide Area Network," Proc. European Dependable Computing Conf., pp. 364-379, 1999.
[34] "perfsonar," http:/www.perfSONAR.net/, 2011.
7 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool