This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management
August 2010 (vol. 21 no. 8)
pp. 1089-1102
ASCII Text
x 
 
Michael Cardosa, Abhishek Chandra, "Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management," IEEE Transactions on Parallel and Distributed Systems, vol. 21, no. 8, pp. 1089-1102, August, 2010.
 
 
BibTex
x
 
@article{ 10.1109/TPDS.2009.143,
author = {Michael Cardosa and Abhishek Chandra},
title = {Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management},
journal ={IEEE Transactions on Parallel and Distributed Systems},
volume = {21},
number = {8},
issn = {1045-9219},
year = {2010},
pages = {1089-1102},
doi = {http://doi.ieeecomputersociety.org/10.1109/TPDS.2009.143},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Parallel and Distributed Systems
TI - Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management
IS - 8
SN - 1045-9219
SP1089
EP1102
EPD - 1089-1102
A1 - Michael Cardosa,
A1 - Abhishek Chandra,
PY - 2010
KW - Resource discovery
KW - aggregation
KW - resource management
KW - machine learning.
VL - 21
JA - IEEE Transactions on Parallel and Distributed Systems
ER -
 
Michael Cardosa, University of Minnesota, Minneapolis
Abhishek Chandra, University of Minnesota, Minneapolis
Resource discovery is an important process for finding suitable nodes that satisfy application requirements in large loosely coupled distributed systems. Besides internode heterogeneity, many of these systems also show a high degree of intranode dynamism, so that selecting nodes based only on their recently observed resource capacities can lead to poor deployment decisions resulting in application failures or migration overheads. However, most existing resource discovery mechanisms rely mainly on recent observations to achieve scalability in large systems. In this paper, we propose the notion of a resource bundle—a representative resource usage distribution for a group of nodes with similar resource usage patterns—that employs two complementary techniques to overcome the limitations of existing techniques: resource usage histograms to provide statistical guarantees for resource capacities and clustering-based resource aggregation to achieve scalability. Using trace-driven simulations and data analysis of a month-long PlanetLab trace, we show that resource bundles are able to provide high accuracy for statistical resource discovery, while achieving high scalability. We also show that resource bundles are ideally suited for identifying group-level characteristics (e.g., hot spots, total group capacity). To automatically parameterize the bundling algorithm, we present an adaptive algorithm that can detect online fluctuations in resource heterogeneity.

[1] D. Anderson, "BOINC: A System for Public-Resource Computing and Storage," Proc. IEEE/ACM Int'l Workshop Grid Computing (GRID), 2004.
[2] V. Lo, D. Zappala, D. Zhou, Y. Liu, and S. Zhao, "Cluster Computing on the Fly: P2P Scheduling of Idle Cycles in the Internet," Proc. IEEE Fourth Int'l Conf. Peer-to-Peer Systems, 2004.
[3] Grid2: Blueprint for a New Computing Infrastructure, I. Foster and C. Kesselman, eds. M. Kauffman, 2004.
[4] Y. Chawathe, S. Ratnasamy, L. Breslau, N. Lanham, and S. Shenker, "Making Gnutella Like P2P Systems Scalable," Proc. ACM SIGCOMM, Aug. 2003.
[5] B. Cohen, "Incentives Build Robustness in Bittorrent," Proc. First Workshop the Economics of P2P Systems, June 2003.
[6] S. Guha, N. Daswani, and R. Jain, "An Experimental Study of the Skype Peer-to-Peer Voip System," Proc. Int'l Workshop Peer-to-Peer Systems (IPTPS), 2006.
[7] B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson, M. Wawrzoniak, and M. Bowman, "PlanetLab: An Overlay Testbed for Broad-Coverage Services," ACM SIGCOMM Computer Comm. Rev., vol. 33, no. 3, pp. 3-12, July 2003.
[8] A. Iamnitchi and I. Foster, "On Fully Decentralized Resource Discovery in Grid Environments," Proc. IEEE/ACM Int'l Workshop Grid Computing (GRID), 2001.
[9] D. Oppenheimer, J. Albrecht, D. Patterson, and A. Vahdat, "Distributed Resource Discovery on PlanetLab with SWORD," Proc. Workshop Real, Large Distributed Systems (WORLDS '04), Dec. 2004.
[10] J.-S. Kim, P. Keleher, M. Marsh, B. Bhattacharjee, and A. Sussman, "Using Content-Addressable Networks for Load Balancing in Desktop Grids," Proc. Int'l Symp. High Performance Distributed Computing (HPDC '07), June 2007.
[11] R. Raman, M. Livny, and M. Solomon, "Matchmaking: Distributed Resource Management for High Throughput Computing," Proc. Int'l Symp. High Performance Distributed Computing (HPDC '98), July 1998.
[12] D. Oppenheimer, B. Chun, D. Patterson, A.C. Snoeren, and A. Vahdat, "Service Placement in a Shared Wide Area Platform," Proc. USENIX Ann. Technical Conf., June 2006.
[13] P. Yalagandula and M. Dahlin, "A Scalable Distributed Information Management System," Proc. ACM SIGCOMM, 2004.
[14] B. Chun, J.M. Hellerstein, R. Huebsch, P. Maniatis, and T. Roscoe, "Design Considerations for Information Planes," Proc. Workshop Real, Large Distributed Systems (WORLDS '04), Dec. 2004.
[15] S. Zhong and J. Ghosh, "A Comparative Study of Generative Models for Document Clustering," Proc. SIAM Int'l Conf. Data Mining (SDM) Workshop Clustering High Dimensional Data and Its Applications, 2003.
[16] K. Park and V.S. Pai, "Comon: A Mostly-Scalable Monitoring System for Planetlab," ACM SIGOPS Operating Systems Rev., vol. 40, no. 1, pp. 65-74, 2006.
[17] J.M. Schopf, "A Practical Methodology for Defining Histograms for Predictions and Scheduling," NU technical report, 1999.
[18] S. Zhong http://www.cse.fau.edu/~zhong/softwareindex.htm , 2009.
[19] "PeerSim," http:/peersim.sourceforge.net, 2010.
[20] B. Urgaonkar, P. Shenoy, and T. Roscoe, "Resource Overbooking and Application Profiling in Shared Hosting Platforms," Proc. Symp. Operating Systems Design and Implementation (OSDI '02), Dec. 2002.
[21] T. Sandholm and K. Lai, "A Statistical Approach to Risk Mitigation in Computational Markets," Proc. Int'l Symp. High Performance Distributed Computing (HPDC), 2007.
[22] A. Gupta, D. Agrawal, and A.E. Abbadi, "Distributed Resource Discovery in Large Scale Computing Systems," Proc. Int'l Symp. Applications and the Internet (SAINT), 2005.
[23] Y.-S. Kee, D. Logothetis, R. Huang, H. Casanova, and A.A. Chien, "Efficient Resource Description and High Quality Selection for Virtual Grids," Proc. IEEE Int'l Symp. Cluster Computing and the Grid (CCGRID '05), pp. 598-606, 2005.
[24] R.V. Renesse, K.P. Birman, and W. Vogels, "Astrolabe: A Robust and Scalable Technology for Distributed System Monitoring, Management, and Data Mining," ACM Trans. Computer Systems, vol. 21, no. 2, pp. 164-206, 2003.
[25] J. Cappos and J.H. Hartman, "San Fermín: Aggregating Large Data Sets Using a Binomial Swap Forest," Proc. USENIX Symp. Networked Systems Design and Implementation (NSDI), 2008.
[26] J. Mickens and B. Noble, "Exploiting Availability Prediction in Distributed Systems," Proc. USENIX Symp. Networked Systems Design and Implementation (NSDI '06), May 2006.
[27] J. Rolia, X. Zhu, M. Arlitt, and A. Andrzejak, "Statistical Service Assurances for Applications in Utility Grid Environments," Technical Report HPL-2002-155, HP Labs, 2002.
[28] B. Urgaonkar, P. Shenoy, A. Chandra, and P. Goyal, "Dynamic Provisioning of Multi-Tier Internet Applications," Proc. Int'l Conf. Autonomic Computing (ICAC '05), 2005.
[29] R. Wolski, "Experiences with Predicting Resource Performance Online in Computational Grid Settings," Proc. ACM SIGMETRICS '03, 2003.
[30] "Network Weather Service," http://nws.cs.ucsb.eduewiki/, 2010.

Index Terms:
Resource discovery, aggregation, resource management, machine learning.
Citation:
Michael Cardosa, Abhishek Chandra, "Resource Bundles: Using Aggregation for Statistical Large-Scale Resource Discovery and Management," IEEE Transactions on Parallel and Distributed Systems, vol. 21, no. 8, pp. 1089-1102, Aug. 2010, doi:10.1109/TPDS.2009.143
Usage of this product signifies your acceptance of the Terms of Use.