This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms
August 2009 (vol. 58 no. 8)
pp. 1111-1125
ASCII Text
x 
 
Sriram Govindan, Jeonghwan Choi, Arjun R. Nath, Amitayu Das, Bhuvan Urgaonkar, Anand Sivasubramaniam, "Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms," IEEE Transactions on Computers, vol. 58, no. 8, pp. 1111-1125, August, 2009.
 
 
BibTex
x
 
@article{ 10.1109/TC.2009.53,
author = {Sriram Govindan and Jeonghwan Choi and Arjun R. Nath and Amitayu Das and Bhuvan Urgaonkar and Anand Sivasubramaniam},
title = {Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms},
journal ={IEEE Transactions on Computers},
volume = {58},
number = {8},
issn = {0018-9340},
year = {2009},
pages = {1111-1125},
doi = {http://doi.ieeecomputersociety.org/10.1109/TC.2009.53},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Computers
TI - Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms
IS - 8
SN - 0018-9340
SP1111
EP1125
EPD - 1111-1125
A1 - Sriram Govindan,
A1 - Jeonghwan Choi,
A1 - Arjun R. Nath,
A1 - Amitayu Das,
A1 - Bhuvan Urgaonkar,
A1 - Anand Sivasubramaniam,
PY - 2009
KW - Virtual machine monitor
KW - hosting platform
KW - scheduler-induced delay
KW - Xen.
VL - 58
JA - IEEE Transactions on Computers
ER -
 
Sriram Govindan, The Pennsylvania State University, University Park
Jeonghwan Choi, The Pennsylvania State University, University Park
Arjun R. Nath, The Pennsylvania State University, University Park
Amitayu Das, The Pennsylvania State University, University Park
Bhuvan Urgaonkar, The Pennsylvania State University, University Park
Anand Sivasubramaniam, The Pennsylvania State University, University Park
Recent advances in software and architectural support for server virtualization have created interest in using this technology in the design of consolidated hosting platforms. Since virtualization enables easier and faster application migration as well as secure colocation of antagonistic applications, higher degrees of server consolidation are likely to result in such virtualization-based hosting platforms (VHPs). We identify two shortcomings in existing virtual machine monitors (VMMs) that prove to be obstacles in operating hosting platforms, such as Internet data centers, under conditions of such high consolidation: 1) CPU schedulers that are agnostic to the communication behavior of modern, multitier applications and 2) inadequate or inaccurate mechanisms for accounting the CPU overheads of I/O virtualization. We develop a new communication-aware CPU scheduling algorithm and a CPU usage accounting mechanism. We implement our algorithms in the Xen VMM and build a prototype VHP on a cluster of 36 servers. Our experimental evaluation with realistic Internet server applications and benchmarks demonstrates the performance/cost benefits and the wide applicability of our algorithms. For example, the TPC-W benchmark exhibited improvements in average response times between 20 percent and 35 percent for a variety of consolidation scenarios. A streaming media server hosted on our prototype VHP was able to satisfactorily service up to 3.5 times as many clients as one running on the default Xen.

[1] M. Adler, Y. Gong, and A. Rosenberg, “Optimal Sharing of Bags of Tasks in Heterogeneous Clusters,” Proc. 15th Ann. ACM Symp. Parallel Algorithms and Architectures (SPAA), Scheduling I, pp. 1-10, 2003.
[2] J. Anderson, L. Berc, J. Dean, S. Ghemawat, M. Henzinger, S. Lueng, M. Vandervoorde, C. Waldspurger, and W. Weihl, “Continuous Profiling: Where Have All the Cycles Gone?,” Proc. 16th ACM Symp. Operating Systems Principles, pp. 1-14, Oct. 1997.
[3] A. Arpaci-Dusseau, “Implicit Coscheduling: Coordinated Scheduling with Implicit Information in Distributed Systems,” ACM Trans. Computer Systems, vol. 19, no. 3, pp. 283-331, 2001.
[4] A. Arpaci-Dusseau and D. Culler, “Extending Proportional-Share Scheduling to a Network of Workstations,” Proc. Conf. Parallel and Distributed Processing Techniques and Applications (PDPTA '97), June 1997.
[5] G. Banga, P. Druschel, and J. Mogul, “Resource Containers: A New Facility for Resource Management in Server Systems,” Proc. Third Symp. Operating System Design and Implementation (OSDI '99), pp. 45-58, Feb. 1999.
[6] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebuer, I. Pratt, and A. Warfield, “Xen and the Art of Virtualization,” Proc. 19th Symp. Operating Systems Principles (SOSP), 2003.
[7] R. Barr, Z.J. Haas, and R. van Renesse, “JiST: An Efficient Approach to Simulation Using Virtual Machines: Research Articles,” Software Practice and Experience, vol. 35, no. 6, pp. 539-576, 2005.
[8] M. Benani and D. Menasce, “Resource Allocation for Autonomic Data Centers Using Analytic Performance Models,” Proc. IEEE Int'l Conf. Autonomic Computing (ICAC '05), June 2005.
[9] M. Bennani and D. Menacse, “Autonomic Virtualized Environments,” Proc. IEEE Int'l Conf. Autonomic and Autonomous Systems (ICAS '06), July 2006.
[10] E. Berger, S. Kaplan, B. Urgaonkar, P. Sharma, A. Chandra, and P. Shenoy, “Scheduler-Aware Virtual Memory Management,” Proc. 19th ACM Symp. Operating Systems Principles (SOSP '03), Oct. 2003.
[11] NAS Parallel Benchmarks, http://www.nas.nasa.gov/Resources/Software npb.html, 2009.
[12] B. Buck and P. Keleher, “Locality and Performance of Page- and Object-Based DSMs,” Proc. First Merged Int'l Parallel Processing Symp./Symp. Parallel and Distributed Processing (IPPS/SPDP '98), pp. 687-693, 1998.
[13] A. Chandra, M. Adler, P. Goyal, and P. Shenoy, “Surplus Fair Scheduling: A Proportional-Share CPU Scheduling Algorithm for Symmetric Multiprocessors,” Proc. Fourth Symp. Operating System Design and Implementation (OSDI '00), Oct. 2000.
[14] Y. Chen, A. Das, Q. Wang, A. Sivasubramaniam, R. Harper, and M. Bland, “Consolidating Clients on Back-End Servers with Co-Location and Frequency Control,” Proc. ACM SIGMETRICS Int'l Conf. Measurement and Modeling of Computer Systems (SIGMETRICS '06), June 2006.
[15] L. Cherkasova and R. Gardner, “Measuring CPU Overhead for I/O Processing in the Xen Virtual Machine Monitor,” Proc. 2005 USENIX Ann. Technical Conf., pp. 387-390, Apr. 2005.
[16] B. Clark, T. Deshane, E. Dow, S. Evanchik, M. Finlayson, J. Herne, and J. Matthewss, “Xen and the Art of Repeated Research,” Proc. USENIX Ann. Technical Conf. FREENIX '04, July 2004.
[17] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, “Live Migration of Virtual Machines,” Proc. Second Symp. Networked Systems Design and Implementation (NSDI'05), May 2005.
[18] M. Crovella and A. Bestavros, “Self-Similarity in World Wide Web Traffic: Evidence and Possible Causes,” Proc. SIGMETRICS '96: The ACM Int'l Conf. Measurement and Modeling of Computer Systems., May 1996.
[19] R. Doyle, J. Chase, O. Asad, W. Jin, and A. Vahdat, “Model-Based Resource Provisioning in a Web Service Utility,” Proc. Fourth USENIX Symp. Internet Technologies and Systems, Mar. 2003.
[20] K.J. Duda and D.R. Cheriton, “Borrowed-Virtual-Time (BVT) Scheduling: Supporting Latency-Sensitive Threads in a General-Purpose Scheduler,” Proc. 17th ACM Symp. Operating Systems Principles, pp. 261-276, 1999.
[21] R. Goldberg, “Survey of Virtual Machine Research,” Computer, vol. 7, pp. 34-45, June 1974.
[22] K. Govil, D. Teodosiu, Y. Huang, and M. Rosenblum, “Cellular Disco: Resource Management Using Virtual Clusters on Shared-Memory Multiprocessors,” Proc. ACM Symp. Operating Systems Principles (SOSP '99), pp. 154-169, Dec. 1999.
[23] D. Gupta, L. Cherkasova, R. Gardner, and A. Vahdat, “Enforcing Performance Isolation Across Virtual Machines in Xen,” Proc. Seventh Int'l Middleware Conf., Nov./Dec. 2006.
[24] D. Gupta, R. Gardner, and L. Cherkasova, “XenMon: QoS Monitoring and Performance Profiling Tool,” Technical Report HPL-2005-187, HP Labs, 2005.
[25] Intel VT, http://www.intel.com/technology/itj/2006/ v10i3foreword.htm, 2009.
[26] S. Iyer and P. Druschel, “Anticipatory Scheduling: A Disk Scheduling Framework to Overcome Deceptive Idleness in Synchronous I/O,” Proc. 18th ACM Symp. Operating Systems Principles (SOSP '01), pp. 117-130, 2001.
[27] The JBoss Application Server, http:/www.jboss.org, 2009.
[28] M.B. Jones, D. Rosu, and M. Rosu, “CPU Reservations and Time Constraints: Efficient, Predictable Scheduling of Independent Activities,” Proc. 16th ACM Symp. Operating Systems Principles (SOSP '97), pp. 198-211, Dec. 1997.
[29] J. Reumann, A. Mehra, K. Shin, and D. Kandlur, “Virtual Services: A New Abstraction for Server Consolidation,” Proc. USENIX Ann. Technical Conf., June 2000.
[30] I. Leslie, D. McAuley, R. Black, T. Roscoe, P. Barham, D. Evers, R. Fairbairns, and E. Hyden, “The Design and Implementation of an Operating System to Support Distributed Multimedia Applications,” IEEE J. Selected Areas in Comm., vol. 14, no. 7, pp. 1280-1297, Sept. 1996.
[31] S.T. Leutenegger and M.K. Vernon, “The Performance of Multiprogrammed Multiprocessor Scheduling Algorithms,” Proc. ACM SIGMETRICS Conf. Measurement and Modeling of Computer Systems (SIGMETRICS '90), pp. 226-236, 1990.
[32] J. Liu, W. Huang, B. Abali, and D.K. Panda, “High Performance VMM-Bypass I/O in Virtual Machines,” Proc. USENIX Ann. Technical Conf. (USENIX '06), May/June 2006.
[33] A. Menon, A. Cox, and W. Zwaenepoel, “Optimizing Network Virtualization in Xen,” Proc. USENIX Ann. Technical Conf. (USENIX '06), May 2006.
[34] J. Moore, D. Irwin, L. Grit, S. Sprenkle, and J. Chase, “Managing Mixed-Use Clusters with Cluster-on-Demand,” technical report, Dept. of Computer Science, Duke Univ., Nov. 2002.
[35] MySQL, http:/www.mysql.com, 2009.
[36] M. Nelson, B. Lim, and G. Hutchins, “Fast Transparent Migration for Virtual Machines,” Proc. 2005 USENIX Ann. Technical Conf., pp.391-394, Apr. 2005.
[37] J. Nieh and M. Lam., “A SMART Scheduler for Multimedia Applications,” ACM Trans. Computer Systems, vol. 21, no. 2, pp.117-163, May 2003.
[38] D. Ongaro, A.L. Cox, and S. Rixner, “Scheduling I/O in Virtual Machine Monitors,” Proc. Fourth ACM SIGPLAN/SIGOPS Int'l Conf. Virtual Execution Environments (VEE '08), 2008.
[39] S. Osman, D. Subhraveti, G. Su, and J. Nieh, “The Design and Implementation of Zap: A System for Migrating Computing Environments,” Proc. Fifth USENIX Symp. Operating Systems Design and Implementation, pp. 361-376, 2002.
[40] A. Parekh and R. Gallager, “A Generalized Processor Sharing Approach to Flow Control in Integrated Services Networks—The Single Node Case,” Proc. IEEE INFOCOM '92, pp. 915-924, May 1992.
[41] P. Pradhan, R. Tewari, S. Sahu, A. Chandra, and P. Shenoy, “An Observation-Based Approach Towards Self-Managing Web Servers,” Proc. 10th Int'L Workshop Quality of Service (IWQoS '02), May 2002.
[42] R. Sailer, T. Jaeger, E. Valdez, R. Caceres, R. Perez, S. Berger, J. Griffin, and L. van Doorn, “Building a MAC-Based Security Architecture for the Xen Opensource Hypervisor,” Proc. 2005 Ann. Computer Security Applications Conference (ACSAC), pp.249-258, 2005.
[43] C. Sapuntzakis, R. Chandra, B. Pfaff, J. Chow, M.S. Lam, and M. Rosenblum, “Optimizing the Migration of Virtual Computers,” Proc. Fifth Symp. Operating Systems Design and Implementation, Dec. 2002.
[44] J. Shafer, D. Carr, A. Menon, S. Rixner, A.L. Cox, W. Zwaenepoel, and P. Willmann, “Concurrent Direct Network Access for Virtual Machine Monitors,” Proc. 2007 IEEE 13th Int'L Symp. High Performance Computer Architecture (HPCA '07), 2007.
[45] K. Shen, H. Tang, T. Yang, and L. Chu, “Integrated Resource Management for Cluster-Based Internet Services,” Proc. Fifth USENIX Symp. Operating Systems Design and Implementation (OSDI), Dec. 2002.
[46] S. Shende, A. Malony, J. Cuny, K. Lindlan, P. Beckman, and S. Karmesin, “Portable Profiling and Tracing for Parallel Scientific Applications Using C++,” Proc. ACM SIGMETRICS Symp. Parallel and Distributed Tools (SPDT), pp. 134-145, Aug. 1998.
[47] J.E. Smith and R. Nair, Virtual Machines: Architectures, Implementations and Applications. Morgan Kauffmann, 2004.
[48] P. Sobalvarro and W.E. Weihl, “Demand-Based Coscheduling of Parallel Jobs on Multiprogrammed Multiprocessors,” Proc. Workshop Job Scheduling Strategies for Parallel Processing (IPPS '95), pp.106-126, 1995.
[49] NYU TPC-W, http://www.cs.nyu.edupdsg/, 2009.
[50] B. Urgaonkar, P. Shenoy, A. Chandra, and P. Goyal, “Dynamic Provisioning of Multi-Tier Internet Applications,” Proc. Second IEEE Int'l Conf. Autonomic Computing (ICAC '05), June 2005.
[51] B. Urgaonkar, P. Shenoy, and T. Roscoe, “Resource Overbooking and Application Profiling in Shared Hosting Platforms,” Proc. Fifth USENIX Symp. Operating Systems Design and Implementation (OSDI '02), Dec. 2002.
[52] C. Waldspurger, “Memory Resource Management in VMWare ESX Server,” Proc. Fifth Symp. Operating System Design and Implementation (OSDI '02), Dec. 2002.
[53] C.A. Waldspurger and W.E. Weihl, “Lottery Scheduling: Flexible Proportional-Share Resource Management,” Proc. USENIX Symp. Operating System Design and Implementation (OSDI '94), Nov. 1994.
[54] A. Whitaker, M. Shaw, and S.D. Gribble, “Scale and Performance in the Denali Isolation Kernel,” Proc. Fifth Symp. Operating System Design and Implementation (OSDI '02), Dec. 2002.
[55] XenSource: Enterprise-Grade Open Source Virtualization, http:/www.xensource.com/, 2009.
[56] Y. Zhang, H. Franke, J. Moreira, and A. Sivasubramaniam, “Improving Parallel Job Scheduling by Combining Gang Scheduling and Backfilling Techniques,” Proc. 14th Int'l Symp. Parallel and Distributed Processing Symp. (IPDPS '00), pp. 133-142, 2000.
[57] Y. Zhang, A. Sivasubramaniam, J.E. Moreira, and H. Franke, “A Simulation-Based Study of Scheduling Mechanisms for a Dynamic Cluster Environment,” Proc. 11th ACM Int'l Conf. Supercomputing (ICS), pp. 100-109, 2000.

Index Terms:
Virtual machine monitor, hosting platform, scheduler-induced delay, Xen.
Citation:
Sriram Govindan, Jeonghwan Choi, Arjun R. Nath, Amitayu Das, Bhuvan Urgaonkar, Anand Sivasubramaniam, "Xen and Co.: Communication-Aware CPU Management in Consolidated Xen-Based Hosting Platforms," IEEE Transactions on Computers, vol. 58, no. 8, pp. 1111-1125, Aug. 2009, doi:10.1109/TC.2009.53
Usage of this product signifies your acceptance of the Terms of Use.