The Community for Technology Leaders
RSS Icon
Issue No.02 - July-December (2013 vol.1)
pp: 201-214
Zhonghong Ou , Aalto University, Espoo
Hao Zhuang , EPFL, Lausanne
Andrey Lukyanenko , Aalto University, Espoo
Jukka K. Nurminen , Aalto University, Espoo
Pan Hui , The Hong Kong University of Science and Technology, Hong Kong and Telekom Innovation Laboratories, Berlin
Vladimir Mazalov , KRC of Russian Academy of Sciences, Karelia
Antti Yla-Jaaski , Alato University, Espoo
Public cloud platforms might start with homogeneous hardware; nevertheless, because of inevitable hardware upgrades, or adding more capacity, the initial homogeneous platform will gradually evolve into heterogeneous as time passes by. The consequent performance heterogeneity is of concern to cloud users. In this paper, we evaluate performance variations from hardware heterogeneity and scheduling mechanisms of public clouds. Amazon Elastic Compute Cloud (Amazon EC2) and Rackspace Cloud are used as the representatives because of their relatively long record and wide usage among small and medium enterprises (SMEs). A comprehensive set of microbenchmarks and application-level macrobenchmarks have been used to investigate performance variation. Several major contributions have been made. First, we find out that heterogeneous hardware is a commonality among the relatively long-lasting cloud platforms, although the level of heterogeneity varies. Second, we observe that heterogeneous hardware is the primary culprit of performance variation of cloud platforms. Third, we discover that varied CPU acquisition percentages and different virtual machine scheduling mechanisms exacerbate the performance variation problem, especially for network related operations. Finally, based on the observations, we propose cost-saving approaches and analyze Nash equilibrium from cloud user perspective. By using a simple "trial-and-betterâ' approach, i.e., keep good-performing instances and discard bad-performing instances, cloud users can achieve up to 30 percent cost saving.
Hardware, Central Processing Unit, Servers, Virtual machine monitors, Benchmark testing, Cloud computing,Amazon EC2, Hardware heterogeneity, VM scheduling mechanism, performance variation, cloud computing
Zhonghong Ou, Hao Zhuang, Andrey Lukyanenko, Jukka K. Nurminen, Pan Hui, Vladimir Mazalov, Antti Yla-Jaaski, "Is the Same Instance Type Created Equal? Exploiting Heterogeneity of Public Clouds", IEEE Transactions on Cloud Computing, vol.1, no. 2, pp. 201-214, July-December 2013, doi:10.1109/TCC.2013.12
[1] P. Mell and T. Grance, "The NIST Definition of Cloud Computing," Technical Report NIST Special Publication 800-145, The Nat'l Inst. of Standards and Technology (NIST), 2011.
[2] "Amazon EC2,", 2013.
[3] "Rackspace," http://www.rackspace.comcloud/, 2013.
[4] "Google Compute Engine," , 2013.
[5] "Microsoft Azure," http://www.windowsazure.comen-us/, 2013.
[6] Z. Ou, H. Zhuang, J.K. Nurminen, A. Ylä-Jääski, and P. Hui, "Exploiting Hardware Heterogeneity within the Same Instance Type of Amazon EC2," Proc. Fourth USENIX Conf. Hot Topics in Cloud Ccomputing (HotCloud '12), pp. 1-5, 2012.
[7] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, "Xen and the Art of Virtualization," Proc. ACM Symp. Operating Systems Principles (SOSP '03), pp. 164-177, 2003.
[8] L. Cherkasova, D. Gupta, and A. Vahdat, "Comparison of the Three CPU Schedulers in Xen," ACM SIGMETRICS Performance Evaluation Rev., vol. 35, no. 2, pp. 42-51, 2007.
[9] E. Walker, "Benchmarking Amazon EC2 for High-Performance Scientific Computing," USENIX; login:, vol. 33, no. 5, pp. 18-23, 2008.
[10] K. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. Wasserman, and N. Wright, "Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud," Proc. IEEE Second Int'l Conf. Cloud Computing Technology and Science (CloudCom '10), pp. 159-168, 2010.
[11] Y. Zhai, M. Liu, J. Zhai, X. Ma, and W. Chen, "Cloud versus In-House Cluster: Evaluating Amazon Cluster Compute Instances for Running MPI Applications," Proc. State of the Practice Reports (SC '11), pp. 1-10, 2011.
[12] A. Li, X. Yang, S. Kandula, and M. Zhang, "CloudCmp: Comparing Public Cloud Providers," Proc. 10th ACM SIGCOMM Conf. Internet Measurement (IMC '10), pp. 1-14, 2010.
[13] A. Li, X. Zong, S. Kandula, X. Yang, and M. Zhang, "Cloudprophet: Towards Application Performance Prediction in Cloud," Proc. ACM SIGCOMM '11 Conf., pp. 426-427, 2011.
[14] A. Lenk, M. Menzel, J. Lipsky, S. Tai, and P. Offermann, "What Are You Paying for? Performance Benchmarking for Infrastructure-as-a-Service Offerings," Proc. IEEE Int'l Conf. Cloud Computing (Cloud '11), pp. 484-491, 2011.
[15] G. Wang and T. Ng, "The Impact of Virtualization on Network Performance of Amazon EC2 Data Center," Proc. IEEE INFOCOM '10, pp. 1-9, Mar. 2010.
[16] J. Schad, J. Dittrich, and J.-A. Quianée-Ruiz, "Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance," Proc. VLDB Endowment, vol. 3, pp. 460-471, Sept. 2010.
[17] S.K. Barker and P. Shenoy, "Empirical Evaluation of Latency Sensitive Application Performance in the Cloud," Proc. First Ann. ACM SIGMM Conf. Multimedia Systems (MMSys '10), pp. 35-46, 2010.
[18] S. Suneja, E. Baron, E. de Lara, and R. Johnson, "Accelerating the Cloud with Heterogeneous Computing," Proc. Third USENIX Conf. Hot Topics in Cloud Computing (HotCloud '11), pp. 1-5, 2011.
[19] G. Lee, B. Chun, and R.H. Katz, "Heterogeneity-Aware Resource Allocation and Scheduling in the Cloud," Proc. Third USENIX Conf. Hot Topics in Cloud Computing (HotCloud '11), pp. 1-5, 2011.
[20] S. Yeo and H. Lee, "Using Mathematical Modeling in Provisioning a Heterogeneous Cloud Computing Environment," Computer, vol. 44, no. 8, pp. 55-62, Aug. 2011.
[21] A. Samih, R. Wang, C. Maciocco, T.-Y.C. Tai, R. Duan, J. Duan, and Y. Solihin, "Evaluating Dynamics and Bottlenecks of Memory Collaboration in Cluster Systems," Proc. IEEE/ACM 12th Int'l Symp. Cluster, Cloud and Grid Computing (CCGrid '12), pp. 107-114, 2012.
[22] B. Farley, V. Varadarajan, K. Bowers, A. Juels, T. Ristenpart, and M. Swift, "More for Your Money: Exploiting Performance Heterogeneity in Public Clouds," Proc. Third ACM Symp. Cloud Computing (SoCC '12), pp. 1-14, 2012.
[23] I. Leslie, D. Mcauley, R. Black, T. Roscoe, P. Barham, D. Evers, R. Fairbairns, and E. Hyden, "The Design and Implementation of an Operating System to Support Distributed Multimedia Applications," IEEE J. Selected Areas in Comm., vol. 14, no. 7, pp. 1280-1297, Sept. 1996.
[24] "Credit-Based CPU Scheduler," xenwikiCreditScheduler , 2013.
[25] "UnixBench,", 2013.
[26] "RAMspeed,", 2013.
[27] "Bonnie++,",++/, 2013.
[28] "Httperf," /, 2013.
[29] "Passmark CPU Benchmarks," http://www.cpubenchmark.nethigh, end cpus.html, 2013.
[30] B. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, "Benchmarking Cloud Serving Systems with YCSB," Proc. First ACM Symp. Cloud Computing (SoCC '10), pp. 143-154, 2010.
276 ms
(Ver 2.0)

Marketing Automation Platform Marketing Automation Tool