This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
BURN: Enabling Workload Burstiness in Customized Service Benchmarks
July-Aug. 2012 (vol. 38 no. 4)
pp. 778-793
ASCII Text
x 
 
Giuliano Casale, Amir Kalbasi, Diwakar Krishnamurthy, Jerry Rolia, "BURN: Enabling Workload Burstiness in Customized Service Benchmarks," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 778-793, July-Aug., 2012.
 
 
BibTex
x
 
@article{ 10.1109/TSE.2011.58,
author = {Giuliano Casale and Amir Kalbasi and Diwakar Krishnamurthy and Jerry Rolia},
title = {BURN: Enabling Workload Burstiness in Customized Service Benchmarks},
journal ={IEEE Transactions on Software Engineering},
volume = {38},
number = {4},
issn = {0098-5589},
year = {2012},
pages = {778-793},
doi = {http://doi.ieeecomputersociety.org/10.1109/TSE.2011.58},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
}
 
 
RefWorks Procite/RefMan/Endnote
x 
 
TY - JOUR
JO - IEEE Transactions on Software Engineering
TI - BURN: Enabling Workload Burstiness in Customized Service Benchmarks
IS - 4
SN - 0098-5589
SP778
EP793
EPD - 778-793
A1 - Giuliano Casale,
A1 - Amir Kalbasi,
A1 - Diwakar Krishnamurthy,
A1 - Jerry Rolia,
PY - 2012
KW - Benchmarking
KW - performance
KW - burstiness
KW - bottleneck migration
KW - overdemand
VL - 38
JA - IEEE Transactions on Software Engineering
ER -
 
Giuliano Casale, Imperial College London, London
Amir Kalbasi, University of Calgary, Calgary
Diwakar Krishnamurthy, University of Calgary, Calgary
Jerry Rolia, HP Labs, Palo Alto
We introduce BURN, a methodology to create customized benchmarks for testing multitier applications under time-varying resource usage conditions. Starting from a set of preexisting test workloads, BURN finds a policy that interleaves their execution to stress the multitier application and generate controlled burstiness in resource consumption. This is useful to study, in a controlled way, the robustness of software services to sudden changes in the workload characteristics and in the usage levels of the resources. The problem is tackled by a model-based technique which first generates Markov models to describe resource consumption patterns of each test workload. Then, a policy is generated using an optimization program which sets as constraints a target request mix and user-specified levels of burstiness at the different resources in the system. Burstiness is quantified using a novel metric called overdemand, which describes in a natural way the tendency of a workload to keep a resource congested for long periods of time and across multiple requests. A case study based on a three-tier application testbed shows that our method is able to control and predict burstiness for session service demands at a fine-grained scale. Furthermore, experiments demonstrate that for any given request mix our approach can expose latency and throughput degradations not found with nonbursty workloads having the same request mix.

[1] C. Amza, A. Ch, A.L. Cox, S. Elnikety, R. Gil, K. Rajamani, E. Cecchet, and J. Marguerite, "Specification and Implementation of Dynamic Web Site Benchmarks," Proc. Fifth Workshop Workload Characterization, 2002.
[2] G. Balbo and G. Serazzi, "Asymptotic Analysis of Multiclass Closed Queueing Networks: Common Bottleneck," Performance Evaluation, vol. 26, no. 1, pp. 51-72, 1996.
[3] G. Balbo and G. Serazzi, "Asymptotic Analysis of Multiclass Closed Queueing Networks: Multiple Bottlenecks," Performance Evaluation, vol. 30, no. 3, pp. 115-152, 1996.
[4] P. Barford and M. Crovella, "Generating Representative Web Workloads for Network and Server Performance Evaluation," ACM Performance Evaluation Rev., vol. 26, no. 1, pp. 151-160, 1998.
[5] G. Bolch, S. Greiner, H. de Meer, and K.S. Trivedi, Queueing Networks and Markov Chains. Wiley, 2006.
[6] G. Casale, A. Kalbasi, D. Krishnamurthy, and J. Rolia, "Automated Stress Testing of Multi-Tier Systems by Dynamic Bottleneck Switch Generation," Technical Report SERG-2009-02, Univ. of Calgary, http://people.ucalgary.ca/dkrishnaSERG-2009-02.pdf , Apr. 2009.
[7] G. Casale, A. Kalbasi, D. Krishnamurthy, and J. Rolia, "Automated Stress Testing of Multi-Tier Systems by Dynamic Bottleneck Switch Generation," Proc. ACM/IFIP/USENIX 10th Int'l Conf. Middleware, pp. 393-413, 2009.
[8] G. Casale and G. Serazzi, "Bottlenecks Identification in Multiclass Queueing Networks Using Convex Polytopes," Proc. IEEE CS 12th Ann. Int'l Symp. Modeling, Analysis, and Simulation of Computer and Telecomm. Systems, pp. 223-230, Oct. 2004.
[9] G. Casale, E.Z. Zhang, and E. Smirni, "KPC-Toolbox: Simple Yet Effective Trace Fitting Using Markovian Arrival Processes," Proc. Fifth Int'l Conf. Quantitative Evaluation of Systems, pp. 83-92, Sept. 2008.
[10] G. Casale, E.Z. Zhang, and E. Smirni, "Trace Data Characterization and Fitting for Markov Modeling," Performance Evaluation, vol. 67, no. 2, pp. 61-79, Feb. 2010.
[11] G. Casale, E.Z. Zhang, and E. Smirni, "KPC-Toolbox: Best Recipes for Automatic Trace Fitting Using Markovian Arrival Processes," Performance Evaluation, vol. 67, no. 9, pp. 873-896, Sept. 2010.
[12] G. Casale, N. Mi, and E. Smirni, "Bound Analysis of Closed Queueing Networks with Workload Burstiness," Proc. ACM SIGMETRICS Int'l Conf. Measurement and Modeling of Computer Systems, pp. 13-24, 2008.
[13] M. Crovella and L. Lipsky, "Long-Lasting Transient Conditions in Simulations with Heavy-Tailed Workloads," Proc. 29th Conf. Winter Simulation, pp. 1005-1012, 1997.
[14] J.J. Dujmovic, "Universal Benchmark Suites," Proc. Seventh Int'l Symp. Modeling, Analysis and Simulation of Computer and Telecomm. Systems, pp. 197-205, 1999.
[15] D. Garcia and J. Garcia, "TPC-W E-Commerce Benchmark Evaluation," Computer, vol. 36, no. 2, pp. 42-48, Feb. 2003.
[16] R. Grace, The Benchmark Book. Prentice Hall, 1996.
[17] R. Gusella, "Characterizing the Variability of Arrival Processes with Indexes of Dispersion," IEEE J. Selected Areas in Comm., vol. 9, no. 2, pp. 203-211, Feb. 1991.
[18] R. Hashemian, D. Krishnamurthy, and M. Arlitt, "Web Workload Generation Challenges—An Empirical Investigation," Technical Report HPL-2010-163, HP Labs, 2010.
[19] H.P. LoadRunner, http://www.hp.com/goloadrunner, 2011.
[20] H. Kobayashi and B.L. Mark, System Modeling and Analysis: Foundations of System Performance Evaluation. Pearson, 2008.
[21] K. Kant, V. Tewary, and R. Iyer, "An Internet Traffic Generator for Server Architecture Evaluation," Proc. Workshop Computer Architecture Evaluation Using Commercial Workloads, Jan. 2001.
[22] D. Krishnamurthy, J.A. Rolia, and S. Majumdar, "A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems," IEEE Trans. Software Eng., vol. 32, no. 11, pp. 868-882, Nov. 2006.
[23] U. Krishnaswamy and D. Scherson, "A Framework for Computer Performance Evaluation Using Benchmark Sets," IEEE Trans. Computers, vol. 49, no. 12, pp. 1325-1338, Dec. 2000.
[24] W.E. Leland, M.S. Taqqu, W. Willinger, and D.V. Wilson, "On the Self-Similar Nature of Ethernet Traffic," IEEE/ACM Trans. Networking, vol. 2, no. 1, pp. 1-15, Feb. 1994.
[25] M. Litoiu, J.A. Rolia, and G. Serazz, "Designing Process Replication and Activation: A Quantitative Approach," IEEE Trans. Software Eng., vol. 26, no. 12, pp. 1168-1178, Dec. 2000.
[26] S. Malkowski, M. Hedwig, and C. Pu, "Experimental Evaluation of N-Tier Systems: Observation and Analysis of Multi-Bottlenecks," Proc. IEEE Int'l Symp. Workload Characterization, pp. 118-127, 2009.
[27] N. Mi, G. Casale, L. Cherkasova, and E. Smirni, "Burstiness in Multi-Tier Applications: Symptoms, Causes, and New Models," Proc. Ninth ACM/IFIP/USENIX Int'l Conf. Middleware, pp. 265-286, 2008.
[28] N. Mi, G. Casale, L. Cherkasova, and E. Smirni, "Injecting Realistic Burstiness to a Traditional Client-Server Benchmark," Proc. Sixth Int'l Conf. Autonomic Computing, pp. 149-158, June 2009.
[29] N. Mi, Q. Zhang, A. Riska, E. Smirni, and E. Riedel, "Performance Impacts of Autocorrelated Flows in Multi-Tiered Systems," Performance Evaluation, vol. 64, nos. 9-12, pp. 1082-1101, 2007.
[30] D. Mosberger and T. Jin, "httperf—A Tool for Measuring Web Server Performance," Technical Report HPL-98-61, Hewlett Packard Laboratories, 1998.
[31] M.F. Neuts, "Renewal Processes of Phase Type," Naval Research Logistics Quarterly, vol. 25, pp. 445-454, 1978.
[32] M.F. Neuts, "A Versatile Markovian Point Process," J. Applied Probability, vol. 16, no. 4, pp. 764-779, Dec. 1979.
[33] A. Panchenko and A. Thümmle, "Efficient Phase-Type Fitting with Aggregated Traffic Traces," Performance Evaluation, vol. 64, nos. 7-8, pp. 629-645, 2007.
[34] A. Riska and E. Riedel, "Long-Range Dependence at the Disk Drive Level," Proc. Third Int'l Conf. Quantitative Evaluation of Systems, pp. 41-50, 2006.
[35] J. Rolia, D. Krishnamurthy, G. Casale, and S. Dawson, "BAP: A Benchmark-Driven Algebraic Method for the Performance Engineering of Customized Services, Invited Paper," Proc. First Joint WOSP/SIPEW Int'l Conf. Performance Eng., Jan. 2010.
[36] A. Kalbasi, D. Krishnamurthy, J. Rolia, and S. Dawson, "DEC: Service Demand Estimation with Confidence," IEEE Trans. Software Eng., vol. 38, no. 3, pp. 561-578, May/June 2012.
[37] SAP Standard Application Benchmark, http://www.sap.com/solutions/benchmarkindex.epx , 2011.
[38] J.W.J. Xue, A.P. Chester, L. He, and S.A. Jarvis, "Dynamic Resource Allocation in Enterprise Systems," Proc. IEEE 14th Int'l Conf. Parallel and Distributed Systems, pp. 203-212, 2008.

Index Terms:
Benchmarking, performance, burstiness, bottleneck migration, overdemand
Citation:
Giuliano Casale, Amir Kalbasi, Diwakar Krishnamurthy, Jerry Rolia, "BURN: Enabling Workload Burstiness in Customized Service Benchmarks," IEEE Transactions on Software Engineering, vol. 38, no. 4, pp. 778-793, July-Aug. 2012, doi:10.1109/TSE.2011.58
Usage of this product signifies your acceptance of the Terms of Use.