This Article 
 Bibliographic References 
 Add to: 
A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems
November 2006 (vol. 32 no. 11)
pp. 868-882
Enterprise applications are often business critical but lack effective synthetic workload generation techniques to evaluate performance. These workloads are characterized by sessions of interdependent requests that often cause and exploit dynamically generated responses. Interrequest dependencies must be reflected in synthetic workloads for these systems to exercise application functions correctly. This poses significant challenges for automating the construction of representative synthetic workloads and manipulating workload characteristics for sensitivity analyses. This paper presents a technique to overcome these problems. Given request logs for a system under study, the technique automatically creates a synthetic workload that has specified characteristics and maintains the correct interrequest dependencies. The technique is demonstrated through a case study involving a TPC-W e--commerce system. Results show that incorrect performance results can be obtained by neglecting interrequest dependencies, thereby highlighting the value of our technique. The study also exploits our technique to investigate the impact of several workload characteristics on system performance. Results establish that high variability in the distributions of session length, session idle times, and request service times can cause increased contention among sessions, leading to poor system responsiveness. To the best of our knowledge, these are the first results of this kind for a session-based system. We believe our technique is of value for studies where fine control over workload is essential.

[1] M. Arlitt, D. Krishnamurthy, and J. Rolia, “Characterizing the Scalability of a Large Web-Based Shopping System,” ACM Trans. Internet Technology, vol. 1, no. 1, pp. 44-69, Aug. 2001.
[2] P. Barford and M. Crovella, “Generating Representative Web Workloads for Network and Server Performance Evaluation,” Proc. ACM SIGMETRICS, pp. 151-160, June 1998.
[3] J. Beran, R. Sherman, M.S. Taqqu, and W. Willinger, “Long-Range Dependence in Variable-Bit-Rate Video Traffic,” IEEE Trans. Comm., vol. 43, no. 2-4, pp. 1566-1579, Mar. 1995.
[4] A. Chandra, W. Gong, and P. Shenoy, “Dynamic Resource Allocation for Shared Data Centers Using Online Measurements,” Proc. Int'l Conf. Measurements and Modeling of Computer Systems, pp.300-301, June 2003,
[5] M. Crovella and A. Bestavros, “Self-Similarity in World Wide Web Traffic: Evidence and Possible Causes,” IEEE/ACM Trans. Networking, vol. 5, no. 6, pp. 835-846, Dec. 1997.
[6] D. Ferrari, “On the Foundation of Artificial Workload Design,” Proc. ACM SIGMETRICS, pp. 8-14, 1984.
[7] W. Gong, Y. Liu, V. Misra, and D. Towsley, “On the Tails of Web File Size Distributions,” Proc. 39th Allerton Conf. Comm., Control, and Computing, Oct. 2001.
[8] R. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. John Wiley & Sons, 1991.
[9] K. Kant, V. Tewary, and R. Iyer, “An Internet Traffic Generator for Server Architecture Evaluation,” Proc. Workshop Computer Architecture Evaluation Using Commercial Workloads, Jan. 2001.
[10] T. Kelly, “Detecting Performance Anomalies in Global Applications,” Proc. Workshop Real, Large, Distributed Systems (WORLDS '05), pp. 43-48, Dec. 2005.
[11] D. Krishnamurthy, “Synthetic Workload Generation for Stress Testing Session-Based Systems,” PhD thesis, Dept. of Systems and Computer Eng., Carleton Univ., Ottawa, Canada, Jan. 2004.
[12] D. Krishnamurthy, J. Rolia, and S. Majumdar, “SWAT: A Tool for Stress Testing Session-Based Web Applications,” Int'l Computer Measurement Group Conf. (CMG '03), pp. 639-649, Dec. 2003.
[13] D. Krishnamurthy, J. Rolia, M. Arlitt, and S. Majumdar, “A Systematic Method for Creating Performance Tests for E-Commerce Systems,” 11th GI/ITG Conf. Measuring, Modeling and Evaluation of Computer and Comm. Systems, pp. 225-241, Sept. 2001.
[14] P. Lax, Linear Algebra. John Wiley & Sons, New York, 1997.
[15] Load Runner, loadrunner/, 2006.
[16] A. Mahanti, D. Eager, and C. Williamson, “Temporal Locality and Its Impact on Web Proxy Cache Performance,” ACM SIGMETRICS Performance Evaluation, vol. 42, no. 3, pp. 187-203, Sept. 2000.
[17] D. Menasce, V. Almeida, R. Reidi, F. Pelegrinelli, R. Fonesca, and W. Meira Jr., “In Search of Invariants in E-Business Workloads,” Proc. ACM Conf. Electronic Commerce, pp. 56-65, Oct. 2000.
[18] D. Menasce, V. Almeida, R. Fonesca, and M. Mendes, “A Methodology for Workload Characterization of E-Commerce Sites,” Proc. ACM Conf. Electronic Commerce, pp. 119-128, 1999.
[19] D. Mosberger and T. Jin, “httperf—A Tool for Measuring Web Server Performance,” Proc. Workshop Internet Server Performance, June 1998.
[20] K. Park, G.T. Kim, and M. Crovella, “On the Relationship between File Sizes, Transport Protocols, and Self-Similar Network Traffic,” Proc. Int'l Conf. Network Protocols, pp. 171-180, Oct. 1996.
[21] V. Paxon and S. Floyd, “Wide Area Traffic: The Failure of Poisson Modeling,” IEEE/ACM Trans. Networking, vol. 3, no. 3, pp.226-244, June 1995.
[22] S. Ranjan, J. Rolia, H. Fu, and E. Knightly, “QoS-Driven Server Migration for Internet Data Centers,” Proc. Int'l Workshop Quality of Service, pp. 3-12, May 2002.
[23] Silk Performer, s_performer s_performer.htm, 2006.
[24] TPC-W Web Commerce Benchmark, http://www.tpc.orgtpcw, 2006.
[25] K. Trivedi, Probability and Statistics with Reliability, Queuing, and Computer Science Applications. John Wiley & Sons, 2002.
[26] U. Vallamsetty, K. Kant, and P. Mohapatra, “Characterization of E-Commerce Traffic,” Electronic Commerce Research J., vol. 3, nos.1-2, pp. 167-192, Jan./Apr. 2003.
[27] W. Willinger, M. Taqqu, M. Sherman, and R. Wilson, “Self-Similarity through High Variability: Statistical Analysis of Ethernet LAN Traffic at the Source Level,” IEEE/ACM Trans. Networking, vol. 5, no. 1, pp. 71-85, Feb. 1997.

Index Terms:
Performance of systems, measurement techniques, modeling techniques, software engineering, testing tools, Internet applications, electronic commerce, Web servers.
Diwakar Krishnamurthy, Jerome A. Rolia, Shikharesh Majumdar, "A Synthetic Workload Generation Technique for Stress Testing Session-Based Systems," IEEE Transactions on Software Engineering, vol. 32, no. 11, pp. 868-882, Nov. 2006, doi:10.1109/TSE.2006.106
Usage of this product signifies your acceptance of the Terms of Use.