This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Applying Statistical Sampling for Fast and Efficient Simulation of Commercial Workloads
November 2007 (vol. 56 no. 11)
pp. 1520-1533

Abstract—Commercial workloads form an important class of applications and have performance characteristics that are distinct from scientific and technical benchmarks such as SPEC CPU. However, due to the prohibitive simulation time of commercial workloads, it is extremely difficult to use them in computer architecture research. In this paper, we study the efficacy of using statistical sampling based simulation methodology for two classes of commercial workloads - a Java server benchmark, SPECjbb2000, and an Online Transaction Processing (OLTP) benchmark, DBT-2. Our results show that although SPECjbb2000 shows distinct garbage collection phases, there are no large-scale phases in the OLTP benchmark. We take advantage of this stationary behavior in steady phase, and propose a statistical sampling based simulation technique, DynaSim, with two dynamic stopping rules. In this approach, the simulation terminates once the target accuracy has been met. We apply DynaSim to simulate commercial workloads and show that with the simulation of only a few million total instructions, the error can be within 3% at a confidence level of 99%. DynaSim compares favorably with random sampling and representative sampling in terms of the total number of instructions simulated (time cost) and with representative sampling in terms of the number of checkpoints (storage cost). DynaSim increases the usability of a sampling based simulation approach for commercial workloads, and will encourage the use of commercial workloads in computer architecture research.

[1] R. Alameldeen and D.A. Wood, “Variability in Architectural Simulations of Multi-Threaded Workloads,” Proc. Ninth Int'l Symp. High-Performance Computer Architecture (HPCA '03), 2003.
[2] M. Biesbrouck, T. Sherwood, and B. Calder, “A Co-Phase Matrix to Guide Simultaneous Multithreading Simulation,” Proc. Fourth IEEE Int'l Symp. Performance Analysis of Software and Systems (ISPASS '04), 2004.
[3] W.G. Cochran, Sampling Techniques, third ed. John Wiley & Sons, 1977.
[4] T.M. Conte, M.A. Hirsch, and K.N. Menezes, “Reducing State Loss for Effective Trace Sampling of Superscalar Processors,” Proc. 1996 Int'l Conf. Computer Design (ICCD), pp. 468-477, 1996.
[5] R.R. Duersch and L.W. Schruben, “An Interactive Run Length Control for Simulations on PCs,” Proc. 1986 Winter Simulation Conf., pp. 866-870, 1986.
[6] M. Ekman and P. Stenstrom, “Enhancing Multiprocessor Architecture Simulation Speed Using Matched-Pair Comparison,” Proc. Fifth IEEE Int'l Symp. Performance Analysis of Systems and Software (ISPASS '05), Mar. 2005.
[7] V.S. Iyengar, L.H. Trevillyan, and P. Bose, “Representative Traces for Processor Models with Infinite Cache,” Proc. Second Int'l Symp. High-Performance Computer Architecture (HPCA '96), 1996.
[8] T. Lafage and A. Seznec, “Choosing Representative Slices of Program Execution for Microarchitecture Simulations: A Preliminary Application to the Data Stream,” Proc. Third IEEE Ann. Workshop Workload Characterization (WWC '00), pp. 102-110, 2000.
[9] W. Liu and M. Huang, “EXPERT: Expedited Simulation Exploiting Program Behavior Repetition,” Proc. 18th Ann. Int'l Conf. Supercomputing (ICS '04), pp. 126-135, June 2004.
[10] Y. Luo and L.K. John, “On Sampling Unit Size in Sampled Microprocessor Simulation,” Proc. 24th IEEE Int'l Performance Computing and Comm. Conf. (IPCCC '05), Apr. 2005.
[11] Y. Luo and L. John, “Simulating Commercial Java Throughput Workloads: A Case Study,” Proc. 23rd Int'l Conf. Computer Design (ICCD '05), 2005.
[12] P.S. Magnusson, M. Christensson, J. Eskilson, D. Forsgren, G. Hållberg, J. Högberg, F. Larsson, A. Moestedt, and B. Werner, “Simics: A Full System Simulation Platform,” Computer, vol. 35, no. 2, pp. 50-58, Feb. 2002.
[13] H. Patil, R. Cohn, M. Charney, R. Kapoor, A. Sun, and A. Karunanidhi, “Pinpointing Representative Portions of Large Intel Itanium Programs with Dynamic Instrumentation,” Proc. 37th Ann. IEEE/ACM Int'l Symp. Microarchitecture (MICRO '04), 2004.
[14] E. Perelman, G. Hamerly, and B. Calder, “Picking Statistically Valid and Early Simulation Points,” Proc. 12th Int'l Conf. Parallel Architectures and Compilation Techniques (PACT '03), pp. 244-255, Sept. 2003.
[15] L. Schruben, “Confidence Interval Estimation Using Standardized Time Series,” Operations Research, vol. 31, no. 6, pp. 1090-1107, 1983.
[16] T. Sherwood, E. Perelman, G. Hamerly, and B. Calder, “Automatically Characterizing Large Scale Program Behavior,” Proc. 10th Int'l Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS '02), pp. 45-57, Oct. 2002.
[17] K. Skadron, P.S. Ahuja, M. Martonosi, and D.W. Clark, “Branch Prediction, Instruction-Window Size, and Cache Size: Performance Trade-Offs and Simulation Techniques,” IEEE Trans. Computers, vol. 48, no. 11, pp. 1260-1281, Nov. 1999.
[18] R. Srinivasan, J. Cook, and S. Cooper, “Fast, Accurate Microarchitecture Simulation Using Statistical Phase Detection,” Proc. Fifth IEEE Int'l Symp. Performance Analysis of Systems and Software (ISPASS '05), Mar. 2005.
[19] R. Todi, “SPEClite: Using Representative Samples to Reduce SPEC CPU2000 Workload,” Proc. Fourth IEEE Ann. Workshop Workload Characterization (WWC '01), 2001.
[20] T. Wenisch, R. Wunderlich, M. Ferdman, A. Ailamaki, B. Falsafi, and J. Hoe, “SimFlex: Statistical Sampling of Computer System Simulation,” IEEE Micro, vol. 26, no. 4, July/Aug. 2006.
[21] R.E. Wunderlich, T.F. Wenisch, B. Falsafi, and J.C. Hoe, “SMARTS: Accelerating Microarchitecture Simulation via Rigorous Statistical Sampling,” Proc. 30th Ann. Int'l Symp. Computer Architecture (ISCA '03), pp. 84-95, 2003.
[22] Open Source Development Labs, Database Test 2, http://www. osdl.org/lab_activities/kernel_testing/ osdl_database_test_suiteosdl_dbt-2 /, 2007.
[23] “Statistical Sampling of Computer System Simulation,” IEEE Micro, vol. 26, no. 4, July/Aug. 2006.
[24] Standard Performance Evaluation Corp., SPEC JBB2000, http://www.spec.orgjbb2000/, 2007.
[25] Standard Performance Evaluation Corp., SPEC CPU2000 V1.2. http://www.spec.orgcpu2000/, 2007.
[26] Transaction Processing Performance Council, TPC-C, http://www.tpc.orgtpcc/, 2007.

Index Terms:
simulation, representative sampling, random sampling, commercial workloads
Citation:
Ajay Joshi, Yue Luo, Lizy K. John, "Applying Statistical Sampling for Fast and Efficient Simulation of Commercial Workloads," IEEE Transactions on Computers, vol. 56, no. 11, pp. 1520-1533, Nov. 2007, doi:10.1109/TC.2007.70748
Usage of this product signifies your acceptance of the Terms of Use.