Improving Computer Architecture Simulation Methodology by Adding Statistical Rigor
November 2005 (vol. 54 no. 11)
pp. 1360-1373
DOI Bookmark:
http://doi.ieeecomputersociety.org/10.1109/TC.2005.184
Due to cost, time, and flexibility constraints, computer architects use simulators to explore the design space when developing new processors and to evaluate the performance of potential enhancements. However, despite this dependence on simulators, statistically rigorous simulation methodologies are typically not used in computer architecture research. A formal methodology can provide a sound basis for drawing conclusions gathered from simulation results by adding statistical rigor and, consequently, can increase the architect's confidence in the simulation results. This paper demonstrates the application of a rigorous statistical technique to the setup and analysis phases of the simulation process. Specifically, we apply a Plackett and Burman design to: 1) identify key processor parameters, 2) classify benchmarks based on how they affect the processor, and 3) analyze the effect of processor enhancements. Our results showed that, out of the 41 user-configurable parameters in SimpleScalar, only 10 had a significant effect on the execution time. Of those 10, the number of reorder buffer entries and the L2 cache latency were the two most significant ones, by far. Our results also showed that Instruction Precomputation—a value reuse-like microarchitectural technique— primarily improves the processor's performance by relieving integer ALU contention.
[1] P. Bannon and Y. Saito, “The Alpha 21164PC Microprocessor,” Proc. IEEE Int'l Computer Conf., 1997.[2] B. Black and J. Shen, “Calibration of Microprocessor Performance Models,” Computer, vol. 31, no. 5, pp. 59-65, May 1998.[3] D. Burger and T. Austin, “The SimpleScalar Tool Set, Version 2.0,” Technical Report #1342, Computer Sciences Dept., Univ. of Wisconsin-Madison, 1997.[4] R. Desikan, D. Burger, and S. Keckler, “Measuring Experimental Error in Microprocessor Simulation,” Proc. Int'l Symp. Computer Architecture, 2001.[5] L. Eeckhout, H. Vandierendonck, and K. De Bosschere, “Workload Design: Selecting Representative Program-Input Pairs,” Proc. Int'l Conf. Parallel Architectures and Compilations Techniques, 2002.[6] J. Gibson, R. Kunz, D. Ofelt, M. Horowitz, J. Hennessy, and M. Heinrich, “FLASH vs. (Simulated) FLASH: Closing the Simulation Loop,” Proc. Int'l Conf. Architectural Support for Programming Languages and Operating Systems, 2000.[7] T. Horel and G. Lauterbach, “UltraSPARC-III: Designing Third-Generation 64-Bit Performance,” IEEE Micro, vol. 19, no. 3, pp. 73-85, May-June 1999.[8] R. Kessler, E. McLellan, and D. Webb, “The Alpha 21264 Microprocessor Architecture,” Proc. Int'l Conf. Computer Design, 1998.[9] A. KleinOsowski and D.J. Lilja, “MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research,” Computer Architecture Letters, vol. 1, 2002.[10] A. Kumar, “The HP PA-8000 RISC CPU,” IEEE Micro, vol. 17, no. 2, pp. 27-32, Mar.-Apr. 1997.[11] D. Lilja, Measuring Computer Performance. Cambridge Univ. Press, 2000.[12] D. Montgomery, Design and Analysis of Experiments, third ed. Wiley, 1991.[13] K. Normoyle, M. Csoppenszky, A. Tzeng, T. Johnson, C. Furman, and J. Mostoufi, “UltraSPARC-IIi: Expanding the Boundaries of a System on a Chip,” IEEE Micro, vol. 18, no. 2, pp. 14-24, Mar.-Apr. 1998.[14] R. Plackett and J. Burman, “The Design of Optimum Multifactorial Experiments,” Biometrika, vol. 33, no. 4, pp. 305-325, June 1946.[15] J. Silc, B. Robic, and T. Ungerer, Processor Architecture: From Dataflow to Superscalar and Beyond. Springer-Verlag, 1999.[16] D. Sima, T. Fountain, and P. Kacsuk, Advanced Computer Architectures, A Design Space Approach. Addison Wesley Longman, 1997.[17] A. Sodani and G. Sohi, “Dynamic Instruction Reuse,” Proc. Int'l Symp. Computer Architecture, 1997.[18] S. Song, M. Denman, and J. Chang, “The PowerPC 604 RISC Microprocessor,” IEEE Micro, vol. 14, no. 5, pp. 8-17, Oct. 1994.[19] M. Tremblay and J.M. O'Connor, “UltraSparc I: A Four-Issue Processor Supporting Multimedia,” IEEE Micro, vol. 16, no. 2, pp. 42-50, Apr. 1996.[20] K. Yeager, “The MIPS R10000 Superscalar Microprocessor,” IEEE Micro, vol. 16, no. 2, pp. 28-40, Apr. 1996.[21] J. Yi, R. Sendag, and D. Lilja, “Increasing Instruction-Level Parallelism with Instruction Precomputation,” Proc. Int'l Euro-Par Conf. Parallel Processing, 2002.[22] J. Yi and D. Lilja, “Improving Processor Performance by Simplifying and Bypassing Trivial Computations,” Proc. Int'l Conf. Computer Design, 2002.[23] J. Yi and D. Lilja, “Effects of Processor Parameter Selection on Simulation Results,” Minnesota Supercomputer Inst. Report 2002/146, 2002.[24] J. Yi, D. Lilja, and D. Hawkins, “A Statistically Rigorous Approach for Improving Simulation Methodology,” Proc. Int'l Conf. High-Performance Computer Architecture, Feb. 2003.
Index Terms:
Index Terms- Performance analysis and design aids, measurement techniques, simulation output analysis.
Citation:
Joshua J. Yi, David J. Lilja, Douglas M. Hawkins, "Improving Computer Architecture Simulation Methodology by Adding Statistical Rigor," IEEE Transactions on Computers, vol. 54, no. 11, pp. 1360-1373, Nov. 2005, doi:10.1109/TC.2005.184
Usage of this product signifies your acceptance of the
Terms of Use.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb