This Article 
   
 Share 
   
 Bibliographic References 
   
 Add to: 
 
Digg
Furl
Spurl
Blink
Simpy
Google
Del.icio.us
Y!MyWeb
 
 Search 
   
Evolutionary Benchmark Subsetting
November/December 2008 (vol. 28 no. 6)
pp. 20-36
Zhanpeng Jin, University of Pittsburgh
Allen C. Cheng, University of Pittsburgh
To improve simulation efficiency and relieve burdened benchmarking efforts, this research proposes a biologically inspired, survival-of-the-fittest evolutionary methodology. The goal is to subset any given benchmark suite based on its inherent workload characteristics, desired workload space coverage, and total execution time. Given a user-specified workload space coverage threshold, the proposed technique can systematically yield the "fittest" time-efficient benchmark subset.

1. J.L. Henning, "SPEC CPU 2000: Measuring CPU Performance in the New Millennium," Computer, vol. 33, no. 7, 2000, pp. 28-35.
2. A. KleinOsowski and D. Lijia, "MinneSPEC: A New Spec Benchmark Workload for Simulation-Based Computer Architecture Research," IEEE Computer Architecture Letters, vol. 1, no. 1, 2002, pp. 7-10.
3. T. Sherwood et al., "Automatically Characterizing Large Scale Program Behavior," Proc. 10th Int'l Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS 02), ACM Press, 2002, pp. 45-57.
4. R. Wunderlich et al., "SMARTS: Accelerating Microarchitectural Simulation via Rigorous Statistical Sampling," Proc. Int'l Symp. Computer Architecture (ISCA 03), IEEE CS Press, 2003, pp. 84-95.
5. J.J. Yi and D.J. Lijia, "Simulation of Computer Architectures: Simulators, Benchmarks, Methodologies, and Recommendations," IEEE Trans. Computers, vol. 55, no. 3, 2006, pp. 268-280.
6. D. Citron, "MisSPECulation: Partial and Misleading Use of SPEC CPU2000 in Computer Architecture Conferences," Proc. Int'l Symp. Computer Architecture (ISCA 03), IEEE CS Press, 2003, pp. 52-61.
7. A. Joshi et al., "Measuring Benchmark Similarity Using Inherent Program Characteristics," IEEE Trans. Computers, vol. 55, no. 6, 2006, pp. 769-782.
8. L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Workload Design: Selecting Representative Program-Input Pairs," Proc. Int'l Conf. Parallel Architectures and Compilation Techniques (PACT 02), IEEE CS Press, 2002, pp. 83-94.
9. Y. Collette and P. Siarry, Multiobjective Optimization, Springer, 2003.
10. C.A. Coello, D.A. Van Veldhuizen, and G.B. Lamont, Evolutionary Algorithms for Solving Multi-Objective Problems, 2nd ed., Springer, 2007.
11. D.E. Goldberg, Genetic Algorithms in Search, Optimization and Machine Learning, 1st ed., Addison-Wesley, 1989.
12. A. Phansalkar et al., Measuring Program Similarity, tech. report TR-050127-01, Lab for Computer Architecture, Univ. of Texas at Austin, 2004.
13. J.E. Goodman and J. O'Rourke, Handbook of Discrete and Computational Geometry, 2nd ed., Chapman &Hall, 2004.
14. T.H. Cormen et al., "Finding the Convex Hull," Introduction to Algorithms, 2nd ed., MIT Press and McGraw-Hill, 2001, pp. 947-957.
15. F.P. Preparata and S.J. Hong, "Convex Hulls of Finite Sets of Points in Two and Three Dimensions," Comm. ACM, vol. 20, no. 2, 1977, pp. 87-93.
16. M. De Berg et al., Computational Geometry, Algorithms and Applications, Springer, 2000.
17. T. Baeck, D.B. Fogel, and Z. Michalewicz, Handbook of Evolutionary Computation, Taylor &Francis Group, 1997.
18. R. Lowen and A. Verschoren, Foundations of Generic Optimization: Volume 2: Applications of Fuzzy Control, Genetic Algorithms and Neural Networks, Springer, 2007.
19. S.K. Pal and P.P. Wang, Genetic Algorithms for Pattern Recognition, CRC Press, 1996.
20. T. Wenisch et al., Applying SMARTS to SPEC CPU2000, CALCM tech. report 2003-1, Carnegie Mellon Univ., 2003.
21. L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Designing Computer Architecture Research Workloads," Computer, vol. 36, no. 2, 2003, pp. 65-71.
22. L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Quantifying the Impact of Input Data Sets on Program Behavior and Its Applications," J. Instruction-Level Parallelism, vol. 5, Feb. 2003, pp. 1-33.
23. L. Eeckhout, J. Sampson, and B. Calder, "Exploiting Program Microarchitecture Independent Characteristics and Phase Behavior for Reduced Benchmark Suite Simulation," Proc. Int'l Symp. Workload Characterization (IISWC 05), IEEE CS Press, 2005, pp. 2-12.
1. A. KleinOsowski and D. Lijia, "MinneSPEC: A New SPEC Benchmark Workload for Simulation-Based Computer Architecture Research," IEEE Computer Architecture Letters, vol. 1, no. 1, 2002, pp. 7-10.
2. A. Alameldeen et al., "Simulating a $2M Commercial Server on a $2K PC," Computer, vol. 36, no. 2, 2003, pp. 50-57.
3. T. Sherwood et al., "Automatically Characterizing Large Scale Program Behavior," Proc. 10th Int'l Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS 02), ACM Press, 2002, pp. 45-57.
4. R. Wunderlich et al., "SMARTS: Accelerating Microarchitectural Simulation via Rigorous Statistical Sampling," Proc. Int'l Symp. Computer Architecture (ISCA 03), IEEE CS Press, 2003, pp. 84-95.
5. T. Wenisch et al., "TurboSMARTS: Accurate Microarchitecture Simulation Sampling in Minutes," Proc. ACM Sigmetrics Int'l Conf. Measurement and Modeling of Computer Systems, ACM Press, 2005, pp. 408-409.
6. P.D. Bryan, M.C. Rosier, and T.M. Conte, "Reverse State Reconstruction for Sampled Microarchitectural Simulation," Proc. Int'l Symp. Performance Analysis of Systems and Software (ISPASS 07), IEEE CS Press, 2007, pp. 190-199.
7. L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Workload Design: Selecting Representative Program-Input Pairs," Proc. Int'l Conf. Parallel Architectures and Compilation Techniques (PACT 02), IEEE CS Press, 2002, pp. 83-94.
8. L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Designing Computer Architecture Research Workloads," Computer, vol. 36, no. 2, 2003, pp. 65-71.
9. L. Eeckhout, H. Vandierendonck, and K. De Bosschere, "Quantifying the Impact of Input Data Sets on Program Behavior and Its Applications," J. Instruction-Level Parallelism, vol. 5, Feb 2003, pp. 1-33.
10. L. Eeckhout, J. Sampson, and B. Calder, "Exploiting Program Microarchitecture Independent Characteristics and Phase Behavior for Reduced Benchmark Suite Simulation," Proc. Int'l Symp. Workload Characterization (IISWC 05), IEEE CS Press, 2005, pp. 2-12.
11. Z. Jin and A.C. Cheng, "Improve Simulation Efficiency Using Statistical Benchmark Subsetting: An ImplantBench Case Study," Proc. Design Automation Conf. (DAC 08), ACM Press, 2008, pp. 970-973.
12. Z. Jin and A.C. Cheng, "ImplantBench: Characterizing and Projecting Representative Benchmarks for Emerging Bioimplantable Computing," IEEE Micro, vol. 28, no. 4, 2008, pp. 71-91.
13. K. Hoste and L. Eeckhout, "Comparing Benchmarks Using Key Microarchitecture-Independent Characteristics," Proc. Int'l Symp. Workload Characterization (IISWC 06), IEEE CS Press, 2006, pp. 83-92.
14. A. Joshi et al., "Measuring Benchmark Similarity Using Inherent Program Characteristics," IEEE Trans. Computers, vol. 55, no. 6, 2006, pp. 769-782.
15. A. Phansalkar et al., Measuring Program Similarity, tech. report TR-050127-01, Lab for Computer Architecture, Univ. of Texas at Austin, 2004.
16. A. Phansalkar, A. Joshi, and L.K. John, "SPEC CPU2006 Analysis: Subsetting the SPEC CPU2006 Benchmark Suite," ACM Sigarch Computer Architecture News, vol. 35, no. 1, 2007, pp. 69-76.
17. A. Phansalkar, A. Joshi, and L.K. John, "Analysis of Redundancy and Application Balance in the SPEC CPU2006 Benchmark Suite," Proc. Int'l Symp. Computer Architecture (ISCA 07), IEEE CS Press, 2007, pp. 412-423.
18. J.J. Yi, D.J. Lijia, and D.M. Hawkins, "A Statistically Rigorous Approach for Improving Simulation Methodology," Proc. Int'l Symp. High-Performance Computer Architecture (HPCA-9 03), IEEE CS Press, 2003, pp. 281-291.
19. J.J. Yi et al., "The Exigency of Benchmark and Compiler Drift: Designing Tomorrow's Processors with Yesterday's Tools," Proc. Int'l Conf. Supercomputing (ICS 06), ACM Press, 2006, pp. 75-86.
20. J.J. Yi et al., "Evaluating Benchmark Subsetting Approaches," Proc. Int'l Symp. Workload Characterization (IISWC 06), IEEE CS Press, 2006, pp. 93-104.

Index Terms:
benchmark, subsetting, genetic algorithm, center of mass, convex hull, fitness
Citation:
Zhanpeng Jin, Allen C. Cheng, "Evolutionary Benchmark Subsetting," IEEE Micro, vol. 28, no. 6, pp. 20-36, Nov.-Dec. 2008, doi:10.1109/MM.2008.87
Usage of this product signifies your acceptance of the Terms of Use.