The Community for Technology Leaders
IEEE International Symposium on High-Performance Comp Architecture (2012)
New Orleans, LA, USA
Feb. 25, 2012 to Feb. 29, 2012
ISBN: 978-1-4673-0827-4
pp: 1-12
Tianshi Chen , State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Yunji Chen , State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Qi Guo , State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Olivier Temam , INRIA, Saclay, France
Yue Wu , State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
Weiwu Hu , State Key Laboratory of Computer Architecture, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
ABSTRACT
As a fundamental task in computer architecture research, performance comparison has been continuously hampered by the variability of computer performance. In traditional performance comparisons, the impact of performance variability is usually ignored (i.e., the means of performance measurements are compared regardless of the variability), or in the few cases where it is factored in using parametric confidence techniques, the confidence is either erroneously computed based on the distribution of performance measurements (with the implicit assumption that it obeys the normal law), instead of the distribution of sample mean of performance measurements, or too few measurements are considered for the distribution of sample mean to be normal. We first illustrate how such erroneous practices can lead to incorrect comparisons. Then, we propose a non-parametric Hierarchical Performance Testing (HPT) framework for performance comparison, which is significantly more practical than standard parametric techniques because it does not require to collect a large number of measurements in order to achieve a normal distribution of the sample mean. This HPT framework has been implemented as an open-source software.
INDEX TERMS
CITATION

T. Chen, W. Hu, Y. Chen, Q. Guo, O. Temam and Y. Wu, "Statistical performance comparisons of computers," IEEE International Symposium on High-Performance Comp Architecture(HPCA), New Orleans, LA, USA, 2012, pp. 1-12.
doi:10.1109/HPCA.2012.6169043
190 ms
(Ver 3.3 (11022016))