Issue No. 08 - August (1995 vol. 28)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/2.402073
System Performance Evaluation Cooperative (SPEC) measures are commonly considered the most effective indicators for evaluating engineering workstation performance. Indeed, these measures have gradually become a de facto industry standard for CPU performance evaluation. But do they truly provide added information value? The authors examine SPEC measures with the aim of learning their limitations and the areas in which caution should be exercised. They present results and evaluations of a comprehensive statistical analysis of SPEC rankings published in the SPEC Newsletter over a number of years. The major findings show that several programs in the benchmark suite contribute little additional information to the measures. In fact, the potency profile of the system can be demonstrated by just a few representative programs. Moreover, the method of aggregating the results to create a single measure (or two measures) leads to distortion in presenting the overall system performance. In evaluating the system, SPEC focuses on the CPU clock rate and thus may not reflect the performance of the entire system. This suggests that the added value of the SPEC information is not very high. The authors thus advise that SPEC should be employed cautiously with full awareness of its shortcomings.
R. Giladi and N. Ahituv, "SPEC as a Performance Evaluation Measure," in Computer, vol. 28, no. , pp. 33-42, 1995.