Issue No. 06 - November/December (2003 vol. 23)
DOI Bookmark: http://doi.ieeecomputersociety.org/10.1109/MM.2003.10010
The topic was handled lightly and briefly; in consequence the panel was much more successful than most panels.
The complaint was that many authors, academics in particular, when presenting evidence in favor of this or that new scheme for improving performance, tend to use SPEC programs without following the SPEC rules. They fail to use the whole suite, and frequently do not use the latest version, sometimes even including programs that have been formally withdrawn.
My personal view, formed after attending the panel discussion, is as follows: SPEC is an organization owned and financed by workstation manufacturers. It enables them to cooperate, at the precompetitive stage of design, by setting performance standards, and therefore determining in outline the architecture of the workstations that they will market competitively in due course (similar to the SIA roadmap's function at the semiconductor level).
SPEC benchmarks thus serve a specific purpose for a limited period. Whole suites or specific programs are then withdrawn.
The programs themselves nevertheless survive. They are useful to research workers because they are programs that are well described and well known. This is, in my view, a legitimate reuse of SPEC programs after they have fulfilled their original purpose.
Maurice V. Wilkes