Nikki Mirghafori
David Patterson
Abstract
The System Performance Evaluation Cooperative (SPEC) benchmarks are a set of integer and floating-point programs that are intended to be “effective and fair in comparing the performance of high performance computing systems”. SPEC ratings are often quoted in company advertising and have been trusted as the de facto measure of comparison for computer systems. Recently, there has been some concern regarding the fairness and the value of these benchmarks for comparing computer systems.
In this paper we investigate the following two questions regarding the SPEC92 benchmark suite: 1) How sensitive are the SPEC ratings to various tunings? 2) How reproducible are the published results? For six vendors, we compare the published SPECpeak and SPECbase ratings, and observe an 11% average improvement in the SPECpeak ratings due to changes in the compiler flags alone. In our own attempt to reproduce the published SPEC ratings, we came across various “explicit” and “hidden” tuning parameters that we consider unrealistic. We suggest a new unit called SPECsimple that requires using only the -O compiler optimization flag, shared libraries, and standard system configuration. SPECsimple is designed to better match the performance experienced by a typical user. Our measured SPECsimples are 65-86% of the advertised SPECpeak performance. We conclude this paper by citing cases compiler optimizations specifically designed for SPEC programs, in which performance decreases drastically or the computed results are incorrect if the compiled program does not exactly match the SPEC benchmark program. These findings show that the fairness and value of the popular SPEC benchmarks are questionable.
- {1} Chan, Y., Sudarsanam, A., and Wolfe A. "The Effect of Compiler-Flag Tuning on SPEC Benchmark Performance," Computer Architecture News, Vol. 22, No. 4, pp. 60-70, September 1994. Google Scholar
Digital Library
- {2} Giladi, R., Ahituv, N. "SPEC as a Performance Evaluation Measure," Computer, Vol. 28, No. 8, pp. 33-42, August 1995. Google ScholarDigital Library
- {3} Glaeser, Christopher. Nullstone Corporation. E-mail communication. April 1995.Google Scholar
- {4} Reilly, Jeff. SPEC CPU Subcommittee Chair. Netnews posting. October 1994.Google Scholar
- {5} Rozhin, Mark. HP SPEC representative. E-mail communication. November 1994.Google Scholar
- {6} SPEC Newsletter, 6:3, September 1994.Google Scholar
- {7} SPEC Newsletter, 6:2, June 1994.Google Scholar
- {8} SPEC Newsletter, 6:1, March 1994.Google Scholar
- {9} SPEC Newsletter, 5:2, June 1993.Google Scholar
- {10} SPEC Open Systems Steering Committee Policy Document, January 1994.Google Scholar
- {11} Trent, Eugene. SGI SPEC representative. Phone communication. November 1994.Google Scholar
- {12} Vitale, Phil. HP SPEC representative. E-mail communications. November 1994 and July 1995.Google Scholar
Index Terms
- Truth in SPEC benchmarks
Recommendations
SPEC benchmarks and competitive results
In less than a year since its Introduction the System performance Evaluation Cooperative (SPEC ((TM))) benchmarks have established themselves as an Important and widely distributed benchmark suite for engineering and scientific workstations, displacing ...
Evolution and evaluation of SPEC benchmarks
We present a method for quantitative evaluation of SPEC benchmarks. The method is used for the analysis of three generations of SPEC component-level benchmarks: SPEC89, SPEC92, and SPEC95. Our approach is suitable for studying (1) the redundancy between ...
Large System Performance of SPEC OMP2001 Benchmarks
ISHPC '02: Proceedings of the 4th International Symposium on High Performance ComputingPerformance characteristics of application programs on large-scale systems are often significantly different from those on smaller systems. SPEC OMP2001 is a benchmark suite intended for measuring performance of modern shared memory parallel systems. ...
Comments