Reinhold Weicker
Abstract
Benchmarks, in particular the SPEC CPU benchmarks, are frequently used in academic computer research. With ASPLOS-7 as an example, observations about such usage are reported, and suggestions are made for a meaningful use of benchmarks in computer architecture research. Forward-looking computer architecture research may need more than one benchmark collection.
- 1. I-Cheng K. Cheng, John T. Coffey, and Trevor N. Mudge: Analysis of Branch Prediction via Data Compression. 7th Int. Conf. on Architectural Support for Programming Languages and Operating Systems, = ACM SIGPLAN Notices 31, 9 (Sept. 1996), 128-137. Google Scholar
Digital Library
- 2. Kaivalya Dixit and Jeff Reilly: SPEC95 Questions and Answers. SPEC Newsletter 7, 3 (Sept. 1995), 7-10. Also in http://www.specbench.org/osg/cpu95/news/cpu95qa.htmlGoogle Scholar
- 3. Jeffrey D. Gee, Mark D. Hill, Dionisios N. Pnevmatikos, and Alan J. Smith: Cache Performance of the SPEC92 Benchmark Suite. IEEE Micro (Aug. 1993), 17-27. Google ScholarDigital Library
- 4. Thomas M. Conte and Wen-mei W. Hu: A Brief Survey of Benchmark Usage in the Architecture Community. Computer Architecture News 19, 4 (June 1991), 37-44. Google ScholarDigital Library
- 5. John H. Fraser and David R. Kaeli: Operating System Impact on Cache Performance. Manuscript, Northeastern University, Boston, 25 pages, 1996.Google Scholar
- 6. Andrew S. Huang and John Paul Shen: The Intrinsic Bandwidth Requirements of Ordinary Programs. 7th Int. Conf. on Architectural Support for Programming Languages and Operating Systems, = ACM SIGPLAN Notices 31, 9 (Sept. 1996), 105-115. Google ScholarDigital Library
- 7. Nikki Mirghafori, Margret Jacoby, and David Patterson: Truth in SPEC Benchmarks. Computer Architecture News 23, 5 (Dec. 1995), 34-42. Google ScholarDigital Library
- 8. David A. Patterson and John L. Hennessy: Computer Architecture. A Quantitative Approach. Morgan Kaufmann, San Francisco 1996 (2nd edition). Google ScholarDigital Library
- 9. Jeff Reilly: SPEC Describes SPEC95 Products and Benchmarks. SPEC Newsletter 7, 3 (Sept. 1995), 4-6. Also in http://www.specbench.org/osg/cpu95/news/cpu95descr.htmlGoogle Scholar
- 10. Stuart Sechrest, Chih-Chieh Lee, and Trevor Mudge: Correlation and Aliasing in Dynamic Branch Prediction. 23rd Ann. Symp. on Computer Architecture, = Computer Architecture News 24, 2 (May 1996), 22-32. Google ScholarDigital Library
- 11. {SPEC} SPECcpu98 Development (Subpage in SPEC's World Wide Web page), http: //www.specbench.org/osg/cpu98, Dec. 1996.Google Scholar
- 12. Reinhold Weicker: An Example of Benchmark Obsolescence: 023.eqntott. SPEC Newsletter 7, 4 (Dec. 1995), 5-6. Also in http://www.specbench.org/osg/cpu95/news /eqntott.htmlGoogle Scholar
- 13. Steven Cameron Woo et al.: The SPLASH-2 Programs: Characterization and Methodological Considerations. 22nd Ann. Symp. on Computer Architecture, = Computer Architecture News 23, 2 (May 1995), 24-36. Google ScholarDigital Library
Index Terms
- On the use of SPEC benchmarks in computer architecture research
Recommendations
Truth in SPEC benchmarks
The System Performance Evaluation Cooperative (SPEC) benchmarks are a set of integer and floating-point programs that are intended to be “effective and fair in comparing the performance of high performance computing systems”. SPEC ratings are often ...
Evolution and evaluation of SPEC benchmarks
We present a method for quantitative evaluation of SPEC benchmarks. The method is used for the analysis of three generations of SPEC component-level benchmarks: SPEC89, SPEC92, and SPEC95. Our approach is suitable for studying (1) the redundancy between ...
Large System Performance of SPEC OMP2001 Benchmarks
ISHPC '02: Proceedings of the 4th International Symposium on High Performance ComputingPerformance characteristics of application programs on large-scale systems are often significantly different from those on smaller systems. SPEC OMP2001 is a benchmark suite intended for measuring performance of modern shared memory parallel systems. ...
Comments