How to Build a Benchmark

ABSTRACT

Standardized benchmarks have become widely accepted tools for the comparison of products and evaluation of methodologies. These benchmarks are created by consortia like SPEC and TPC under confidentiality agreements which provide little opportunity for outside observers to get a look at the processes and concerns that are prevalent in benchmark development. This paper introduces the primary concerns of benchmark development from the perspectives of SPEC and TPC committees. We provide a benchmark definition, outline the types of benchmarks, and explain the characteristics of a good benchmark. We focus on the characteristics important for a standardized benchmark, as created by the SPEC and TPC consortia. To this end, we specify the primary criteria to be employed for benchmark design and workload selection. We use multiple standardized benchmarks as examples to demonstrate how these criteria are ensured.

References

1. R. García-Castro and A. Gómez-Pérez. Benchmark Suites for Improving the RDF(S) Importers and Exporters of Ontology Development Tools. In Y. Sure and J. Domingue, editors, The Semantic Web: Research and Applications, volume 4011 of Lecture Notes in Computer Science, pages 155--169. Springer Berlin Heidelberg, 2006.
2. J. Gustafson and Q. Snell. HINT: A new way to measure computer performance. In System Sciences, 1995. Proceedings of the Twenty-Eighth Hawaii International Conference on, volume 2, pages 392--401 vol.2, Jan 1995.
3. J. L. Henning. SPEC CPU2000: measuring CPU performance in the New Millennium. Computer, 33(7):28--35, Jul 2000.
4. K. Huppler. The Art of Building a Good Benchmark. In R. Nambiar and M. Poess, editors, Performance Evaluation and Benchmarking, volume 5895 of Lecture Notes in Computer Science, pages 18--30. Springer Berlin Heidelberg, 2009.
5. K. Huppler and D. Johnson. TPC Express - A New Path for TPC Benchmarks. In R. Nambiar and M. Poess, editors, Performance Characterization and Benchmarking, volume 8391 of Lecture Notes in Computer Science, pages 48--60. Springer International Publishing, 2014.
6. K. Sachs. Performance Modeling and Benchmarking of Event-Based Systems. PhD thesis, TU Darmstadt, 2010. SPEC Distinguished Dissertation Award 2011.
7. S. E. Sim, S. Easterbrook, and R. C. Holt. Using Benchmarking to Advance Research: A Challenge to Software Engineering. In Proceedings of the 25th International Conference on Software Engineering, ICSE '03, pages 74--83, Washington, DC, USA, 2003. IEEE Computer Society.
8. K. Skadron, M. Martonosi, D. I. August, M. D. Hill, D. J. Lilja, and V. S. Pai. Challenges in Computer Architecture Evaluation. Computer, 36(8):30--36, Aug. 2003.
9. Standard Performance Evaluation Corporation. SPEC fair use rule. http://www.spec.org/fairuse.html.
10. Standard Performance Evaluation Corporation. SPEC Power and Performance Benchmark Methodology. http://spec.org/power/docs/SPEC-Power_and_Performance_Methodology.pdf.
11. F. Stefani, A. Moschitta, D. Macii, and D. Petri. FFT benchmarking for digital signal processing technologies. In 17th IMEKO World Congress, 2003.
12. M. Vieira, H. Madeira, K. Sachs, and S. Kounev. Resilience Benchmarking. In K. Wolter, A. Avritzer, M. Vieira, and A. van Moorsel, editors, Resilience Assessment and Evaluation of Computing Systems, XVIII. Springer-Verlag, Berlin, Heidelberg, 2012. ISBN: 978-3-642-29031-2.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Performance