Benchmark semantics

ABSTRACT

Be careful when interpreting benchmark measurements that compare two languages or two implementations of the same language. A program expressed in two different languages rarely computes the exact same function in both cases. The same goes for a program run on two different implementations of the same language. The implementation details ultimately affect the language semantics as well as the benchmark performance. Here are some simple examples of this effect.

REFERENCES

• 1: Doug Cooper, Standard Pascal: User Reference Manual, W. W. Norton & Co., Inc., New York, NY, 1983
• 2: [2] G. Birtwistle, L. Enderin, M. Ohlin, and J. Palme, DECSYSTEM-10 SIMULA Language Handbook, 1977 Swedish National Defense Institute.
• 3: [3] C. W. Gear, PL/I and PL/C Language Manual Science Research Assoc, Inc. 1978.
• 4: Guy L. Steele, Jr., Common LISP: the language, Digital Press, Newton, MA, 1984
• 5: Adele Goldberg , David Robson, Smalltalk-80: the language and its implementation, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1983
• 6: Richard P. Gabriel, Performance and evaluation of LISP systems, Massachusetts Institute of Technology, Cambridge, MA, 1985
• 7: R. A. O'Keefe, Prolog compared with LISP?, ACM SIGPLAN Notices, v.18 n.5, p.46-56, May 1983 [doi>10.1145/948249.948255]

INDEX TERMS

Primary Classification:
  D. Software
  D.3 PROGRAMMING LANGUAGES
      D.3.0 General
          Subjects: Standards

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.8 Metrics
          Subjects: Performance measures

General Terms:
Languages, Measurement, Standardization