Relative factors in performance analysis of Java virtual machines

ABSTRACT

Many new Java runtime optimizations report relatively small, single-digit performance improvements. On modern virtual and actual hardware, however, the performance impact of an optimization can be influenced by a variety of factors in the underlying systems. Using a case study of a new garbage collection optimization in two different Java virtual machines, we show the relative effects of issues that must be taken into consideration when claiming an improvement. We examine the specific and overall performance changes due to our optimization and show how unintended side-effects can contribute to, and distort the final assessment. Our experience shows that VM and hardware concerns can generate variances of up to 9.5% in whole program execution time. Consideration of these confounding effects is critical to a good, objective understanding of Java performance and optimization.

REFERENCES

• 1: B. Alpern, C. R. Attanasio, A. Cocchi, D. Lieber, S. Smith, T. Ngo, J. J. Barton, S. F. Hummel, J. C. Sheperd, and M. Mergen. Implementing Jalapeño in Java. In OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, pages 314--324, Oct. 1999. ISBN1-58113-238-7.
• 2: J. M. Anderson, L. M. Berc, J. Dean, S. Ghemawat, M. R. Henzinger, S.-T. A. Leung, R. L. Sites, M. T. Vandevoorde, C. A. Waldspurger, and W. E. Weihl. Continuous profiling: where have all the cycles gone? ACM Trans. Comput. Syst., 15(4):357--390, Nov. 1997. ISSN0734-2071.
• 3: R. Berrendorf, H. Ziegler, and B. Mohr. PCL-the performance counter library. http://www.fz-juelich.de/zam/PCL/.
• 4: S. M. Blackburn, P. Cheng, and K. S. McKinley. Myths and realities: The performance impact of garbage collection. In Proceedings of the ACM SIGMETRICS Conference on Measurement & Modeling Computer Systems, pages 25--36, June 2004.
• 5: S. M. Blackburn, R. Jones, K. S. McKinley, and J. E. B. Moss. Beltway: getting around garbage collection gridlock. SIGPLAN Not., 37(5):153--164, June 2002. ISSN0362-1340.
• 6: H.-J. Boehm. Reducing garbage collector cache misses. In ISMM '00: Proceedings of the 2nd international symposium on Memory management, pages 59--64, Oct. 2000. ISBN1-58113-263-8.
• 7: S. Brown, J. Dongarra, N. Garner, K. London, and P. Mucci. PAPI. http://icl.cs.utk.edu/papi.
• 8: C.-Y. Cher, A. L. Hosking, and T. N. Vijaykumar. Software prefetching for mark-sweep garbage collection: hardware analysis and software redesign. In ASPLOS-XI: Proceedings of the 11th international conference on Architectural support for programming languages and operating systems, pages 199--210, Oct. 2004. ISBN1-58113-804-0.
• 9: E. Duesterwald, C. Cascaval, and S. Dwarkadas. Characterizing and predicting program behavior and its variability. In PACT '03: Proceedings of the 12th International Conference on Parallel Architectures and Compilation Techniques, page 220. IEEE Computer Society, Sep. 2003. ISBN0-7695-2021-9.
• 10: B. Dufour, K. Driesen, L. Hendren, and C. Verbrugge. Dynamic metrics for Java. In Proceedings of the ACM SIGPLAN 2003 Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA), pages 149--168, Oct. 2003. ISBN1-58113-712-5.
• 11: L. Eeckhout, A. Georges, and K. De Bosschere. How Java programs interact with virtual machines at the microarchitectural level. In Proceedings of the 18th ACM SIGPLAN conference on Object-oriented programing, systems, languages, and applications, pages 169--186, Oct. 2003. ISBN1-58113-712-5.
• 12: E. M. Gagnon. SableVM. http://www.sablevm.org/.
• 13: E. M. Gagnon. A Portable Research Framework for the Execution of Java Bytecode. PhD thesis, McGill University, 2002.
• 14: E. M. Gagnon and L. J. Hendren. SableVM:A Research Framework for the Efficient Execution of Java Bytecode. In Proceedings of the Java Virtual Machine Research and Technology Symposium (JVM '01), pages 27--40. USENIX Association, Apr. 2001.
• 15: D. Gu, C. Verbrugge, and E. Gagnon. Assessing the impact of optimization in Java virtual machines. Technical Report SABLE-TR-2005-4, Sable Research Group, McGill University, Oct. 2005.
• 16: D. Gu, C. Verbrugge, and E. Gagnon. Code layout as a source of noise in JVM performance. Studia Informatica Universalis, 4(1):83--99, March 2005. ISBN2-912590-31-0.
• 17: S. Z. Guyer and K. S. McKinley. Finding your cronies: static analysis for dynamic object colocation. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications, pages 237--250, Oct. 2004. ISBN1-58113-831-9.
• 18: K. Hammond, G. L. Burn, and D. B. Howe. Spiking your caches. In J. T. O. Donnell and K. Hammond, editors, GLA, pages 58--68. Springer-Verlag, July 1993.
• 19: M. Hauswirth, P. F. Sweeney, A. Diwan, and M. Hind. Vertical profiling: understanding the behavior of object-priented applications. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications, pages 251--269, Oct. 2004. ISBN1-58113-831-9.
• 20: X. Huang, S. M. Blackburn, K. S. McKinley, J. E. B. Moss, Z. Wang, and P. Cheng. The garbage collection advantage: improving program locality. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN Conference on Object-oriented programming, systems, languages, and applications, pages 69--80, Oct. 2004. ISBN1-58113-831-9.
• 21: R. Jones and R. Lins. Garbage Collection: Algorithms for Automatic Dynamic Memory Management. John Wiley and Sons, Ltd, 1996.
• 22: J. Lau, J. Sampson, E. Perelman, G. Hamerly, and B. Calder. The strong correlation between code signatures and performance. In ISPASS '05: Proceedings of the IEEE International Symposium on Performance Analysis of Systems and Software, page 220. IEEE Computer Society, March 2005.
• 23: D. Project. The DaCapo benchmark suite (beta050224). http://www-ali.cs.umass.edu/DaCapo/index.html, 2003.
• 24: F. Qian and L. Hendren. An adaptive, region-based allocator for java. In ISMM '02: Proceedings of the 3rd international symposium on Memory management, pages 127--138, June 2002. ISBN1-58113-539-4.
• 25: R. M. Rabbah, H. Sandanagobalane, M. Ekpanyapong, and W.-F. Wong. Compiler orchestrated prefetching via speculation and predication. In ASPLOS-XI: Proceedings of the 11th international conference on Architectural support for programming languages and operating systems, pages 189--198, Oct. 2004. ISBN1-58113-804-0.
• 26: F. Schneider and T. R. Gross. Using platform-specific performance counters for dynamic compilation. In Proceedings of the 18th International Workshop on Languages and Compilers for Parallel Computing (LCPC'05), Oct 2005. to appear.
• 27: S. Soman, C. Krintz, and D. F. Bacon. Dynamic selection of application-specific garbage collectors. In ISMM '04: Proceedings of the 4th international symposium on Memory management, pages 49--60, Oct 2004. ISBN1-58113-945-4.
• 28: Standard Performance Evaluation Corporation. SPECjvm98 benchmarks. http://www.spec.org/osg/jvm98.
• 29: D. Stefanović, M. Hertz, S. M. Blackburn, K. S. McKinley, and J. E. B. Moss. Older-first garbage collection in practice: evaluation in a java virtual machine. In MSP '02: Proceedings of the 2002 workshop on Memory system performance, pages 25--36, June 2002.
• 30: D. Stefanović, K. S. McKinley, and J. E. B. Moss. Age-based garbage collection. In OOPSLA '99: Proceedings of the 14th ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, pages 370--381, Oct. 1999. ISBN1-58113-238-7.
• 31: P. F. Sweeney, M. Hauswirth, B. Cahoon, P. Cheng, A. Diwan, D. Grove, and M. Hind. Using hardware performance monitors to understand the behavior of Java applications. In VM'04:Proceedings of the 3rd Virtual Machine Research and Technology Symposium, May 2004.
• 32: D. Ungar. Generation scavenging: A non-disruptive high performance storage reclamation algorithm. In SDE 1: Proceedings of the first ACM SIGSOFT/SIGPLAN software engineering symposium on Practical software development environments, pages 157--167, Apr. 1984. ISBN0-89791-131-8.
• 33: X. Vera and J. Xue. Let's study whole program cache behavior analytical. In International Symposium on High-Performance Computer Architecture (HPCA 8) (IEEE), pages 175--186, Feb. 2002.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Measurement techniques

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Design studies

General Terms:
Experimentation, Measurement, Performance

Keywords:
Java, caches, garbage collection, hardware counters, performance analysis