Towards efficient supercomputing: searching for the right efficiency metric

ABSTRACT

Efficiency in supercomputing has traditionally focused on execution time. In early 2000's, the concept of total cost of ownership was re-introduced, with the introduction of efficiency measure to include aspects such as energy and space. Yet the supercomputing community has never agreed upon a metric that can cover these aspects completely and also provide a fair basis for comparison. This paper examines the metrics that have been proposed in the past decade, and proposes a vector-valued metric for efficient supercomputing. Using this metric, the paper presents a study of where the supercomputing industry has been and where it stands today with respect to efficient supercomputing.

References

1. L. Barroso and U. Hölzle. The case for energy-proportional computing. IEEE Computer, 40(12):33--37, Dec. 2007.
2. C. Bekas and A. Curioni. A new energy aware performance metric. In International Conference on Energy-Aware High Performance Computing, Sept. 2010.
3. K. D. Bois, T. Schaeps, S. Polfliet, F. Ryckbosch, and L. Eeckhout. SWEEP: Evaluating computer system energy efficiency using synthetic workloads. In International Conference on High Performance Embedded Architectures and Compilers, Jan. 2011.
4. D. Brooks, P. Bose, S. Schuster, H. Jacobson, P. Kudva, A. Buyuktosunoglu, J.-D. Wellman, V. Zyuban, M. Gupta, and P. Cook. Power-aware microarchitecture: Design and modeling challenges for the next generation microprocessors. IEEE Micro, 20(6):26--44, November/December 2000.
5. T. Conte and W. Hwu. Advances in benchmarking techniques: New standards and quantitative metrics. Advances in Computers, 41:231--253, 1995.
6. J. Dongarra, P. Luszczek, and A. Petitet. The LINPACK benchmark: Past, present, and future. Concurrency and Computation: Practice and Experience, 15(9):803--820, Aug. 2003.
7. Energy Efficient High Performance Computing Working Group. http://eehpcwg.lbl.gov.
8. A. Fanara, E. Haines, and A. Howard. The state of energy and performance benchmarking for enterprise servers. In TPC Technology Conference on Performance Evaluation and Benchmarking, Aug. 2009.
9. R. Ge, X. Feng, and K. Cameron. Improvement of power-performance efficiency for high-end computing. In Workshop on High-Performance, Power-Aware Computing, Apr. 2005.
10. R. Gonzalez and M. Horowitz. Energy dissipation in general purpose microprocessors. IEEE Journal of Solid-State Circuits, 31(9):1277--1284, Sept. 1996.
11. The Graph 500 list. http://www.graph500.org.
12. L. Gray, A. Kumar, and H. Li. Workload characterization of the SPECpower_ssj2008 benchmark. In SPEC International Performance Evaluation Workshop, June 2008.
13. The Green500 list: Environmentally responsible supercomputing. http://www.green500.org.
14. D. Hackenberg, R. Schöne, D. Molka, M. Müller, and A. Knüpfer. Quantifying power consumption variations of HPC systems using SPEC MPI benchmarks. In International Conference on Energy-Aware High Performance Computing, Sept. 2010.
15. HPC Challenge benchmark. http://icl.cs.utk.edu/hpcc/.
16. C. Hsu, W. Feng, and J. Archuleta. Towards efficient supercomputing: A quest for the right metric. In Workshop on High-Performance, Power-Aware Computing, Apr. 2005.
17. J. Koomey, S. Berard, M. Sanchez, and H. Wong. Implications of historical trends in the electrical efficiency of computing. IEEE Annals of the History of Computing, 33(3):46--54, July-September 2011.
18. K.-D. Lange and M. Tricker. The design and development of the server efficiency rating tool (SERT). In International Conference on Performance Engineering, Mar. 2011.
19. T. Makimoto, K. Eguchi, and M. Yoneyama. The cooler the better: New directions in the nomadic age. IEEE Computer, 34(4):38--42, Apr. 2001.
20. A. Martin. Towards an energy complexity of computation. Information Processing Letters, 77:181--187, Feb. 2001.
21. D. Meisner and T. Wenisch. Does low-power design imply energy efficiency for data centers. In International Symposium on Low Power Electronics and Design, Aug. 2011.
22. S. Microsystems. The SWaP (Space, Watts and Performance) metric, 2007.
23. D. Molka, D. Hackenberg, R. Schöne, T. Minartz, and W. Nagel. Flexible workload generation for HPC cluster efficiency benchmarking. In International Conference on Energy-Aware High Performance Computing, Sept. 2011.
24. T. Mudge and U. Hölzle. Challenges and opportunities for extremely energy-efficient processors. IEEE Micro, 30(4):20--24, July/August 2010.
25. M. Müller, M. van Waveren, R. Lieberman, B. Whitney, H. Saito, K. Kumaran, J. Baron, W. Brantley, C. Parrott, T. Elken, H. Feng, and C. Ponder. SPEC MPI2007:an application benchmark suite for parallel systems using MPI. Concurrency and Computation: Practice and Experience, 22(2):191--205, Feb. 2010.
26. V. Pel. Energy efficiency aspects in Cray supercomputers. In International Conference on Energy-Aware High Performance Computing, Sept. 2010.
27. P. Pénzes and A. Martin. Energy-delay efficiency of VLSI computations. In ACM Great Lakes Symposium on VLSI, Apr. 2002.
28. M. Poess, R. Nambiar, and K. Vaid. Optimizing benchmark configurations for energy efficiency. In International Conference on Performance Engineering, Mar. 2011.
29. M. Poess, R. Nambiar, K. Vaid, J. J. M. Stephens, K. Huppler, and E. Haines. Energy benchmarks: A detailed analysis. In International Conference on Energy-Efficient Computing and Networking, Apr. 2010.
30. S. Rivoire, M. Shah, P. Ranganathan, and C. Kozyrakis. JouleSort: A balanced energy-efficiency benchmark. In International Conference on Management of Data, June 2007.
31. F. Ryckbosch, S. Polfliet, and L. Eeckhout. Trend in server energy-proportionality. IEEE Computer, 44(9):69--72, Sept. 2011.
32. V. Sarkar. ExaScale software study: Software challenges in extreme scale systems. DARPA IPTO Report, Sept. 2009.
33. D. Schall, V. Hoefner, and M. Kern. Towards an enhanced benchmark advocating energy-efficient systems. In TPC Technology Conference on Performance Evaluation & Benchmarking, Aug. 2011.
34. T. Scogland, B. Subramaniam, and W. Feng. Emerging trends on the evolving Green500: Year three. In Workshop on High-Performance, Power-Aware Computing, May 2011.
35. SPECpower_ssj2008 Results. http://www.spec.org/power_ssj2008/results/.
36. E. Strohmaier. Generalized utility metrics for supercomputers. June 2009.
37. The Green Grid. The Green Grid power efficiency metrics: PUE and DCiE, 2007.
38. The Green Grid. Recommendations for measuring and reporting overall data center efficiency (version 2), May 2011.
39. TOP500 supercomputing sites. http://www.top500.org.
40. G. Varsamopoulos and S. Gupta. Energy proportionality and the future: Metrics and directions. In International Workshop on Green Computing, Sept. 2010.

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Measurement

    • Metrics

    • Performance

• Social and professional topics

  • Professional topics

    • Management of computing and information systems

      • Software management

        • Software selection and adaptation

• Software and its engineering

  • Software organization and properties

    • Extra-functional properties

      • Software performance