Enabling callback-driven runtime introspection via MPI_T

ABSTRACT

Understanding the behavior of parallel applications that use the Message Passing Interface (MPI) is critical for optimizing communication performance. Performance tools for MPI currently rely on the PMPI Profiling Interface or the MPI Tools Information Interface, MPI_T, for portably collecting information for performance measurement and analysis. While tools using these interfaces have proven to be extremely valuable for performance tuning, these interfaces only provide synchronous information, i.e., when an MPI or an MPI_T function is called. There is currently no option for collecting information about asynchronous events from within the MPI library. In this work we propose a callback-driven interface for event notification from MPI implementations. Our approach is integrated in the existing MPI_T interface and provides a portable API for tools to discover and register for events of interest. We demonstrate the functionality and usability of the interface with a prototype implementation in Open MPI, a small logging tool (MEL) and the measurement infrastructure Score-P.

References

1. Dan Bonachea. 2006. GASNet Specification. Technical Report UCB/CSD-02-1207. Lawrence Berkeley National Laboratory.
2. Ron Brightwell, Sue Goudy, and Keith Underwood. 2005. A Preliminary Analysis of the MPI Queue Characteristics of Several Applications. In Proceedings of the 2005 International Conference on Parallel Processing (ICPP '05). IEEE, 175--183.
3. Kevin A Brown, Jens Domke, and Satoshi Matsuoka. 2014. Tracing Data Movements Within MPI Collectives. In Proc. 21st Eur. MPI Users' Gr. Meet. (EuroMPI/ASIA '14). ACM, New York, NY, USA, 117:117---117:118.
4. Kevin A Brown, Jens Domke, and Satoshi Matsuoka. 2015. Hardware-Centric Analysis of Network Performance for MPI Applications. In 2015 IEEE 21st Int. Conf. Parallel Distrib. Syst. 692--699.
5. Isaias Compres. {n. d.}. On-line Application-specific Tuning with the Periscope Tuning Framework and the MPI Tools Interface. Presentation at the 2014 Petascale Tools Workshop, Madison, WI, August 2014. ({n. d.}).
6. Rossen Dimitrov, Anthony Skjellum, Terry Jones, Bronis de Supinski, Ron Brightwell, Curtis Janssen, and MaryDell Nochumson. 2002. PERUSE: An MPI Performance Revealing Extensions Interface. Sixth IBM System Scientific Computing User Group (2002).
7. Matthew G. F. Dosanjh, Taylor Groves, Ryan E. Grant, Ron Brightwell, and Patrick G. Bridges. 2016. RMA-MT: A Benchmark Suite for Assessing MPI Multithreaded RMA Performance. In 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid). 550--559.
8. Alexandre E Eichenberger, John Mellor-Crummey, Martin Schulz, Michael Wong, Nawal Copty, Robert Dietrich, Xu Liu, Eugene Loh, and Daniel Lorenz. 2013. OMPT: An OpenMP Tools Application Programming Interface for Performance Analysis. In OpenMP Era Low Power Devices Accel., Alistair P Rendell, Barbara M Chapman, and Matthias S Müller (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 171--185.
9. Dominic Eschweiler, Michael Wagner, Markus Geimer, Andreas Knüpfer, Wolfgang E. Nagel, and Felix Wolf. 2012. Open Trace Format 2: The next generation of scalable trace formats and support libraries. Adv. Parallel Comput. 22 (2012), 481--490.
10. Esthela Gallardo, Jerome Vienne, Leonardo Fialho, Patricia Teller, and James Browne. 2015. MPI Advisor: A Minimal Overhead Tool for MPI Library Performance Tuning. In Proc. 22nd Eur. MPI Users' Gr. Meet. (EuroMPI '15). ACM, New York, NY, USA, 6:1---6:10.
11. Esthela Gallardo, Jerome Vienne, Leonardo Fialho, Patricia Teller, and James Browne. 2017. Employing MPI_T in MPI Advisor to optimize application performance. The International Journal of High Performance Computing Applications 0, 0 (2017).
12. Markus Geimer, Felix Wolf, Brian J. N. Wylie, Erika Ábrahám, Daniel Becker, and Bernd Mohr. 2010. The Scalasca performance toolset architecture. Concurrency and Computation: Practice and Experience 22, 6 (April 2010), 702--719.
13. Samuel K. Gutierrez, Nathan T. Hjelm, Manjunath G. Venkata, and Richard L. Graham. 2012. Performance Evaluation of Open MPI on Cray XE/XK Systems. In 2012 IEEE 20th Annual Symposium on High-Performance Interconnects. 40--47.
14. Tanzima Islam, Kathryn Mohror, and Martin Schulz. 2014. Exploring the Capabilities of the New MPI_T Interface. In Proceedings of the 21st European MPI Users' Group Meeting (EuroMPI/ASIA '14).
15. Terry Jones, Brian W Barrett, David E. Bernholdt, Ron Brightwell, Lars Ailo Bongo, George Bosilca, Ana Cortés, Toni Cortés, James Coyle, Bronis R de Supinski, Rossen Dimitrov, Sevki Erdogon, Hans-Christian Hoppe, Graham Fagg, Ferdinand Geier, Judit Gimenez, Richard L. Graham, David Gunter, Steven T. Healey, Curtis Janssen, Karen L. Karavanic, Rainer Keller, Bernie King-Smith, Darren J. Kerbyson, Jesús Labarta, Brian LePore, Andrew Lumsdaine, Chee Wai Lee, Ewing L. Lusk, Dave Merril, Bernd Mohr, Kathryn Mohror, Matthias S. Müller, Beth Noble, Robert W. Numrich, Patrick Ohly, Dhabaleswar K. Panda, Kurt Pinnow, Kumaran Pajaram, Hubert Ritzdorf, Philip C. Roth, Martin Schulz, Miquel Senar, Anthony Skjellum, Jeff Squyres, Richard Treumann, and Tim Woodall. 2006. MPI PERUSE: An MPI Extension for Revealing Unexposed Implementation Information. Technical Report. LLNL.
16. Rainer Keller, George Bosilca, Graham Fagg, Michael Resch, and Jack J. Dongarra. 2006. Implementation and Usage of the PERUSE-Interface in Open MPI. In Recent Adv. Parallel Virtual Mach. Messag. Passing Interface, Bernd Mohr, Jesper Larsson Träff, Joachim Worringen, and Jack Dongarra (Eds.). LNCS, Vol. 4192. Springer Berlin Heidelberg, 347--355.
17. Rainer Keller and Richard L Graham. 2010. Characteristics of the Unexpected Message Queue of MPI Applications. In Recent Adv. Messag. Passing Interface, Rainer Keller, Edgar Gabriel, Michael Resch, and Jack Dongarra (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 179--188.
18. Andreas Knüpfer, Holger Brunst, Jens Doleschal, Matthias Jurenz, Matthias Lieber, Holger Mickler, Matthias S Müller, and Wolfgang E Nagel. 2008. The Vampir performance analysis toolset. In Tools for High Perf. Comp. Springer, 139--155.
19. Andreas Knüpfer, Christian Rössel, Dieter an Mey, Scott Biersdorff, Kai Diethelm, Dominic Eschweiler, Markus Geimer, Michael Gerndt, Daniel Lorenz, Allen D. Malony, Wolfgang E. Nagel, Yury Oleynik, Peter Philippen, Pavel Saviankou, Dirk Schmidl, Sameer S Shende, Ronny Tschüter, Michael Wagner, Bert Wesarg, and Felix Wolf. 2012. Score-P: A Joint Performance Measurement Run-Time Infrastructure for Periscope, Scalasca, TAU, and Vampir. In Tools High Perform. Comput. 2011, Holger Brunst, Matthias S Müller, Wolfgang E Nagel, and Michael M Resch (Eds.). Springer Berlin Heidelberg, 79--91.
20. Julian M. Kunkel, Yuichi Tsujita, Olga Mordvinova, and Thomas Ludwig. 2009. Tracing Internal Communication in MPI and MPI-I/O. In Int. Conf. on Parallel and Distrib. Comp., Applications and Technologies. IEEE, 280--286.
21. Adam Leko, Dan Bonachea, Hung-Hsun Su, and Alan D. George. 2006. GASP: A Performance Analysis Tool Interface for Global Address Space Programming Models. Technical Report LBNL-61606. Lawrence Berkeley National Lab. 1--12 pages.
22. Adam Leko, Hung-Hsun Su, Dan Bonachea, Bryan Golden, Max Billingsley III., and Alan D. George. 2006. Parallel performance wizard: a performance analysis tool for partitioned global-address-space programming models. In SC '06 Proc. 2006 ACM/IEEE Conf. Supercomput. ACM, New York, NY, USA, 186.
23. Matthias S. Müller, Matthijs van Waveren, Ron Lieberman, Brian Whitney, Hideki Saito, Kalyan Kumaran, John Baron, William C. Brantley, Chris Parrott, Tom Elken, Huiyu Feng, and Carl Ponder. 2007. SPEC MPI2007 -- an application benchmark suite for parallel systems using MPI. Concurrency and Computation: Practice and Experience 22, 2 (2007), 191--205.
24. OpenMP Architecture Review Board. 2015. OpenMP 4.5 Specification.
25. Raghunath Rajachandrasekar, Jonathan Perkins, Khaled Hamidouche, Mark Arnold, and Dhabaleswar K. Panda. 2014. Understanding the Memory-Utilization of MPI Libraries: Challenges and Designs in Implementing the MPI_T Interface. In Proc. of the 21st European MPI Users' Group Meeting (EuroMPI/ASIA '14).
26. Srinivasan Ramesh, Aurèle Mahéo, Sameer Shende, Allen D. Malony, Hari Subramoni, and Dhabaleswar K Panda. 2017. MPI Performance Engineering with the MPI Tool Interface: The Integration of MVAPICH and TAU. In Proceedings of the 24th European MPI Users' Group Meeting (EuroMPI '17).
27. Hung-Hsun Su. 2010. Parallel Performance Wizard: Framework and Techniques for Parallel Application Optimization. Ph.D. Dissertation. University of Florida.
28. The Message Passing Interface Forum. 2015. MPI: A Message Passing Interface Standard, Version 3.1.
29. UPC Consortium. 2013. UPC Language Specifications. (Nov. 2013).

Index Terms

• General and reference

  • Cross-computing tools and techniques

    • Measurement

    • Performance

• Software and its engineering

  • Software notations and tools

    • Software libraries and repositories