Sergei Shudler
Michael E. Papka
ABSTRACT
In-situ data analysis and visualization is a promising technique to handle the enormous amount of data an extreme-scale application produces. One challenge users often face in adopting in-situ techniques is setting the right environment on a target machine. Platforms such as SENSEI require complex software stacks that consist of various analysis packages and visualization applications. The user has to make sure all these prerequisites exist on the target machine, which often involves compiling and setting them up from scratch. In this paper, we leverage the containers technology (e.g., light-weight virtualization images) and provide users with Singularity containers that encapsulate ready-to-use, movable in-situ software stacks. Moreover, we make use of Spack to ease the process of creating these containers. Finally, we evaluate this solution by running in-situ analysis from within a container on an HPC system.
- Utkarsh Ayachit, Andrew Bauer, Earl P. N. Duque, Greg Eisenhauer, Nicola Ferner, Junmin Gu, Kenneth E. Jansen, Burlen Loring, Zarija Lukic, Suresh Menon, Dmitriy Morozov, Patrick O'Leary, Reetesh Ranjan, Michel Rasquin, Christopher P. Stone, Venkat Vishwanath, Gunther H. Weber, Brad Whitlock, Matthew Wolf, K. John Wu, and E. Wes Bethel. 2016. Performance Analysis, Design Considerations, and Applications of Extreme-scale in Situ Infrastructures. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '16). IEEE Press, Piscataway, NJ, USA, Article 79, 12 pages.Google Scholar
- Utkarsh Ayachit, Brad Whitlock, Matthew Wolf, Burlen Loring, Berk Geveci, David Lonie, and E. Wes Bethel. 2016. The SENSEI Generic in Situ Interface. In Proceedings of the 2nd Workshop on In Situ Infrastructures for Enabling Extreme-scale Analysis and Visualization (ISAV'16). IEEE Press, Piscataway, NJ, USA, 40--44. Google ScholarCross Ref
- A. C. Bauer, H. Abbasi, J. Ahrens, H. Childs, B. Geveci, S. Klasky, K. Moreland, P. O'Leary, V. Vishwanath, B. Whitlock, and E. W. Bethel. 2016. In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms. Computer Graphics Forum 35, 3 (2016), 577--597. Google ScholarDigital Library
- Cooley - Argonne Leadership Computing Facility. [n. d.]. Retrieved July 19, 2019 from https://www.alcf.anl.gov/user-guides/cooleyGoogle Scholar
- Gregory M. Kurtzer et al. 2015. Singularity. Retrieved July 8, 2019 from https://sylabs.io/singularityGoogle Scholar
- Todd Gamblin et al. 2015. Spack - A flexible package manager that supports multiple versions, configurations, platforms, and compilers. Retrieved July 8, 2019 from https://spack.ioGoogle Scholar
- Todd Gamblin, Matthew LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and Scott Futral. 2015. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Proc. of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '15). ACM, Article 40, 12 pages.Google ScholarDigital Library
- Gregory M. Kurtzer, Vanessa Sochat, and Michael W. Bauer. 2017. Singularity: Scientific containers for mobility of compute. PLOS ONE 12, 5 (05 2017), 1--20.Google Scholar
- Matthew Larsen, James Ahrens, Utkarsh Ayachit, Eric Brugger, Hank Childs, Berk Geveci, and Cyrus Harrison. 2017. The ALPINE In Situ Infrastructure: Ascending from the Ashes of Strawman. In Proceedings of the 3rd Workshop In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization (ISAV'17). ACM, New York, NY, USA, 42--46. Google ScholarDigital Library
- Emily Le and David Paz. 2017. Performance Analysis of Applications Using Singularity Container on SDSC Comet. In Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact (PEARC17). ACM, Article 66, 4 pages. Google ScholarDigital Library
- Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, Manish Parashar, Nagiza Samatova, Karsten Schwan, Arie Shoshani, Matthew Wolf, Kesheng Wu, and Weikuan Yu. 2014. Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience 26, 7 (2014), 1453--1473. Google ScholarDigital Library
- ParaView/Catalyst. [n.d.]. Retrieved July 19, 2019 from https://www.paraview.org/in-situGoogle Scholar
- SENSEI - Scalable in situ analysis and visualization. 2015. Retrieved July 19, 2019 from https://sensei-insitu.orgGoogle Scholar
- Visit. [n.d.]. Retrieved July 19, 2019 from https://visit.llnl.govGoogle Scholar
- Brad Whitlock, Jean M. Favre, and Jeremy S. Meredith. 2011. Parallel in Situ Coupling of Simulation with a Fully Featured Visualization System. In Proceedings of the 11th Eurographics Conference on Parallel Graphics and Visualization (EGPGV '11). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 101--109. Google ScholarCross Ref
- A. J. Younge, K. Pedretti, R. E. Grant, and R. Brightwell. 2017. A Tale of Two Systems: Using Containers to Deploy HPC Applications on Supercomputers and Clouds. In 2017 IEEE International Conference on Cloud Computing Technology and Science (CloudCom). 74--81. Google ScholarCross Ref
Index Terms
- Spack meets singularity: creating movable in-situ analysis stacks with ease
Recommendations
Traditional High-Performance Computing with Container Technology (THPC): HPC using Container Technology, EasyBuild, Spack, and IBM LSF scheduler
PEARC '23: Practice and Experience in Advanced Research ComputingHigh-performance Computing (HPC) has been around for decades but maintaining software for HPC with heterogeneous compute nodes remains a challenging task for system administrators. Many software package frameworks have been developed over the years to ...
Integrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker
VTDC '15: Proceedings of the 8th International Workshop on Virtualization Technologies in Distributed ComputingWorkflows are a widely used abstraction for representing large scientific applications and executing them on distributed systems such as clusters, clouds, and grids. However, workflow systems have been largely silent on the question of precisely what ...
Survey of adaptive containerization architectures for HPC
SC-W '23: Proceedings of the SC '23 Workshops of The International Conference on High Performance Computing, Network, Storage, and AnalysisContainers offer an array of advantages that benefit research reproducibility and portability. As container tools mature, container security improves, and high-performance computing (HPC) and cloud system tools converge, supercomputing centers are ...
Comments