Dynamic Resource Management for In-Situ Techniques Using MPI-SessionsShow others and affiliations
2025 (English)In: Recent advances in the message passing interface, EUROMPI 2024 / [ed] Blaas-Schenner, C Niethammer, C Haas, T, Springer Nature , 2025, p. 105-120Conference paper, Published paper (Refereed)
Abstract [en]
The computational power of High-Performance Computing (HPC) systems increases continuously and rapidly. Data-intensive applications are designed to leverage the high computational capacity of HPC resources and typically generate a large amount of data for traditional post-processing data analytics. However, the HPC systems' in-/output (IO) subsystem develops relatively slowly, and the storage capacity is limited. This could lead to limited actual performance and scientific discovery. In-situ techniques are a partial remedy to these problems by reducing or avoiding the data flow through the IO subsystem to/from the storage. However, in current practice, asynchronous in-situ techniques with static resource management often allocate separate computing resources for executing in-situ task(s), which remain idle if no in-situ work is at hand. In the present work, we target improving the efficiency of computing resource usage by launching and releasing necessary additional computing resources for in-situ task(s). Our approach is based on extensions for MPI Sessions that enable the required dynamic resource management. In this paper, we propose a basic and an advanced in-situ techniques with dynamic resource management enabled by MPI Sessions, their implementations on two real-world use cases, and a critical analysis of the experimental results.
Place, publisher, year, edition, pages
Springer Nature , 2025. p. 105-120
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 15267
Keywords [en]
In-situ, HPC, Dynamic resource management, MPI Session
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-357272DOI: 10.1007/978-3-031-73370-3_7ISI: 001329986700007Scopus ID: 2-s2.0-85206070581OAI: oai:DiVA.org:kth-357272DiVA, id: diva2:1918931
Conference
31st European MPI Users' Group Meeting (EuroMPI), September 25-27, 2024, Pawsey Supercomput Res Centre, Perth, Australia
Note
Part of ISBN 978-3-031-73369-7, 978-3-031-73370-3
QC 20241206
2024-12-062024-12-062024-12-06Bibliographically approved