Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Hybrid Resource Management for HPC and Data Intensive Workloads
Umea Univ, Dept Comp Sci, Umea, Sweden..
KTH, School of Electrical Engineering and Computer Science (EECS), Centres, Centre for High Performance Computing, PDC.ORCID iD: 0000-0001-7604-2969
KTH, School of Electrical Engineering and Computer Science (EECS), Centres, Centre for High Performance Computing, PDC.ORCID iD: 0000-0002-9901-9857
Umea Univ, Dept Comp Sci, Umea, Sweden..
2019 (English)In: 2019 19TH IEEE/ACM INTERNATIONAL SYMPOSIUM ON CLUSTER, CLOUD AND GRID COMPUTING (CCGRID), IEEE , 2019, p. 399-409Conference paper, Published paper (Refereed)
Abstract [en]

High Performance Computing (HPC) and Data Intensive (DI) workloads have been executed on separate clusters using different tools for resource and application management. With increasing convergence, where modern applications are composed of both types of jobs in complex workflows, this separation becomes a growing overhead and the need for a common platform increases. Executing both workload classes on the same clusters not only enables hybrid workflows, but can also increase system efficiency, as available hardware often is not fully utilized by applications. While HPC systems are typically managed in a coarse grained fashion, with exclusive resource allocations, DI systems employ a finer grained regime, enabling dynamic allocation and control based on application needs. On the path to full convergence, a useful and less intrusive step is a hybrid resource management system allowing the execution of DI applications on top of standard HPC scheduling systems. In this paper we present the architecture of a hybrid system enabling dual-level scheduling for DI jobs in HPC infrastructures. Our system takes advantage of real-time resource profiling to efficiently co-schedule HPC and DI applications. The architecture is easily extensible to current and new types of distributed applications, allowing efficient combination of hybrid workloads on HPC resources with increased job throughput and higher overall resource utilization. The implementation is based on the Sturm and Mesos resource managers for HPC and DI jobs. Experimental evaluations in a real cluster based on a set of representative HPC and DI applications demonstrate that our hybrid architecture improves resource utilization by 20%, with 12% decrease on queue makespan while still meeting all deadlines for HPC jobs.

Place, publisher, year, edition, pages
IEEE , 2019. p. 399-409
Series
IEEE-ACM International Symposium on Cluster Cloud and Grid Computing, ISSN 2376-4414
Keywords [en]
Resource Management, High Performance Computing, Data Intensive Computing, Mesos, Sturm, Bootstrapping
National Category
Computational Mathematics
Identifiers
URN: urn:nbn:se:kth:diva-260219DOI: 10.1109/CCGRID.2019.000.54ISI: 000483058700045Scopus ID: 2-s2.0-85069469164ISBN: 978-1-7281-0912-1 (print)OAI: oai:DiVA.org:kth-260219DiVA, id: diva2:1355726
Conference
19th Annual IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGRID), MAY 14-17, 2019, Larnaca, CYPRUS
Note

QC 20190930

Available from: 2019-09-30 Created: 2019-09-30 Last updated: 2019-09-30Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records BETA

Laure, Erwin

Search in DiVA

By author/editor
Rezaei, MohamadLaure, Erwin
By organisation
Centre for High Performance Computing, PDC
Computational Mathematics

Search outside of DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric score

doi
isbn
urn-nbn
Total: 3 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf