Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Copernicus, a hybrid dataflow and peer-to-peer scientific computing platform for efficient large-scale ensemble sampling
KTH, Centres, SeRC - Swedish e-Science Research Centre.
KTH, Centres, SeRC - Swedish e-Science Research Centre.
KTH, School of Engineering Sciences (SCI), Physics, Theoretical & Computational Biophysics. KTH, Centres, SeRC - Swedish e-Science Research Centre.ORCID iD: 0000-0002-2734-2794
2017 (English)In: Future generations computer systems, ISSN 0167-739X, E-ISSN 1872-7115, Vol. 71, p. 18-31Article in journal (Refereed) Published
Abstract [en]

Compute-intensive applications have gradually changed focus from massively parallel supercomputers to capacity as a resource obtained on-demand. This is particularly true for the large-scale adoption of cloud computing and MapReduce in industry, while it has been difficult for traditional high-performance computing (HPC) usage in scientific and engineering computing to exploit this type of resources. However, with the strong trend of increasing parallelism rather than faster processors, a growing number of applications target parallelism already on the algorithm level with loosely coupled approaches based on sampling and ensembles. While these cannot trivially be formulated as MapReduce, they are highly amenable to throughput computing. There are many general and powerful frameworks, but in particular for sampling-based algorithms in scientific computing there are some clear advantages from having a platform and scheduler that are highly aware of the underlying physical problem. Here, we present how these challenges are addressed with combinations of dataflow programming, peer-to-peer techniques and peer-to-peer networks in the Copernicus platform. This allows automation of sampling-focused workflows, task generation, dependency tracking, and not least distributing these to a diverse set of compute resources ranging from supercomputers to clouds and distributed computing (across firewalls and fragile networks). Workflows are defined from modules using existing programs, which makes them reusable without programming requirements. The system achieves resiliency by handling node failures transparently with minimal loss of computing time due to checkpointing, and a single server can manage hundreds of thousands of cores e.g. for computational chemistry applications.

Place, publisher, year, edition, pages
Elsevier, 2017. Vol. 71, p. 18-31
Keywords [en]
Dataflow programming, Distributed computing, Job resiliency, Peer-too-peer, Scientific computing
National Category
Other Computer and Information Science
Identifiers
URN: urn:nbn:se:kth:diva-202024DOI: 10.1016/j.future.2016.11.004ISI: 000396950100002Scopus ID: 2-s2.0-85009875682OAI: oai:DiVA.org:kth-202024DiVA, id: diva2:1076515
Funder
Swedish Research Council, 2010-491 2010-5107Swedish eā€Science Research CenterEU, Horizon 2020, 675728
Note

QC 20170223

Available from: 2017-02-23 Created: 2017-02-23 Last updated: 2018-02-28Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Pouya, ImanPronk, SanderLindahl, Erik R.
By organisation
SeRC - Swedish e-Science Research CentreTheoretical & Computational Biophysics
In the same journal
Future generations computer systems
Other Computer and Information Science

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 179 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf