kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Scalable Execution of Big Data Workflows using Software Containers
KTH.
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS.ORCID iD: 0000-0002-4722-0823
Show others and affiliations
2020 (English)In: Proceedings of the 12th International Conference on Management of Digital EcoSystems, MEDES 2020, Association for Computing Machinery, Inc , 2020, p. 76-83Conference paper, Published paper (Refereed)
Abstract [en]

Big Data processing involves handling large and complex data sets, incorporating different tools and frameworks as well as other processes that help organisations make sense of their data collected from various sources. This set of operations, referred to as Big Data workflows, require taking advantage of the elasticity of cloud infrastructures for scalability. In this paper, we present the design and prototype implementation of a Big Data workflow approach based on the use of software container technologies and message-oriented middleware (MOM) to enable highly scalable workflow execution. The approach is demonstrated in a use case together with a set of experiments that demonstrate the practical applicability of the proposed approach for the scalable execution of Big Data workflows. Furthermore, we present a scalability comparison of our proposed approach with that of Argo Workflows-one of the most prominent tools in the area of Big Data workflows.

Place, publisher, year, edition, pages
Association for Computing Machinery, Inc , 2020. p. 76-83
Keywords [en]
Big Data workflows, Domain-specific languages, Software containers, Big data, Containers, Ecosystems, Middleware, Scalability, Cloud infrastructures, Complex datasets, Message oriented middleware, Prototype implementations, Work-flows, Workflow execution, Data handling
National Category
Computer Systems Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-301230DOI: 10.1145/3415958.3433082ISI: 001429310400013Scopus ID: 2-s2.0-85097872671OAI: oai:DiVA.org:kth-301230DiVA, id: diva2:1591570
Conference
12th International Conference on Management of Digital EcoSystems, MEDES 2020, 2 November 2020 through 4 November 2020
Note

QC 20210907

Available from: 2021-09-07 Created: 2021-09-07 Last updated: 2025-12-05Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Dessalk, Yared DejeneMatskin, Mihhail

Search in DiVA

By author/editor
Dessalk, Yared DejeneMatskin, Mihhail
By organisation
KTHSoftware and Computer systems, SCS
Computer SystemsComputer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 152 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf