A survey of big data pipeline orchestration tools from the perspective of the datacloud projectShow others and affiliations
2021 (English)In: CEUR Workshop Proceedings, CEUR-WS , 2021, Vol. 3036, p. 63-78Conference paper, Published paper (Refereed)
Abstract [en]
This paper presents a survey of existing tools for Big Data pipeline orchestration based on a comparative framework developed in the DataCloud project. We propose criteria for evaluating the tools to support reusability, flexible pipeline communication modes, and separation of concerns in Big Data pipeline descriptions. This survey aims to identify research and technological gaps and to recommend approaches for filling them. Further work in the DataCloud project is oriented towards the design, implementation, and practical evaluation of the recommended approaches.
Place, publisher, year, edition, pages
CEUR-WS , 2021. Vol. 3036, p. 63-78
Series
CEUR Workshop Proceedings, ISSN 16130073 ; 3036
Keywords [en]
Big data pipeline, Orchestration tools, Reusability, Big data, Pipelines, Surveys, Communication mode, Data pipelines, Design implementation, Flexible pipeline, Further works, Orchestration tool, Pipeline communication, Separation of concerns
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-313861Scopus ID: 2-s2.0-85121276663OAI: oai:DiVA.org:kth-313861DiVA, id: diva2:1668403
Conference
Supplementary 23rd International Conference on Data Analytics and Management in Data Intensive Domains, DAMDID/RCDL 2021, Virtual, Moscow, 26-29 October 2021
Note
QC 20220613
2022-06-132022-06-132022-06-25Bibliographically approved