SAASFEE: Scalable scientific workflow execution engine
2015 (English)In: Proceedings of the VLDB Endowment, ISSN 2150-8097, E-ISSN 2150-8097, Vol. 8, no 12, 1892-1895 p.Article in journal (Refereed) PublishedText
Across many fields of science, primary data sets like sensor read-outs, time series, and genomic sequences are analyzed by complex chains of specialized tools and scripts exchanging intermediate results in domain-specific file formats. Scientific work ow management systems (SWfMSs) support the development and execution of these tool chains by providing work ow specification languages, graphical editors, fault-tolerant execution engines, etc. However, many SWfMSs are not prepared to handle large data sets because of inadequate support for distributed computing. On the other hand, most SWfMSs that do support distributed computing only allow static task execution orders. We present SAASFEE, a SWfMS which runs arbitrarily complex work ows on Hadoop YARN. Work ows are specified in Cuneiform, a functional work ow language focusing on parallelization and easy integration of existing software. Cuneiform work ows are executed on Hi-WAY, a higher-level scheduler for running work ows on YARN. Distinct features of SAASFEE are the ability to execute iterative work ows, an adaptive task scheduler, re-executable provenance traces, and compatibility to selected other work ow systems. In the demonstration, we present all components of SAASFEE using real-life work ows from the field of genomics.
Place, publisher, year, edition, pages
Association for Computing Machinery (ACM), 2015. Vol. 8, no 12, 1892-1895 p.
Chains, Computational linguistics, Engines, Information management, Specification languages, Wool, Yarn, Execution engine, Genomic sequence, Graphical editors, Intermediate results, Management systems, Parallelizations, Scientific workflows, Specialized tools, Distributed computer systems
IdentifiersURN: urn:nbn:se:kth:diva-181254ScopusID: 2-s2.0-84953879839OAI: oai:DiVA.org:kth-181254DiVA: diva2:900934
11 September 2006 through 11 September 2006, Seoul
QC 201602052016-02-052016-01-292016-02-05Bibliographically approved