kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Failure Transparency in Stateful Dataflow Systems
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Theoretical Computer Science, TCS. KTH, School of Electrical Engineering and Computer Science (EECS), Centres, Digital futures.ORCID iD: 0000-0002-5091-9811
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Theoretical Computer Science, TCS. KTH, School of Electrical Engineering and Computer Science (EECS), Centres, Digital futures.ORCID iD: 0000-0002-7119-5234
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Software and Computer systems, SCS. KTH, School of Electrical Engineering and Computer Science (EECS), Centres, Digital futures. Digital Systems, RISE Research Institutes of Sweden, Stockholm, Sweden.ORCID iD: 0000-0002-9351-8508
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Theoretical Computer Science, TCS. KTH, School of Electrical Engineering and Computer Science (EECS), Centres, Digital futures.ORCID iD: 0000-0002-2659-5271
2024 (English)In: 38th European Conference on Object-Oriented Programming (ECOOP 2024) / [ed] Aldrich J.; Salvaneschi G., Schloss Dagstuhl - Leibniz-Zentrum für Informatik , 2024, Vol. 313, p. 42:1-42:31, article id 37Conference paper, Published paper (Refereed)
Abstract [en]

Failure transparency enables users to reason about distributed systems at a higher level of abstraction, where complex failure-handling logic is hidden. This is especially true for stateful dataflow systems, which are the backbone of many cloud applications. In particular, this paper focuses on proving failure transparency in Apache Flink, a popular stateful dataflow system. Even though failure transparency is a critical aspect of Apache Flink, to date it has not been formally proven. Showing that the failure transparency mechanism is correct, however, is challenging due to the complexity of the mechanism itself. Nevertheless, this complexity can be effectively hidden behind a failure transparent programming interface. To show that Apache Flink is failure transparent, we model it in small-step operational semantics. Next, we provide a novel definition of failure transparency based on observational explainability, a concept which relates executions according to their observations. Finally, we provide a formal proof of failure transparency for the implementation model; i.e., we prove that the failure-free model correctly abstracts from the failure-related details of the implementation model. We also show liveness of the implementation model under a fair execution assumption. These results are a first step towards a verified stack for stateful dataflow systems.

Place, publisher, year, edition, pages
Schloss Dagstuhl - Leibniz-Zentrum für Informatik , 2024. Vol. 313, p. 42:1-42:31, article id 37
Series
Leibniz International Proceedings in Informatics (LIPIcs), ISSN 1868-8969 ; 313
Keywords [en]
checkpoint recovery, Failure transparency, operational semantics, stateful dataflow
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-366419DOI: 10.4230/LIPIcs.ECOOP.2024.42ISI: 001533999700042Scopus ID: 2-s2.0-85204981346OAI: oai:DiVA.org:kth-366419DiVA, id: diva2:1982257
Conference
38th European Conference on Object-Oriented Programming (ECOOP 2024), Vienna, Austria, September 16-20, 2024
Note

Part of ISBN 9783959773416

QC 20250923

Available from: 2025-07-07 Created: 2025-07-07 Last updated: 2025-11-11Bibliographically approved
In thesis
1. Programming Models for Failure-Transparent Distributed Systems
Open this publication in new window or tab >>Programming Models for Failure-Transparent Distributed Systems
2025 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Failure-transparent programming models abstract from failures by fully masking them from the programmer. They are widely used for programming distributed systems, as failures otherwise are considered a core difficulty. The most widely used of its kind for processing data is stateful dataflow streaming, a model restricted to static, directed, acyclic graphs of stateful stream processors. However, its restrictions limit the applicability of the model, as it lacks support for compositional patterns and replicated data types, making it difficult to express certain applications. Moreover, there is a lack of formal foundations and proofs of failure transparency.

This thesis contributes a semantics-agnostic definition of failure transparency, and two proofs of failure transparency, one of which is for a model of a stateful dataflow streaming system. It additionally contributes two novel programming models based on stateful dataflow streaming. The first provides extensions for compositional patterns, allowing it to express use cases such as a shopping cart. The second provides extensions for windowed conflict-free replicated data types, implemented in a low-latency programming system for global aggregations.

This thesis demonstrates the utility of failure-transparent programming models for distributed systems by contributions to its formal foundations and by making it applicable to a wider range of applications.

Abstract [sv]

Feltransparenta programmeringsmodeller abstraherar från fel genom att helt dölja dem för programmeraren. De används ofta för programmering av distribuerade system, eftersom fel annars anses vara ett centralt problem. Den mest använda modellen för databehandling är stateful dataflow streaming, en modell som är begränsad till statiska, riktade, acykliska grafer av stateful stream-processorer. Dess begränsningar begränsar dock modellens tillämpbarhet, eftersom den saknar stöd för kompositionella mönster och replikerade datatyper, vilket gör det svårt att uttrycka vissa applikationer. Dessutom saknas formella grunder och bevis för feltransparens.

Denna avhandling bidrar med en semantiksagnostisk definition av feltransparens och två bevis för feltransparens, varav ett är för en modell av ett stateful dataflow streaming system. Den bidrar dessutom med två nya programmeringsmodeller baserade på stateful dataflow streaming. Den första tillhandahåller tillägg för kompositionella mönster, vilket gör det möjligt att uttrycka användningsfall som till exempel en kundvagn. Den andra tillhandahåller tillägg för fönsterbaserade konfliktfria replikerade datatyper, implementerade i ett programmeringssystem med låg latens för globala aggregeringar.

Denna avhandling demonstrerar nyttan av feltransparenta programmeringsmodeller för distribuerade system genom bidrag till dess formella grunder och genom att göra den tillämpbar på ett bredare spektrum av applikationer.

Place, publisher, year, edition, pages
Stockholm, Sweden: KTH Royal Institute of Technology, 2025. p. xiii, 70
Series
TRITA-EECS-AVL ; 2025:102
Keywords
Failure transparency, Programming models, Stateful dataflow streaming, Operational semantics, Distributed systems, Feltransparens, Programmeringsmodeller, Stateful dataflow streaming, Operationell semantik, Distribuerade system
National Category
Computer Sciences Networked, Parallel and Distributed Computing
Identifiers
urn:nbn:se:kth:diva-372645 (URN)978-91-8106-456-8 (ISBN)
Public defence
2025-12-11, https://kth-se.zoom.us/j/65545597811, Kollegiesalen, Brinellvägen 8, Stockholm, 09:00 (English)
Opponent
Supervisors
Note

QC 20251112

Available from: 2025-11-12 Created: 2025-11-11 Last updated: 2025-11-24Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Veresov, AlekseySpenger, JonasCarbone, ParisHaller, Philipp

Search in DiVA

By author/editor
Veresov, AlekseySpenger, JonasCarbone, ParisHaller, Philipp
By organisation
Theoretical Computer Science, TCSDigital futuresSoftware and Computer systems, SCS
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 127 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf