Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
State Management in Apache Flink: Consistent Stateful Distributed Stream Processing
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.ORCID iD: 0000-0002-9351-8508
data Artisans.
King Digital Entertainment Limited.
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.
Show others and affiliations
2017 (English)In: Proceedings of the VLDB Endowment, ISSN 2150-8097, E-ISSN 2150-8097, Vol. 10, p. 1718-1729, article id 12Article in journal (Refereed) Published
Abstract [en]

Stream processors are emerging in industry as an apparatus that drives analytical but also mission critical services handling the core of persistent application logic. Thus, apart from scalability and low-latency, a rising system need is first-class support for application state together with strong consistency guarantees, and adaptivity to cluster reconfigurations, software patches and partial failures. Although prior systems research has addressed some of these specific problems, the practical challenge lies on how such guarantees can be materialized in a transparent, non-intrusive manner that relieves the user from unnecessary constraints. Such needs served as the main design principles of state management in Apache Flink, an open source, scalable stream processor.

We present Flink’s core pipelined, in-flight mechanism which guarantees the creation of lightweight, consistent, distributed snapshots of application state, progressively, without impacting continuous execution. Consistent snapshots cover all needs for system reconfiguration, fault tolerance and version management through coarse grained rollback recovery. Application state is declared explicitly to the system, allowing efficient partitioning and transparent commits to persistent storage. We further present Flink’s backend implementations and mechanisms for high availability, external state queries and output commit. Finally, we demonstrate how these mechanisms behave in practice with metrics and large deployment insights exhibiting the low performance trade-offs of our approach and the general benefits of exploiting asynchrony in continuous, yet sustainable system deployments.

Place, publisher, year, edition, pages
ACM Digital Library, 2017. Vol. 10, p. 1718-1729, article id 12
Keywords [en]
stream processing, data management, computer systems, databases, data processing, data science, big data
National Category
Computer Systems
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:kth:diva-220133DOI: 10.14778/3137765.3137777OAI: oai:DiVA.org:kth-220133DiVA, id: diva2:1166553
Conference
VLDB Very Large Data Bases Conference Munich, 2017
Projects
End-to-End Clouds - SSF (RIT10-0043)Streamline - Horizon 2020 (688191)
Funder
Swedish Foundation for Strategic Research , RIT10-0043
Note

QC 20171218

Available from: 2017-12-15 Created: 2017-12-15 Last updated: 2017-12-18Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full texthttps://dl.acm.org/citation.cfm?id=3137777

Search in DiVA

By author/editor
Carbone, ParisHaridi, Seif
By organisation
Software and Computer systems, SCS
In the same journal
Proceedings of the VLDB Endowment
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 143 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf