Change search
ReferencesLink to record
Permanent link

Direct link
Lightweight Asynchronous Snapshots for Distributed Dataflows
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.ORCID iD: 0000-0002-9351-8508
CSL Computer Systems Laboratory, SICS Swedish Institute of Compute Science.
Data Artisans GmbH.
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.ORCID iD: 0000-0002-6718-0144
Show others and affiliations
2015 (English)Report (Other academic)
Abstract [en]

Distributed stateful stream processing enables the deployment and execution of large scale continuous computations in the cloud, targeting both low latency and high throughput. One of the most fundamental challenges of this paradigm is providing processing guarantees under potential failures. Existing approaches rely on periodic global state snapshots that can be used for failure recovery. Those approaches suffer from two main drawbacks. First, they often stall the overall computation which impacts ingestion. Second, they eagerly persist all records in transit along with the operation states which results in larger snapshots than required. In this work we propose Asynchronous Barrier Snapshotting (ABS), a lightweight algorithm suited for modern dataflow execution engines that minimises space requirements. ABS persists only operator states on acyclic execution topologies while keeping a minimal record log on cyclic dataflows. We implemented ABS on Apache Flink, a distributed analytics engine that supports stateful stream processing. Our evaluation shows that our algorithm does not have a heavy impact on the execution, maintaining linear scalability and performing well with frequent snapshots. 

Place, publisher, year, edition, pages
2015. , 8 p.
, TRITA-ICT, 2015:08
Keyword [en]
fault tolerance, distributed computing, stream processing, dataflow, cloud computing, state management
National Category
Computer Systems
Research subject
Information and Communication Technology; Computer Science
URN: urn:nbn:se:kth:diva-170185ISBN: 978-91-7595-651-0OAI: diva2:827567

QC 20150630

Available from: 2015-06-28 Created: 2015-06-28 Last updated: 2015-06-30Bibliographically approved

Open Access in DiVA

Report(611 kB)40 downloads
File information
File name FULLTEXT01.pdfFile size 611 kBChecksum SHA-512
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Carbone, ParisHaridi, Seif
By organisation
Software and Computer systems, SCS
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 40 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Total: 137 hits
ReferencesLink to record
Permanent link

Direct link