Change search
ReferencesLink to record
Permanent link

Direct link
The power of both choices: Practical load balancing for distributed stream processing engines
KTH, School of Electrical Engineering (EES), Communication Networks.ORCID iD: 0000-0001-5872-7809
Show others and affiliations
2015 (English)In: Proceedings - International Conference on Data Engineering, IEEE conference proceedings, 2015, 137-148 p.Conference paper (Refereed)
Abstract [en]

We study the problem of load balancing in distributed stream processing engines, which is exacerbated in the presence of skew. We introduce Partial Key Grouping (PKG), a new stream partitioning scheme that adapts the classical 'power of two choices' to a distributed streaming setting by leveraging two novel techniques: key splitting and local load estimation. In so doing, it achieves better load balancing than key grouping while being more scalable than shuffle grouping. We test PKG on several large datasets, both real-world and synthetic. Compared to standard hashing, PKG reduces the load imbalance by up to several orders of magnitude, and often achieves nearly-perfect load balance. This result translates into an improvement of up to 60% in throughput and up to 45% in latency when deployed on a real Storm cluster.

Place, publisher, year, edition, pages
IEEE conference proceedings, 2015. 137-148 p.
Keyword [en]
Balancing, Distributed parameter control systems, Engines, Distributed stream processing, Distributed streaming, Large datasets, Load balance, Load imbalance, Novel techniques, Orders of magnitude, Power-of-two, Network management
National Category
Electrical Engineering, Electronic Engineering, Information Engineering
URN: urn:nbn:se:kth:diva-176121DOI: 10.1109/ICDE.2015.7113279ScopusID: 2-s2.0-84940858966ISBN: 9781479979639OAI: diva2:874693
2015 31st IEEE International Conference on Data Engineering, ICDE 2015, 13 April - 17 April 2015

QC 20151127

Available from: 2015-11-27 Created: 2015-11-02 Last updated: 2015-11-27Bibliographically approved

Open Access in DiVA

No full text

Other links

Publisher's full textScopus

Search in DiVA

By author/editor
Nasir, M. Anis U.
By organisation
Communication Networks
Electrical Engineering, Electronic Engineering, Information Engineering

Search outside of DiVA

GoogleGoogle Scholar
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

Altmetric score

Total: 10 hits
ReferencesLink to record
Permanent link

Direct link