Endre søk
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf
The power of both choices: Practical load balancing for distributed stream processing engines
KTH, Skolan för elektro- och systemteknik (EES), Kommunikationsnät.ORCID-id: 0000-0001-5872-7809
Vise andre og tillknytning
2015 (engelsk)Inngår i: Proceedings - International Conference on Data Engineering, IEEE conference proceedings, 2015, 137-148 s.Konferansepaper, Publicerat paper (Fagfellevurdert)
Abstract [en]

We study the problem of load balancing in distributed stream processing engines, which is exacerbated in the presence of skew. We introduce Partial Key Grouping (PKG), a new stream partitioning scheme that adapts the classical 'power of two choices' to a distributed streaming setting by leveraging two novel techniques: key splitting and local load estimation. In so doing, it achieves better load balancing than key grouping while being more scalable than shuffle grouping. We test PKG on several large datasets, both real-world and synthetic. Compared to standard hashing, PKG reduces the load imbalance by up to several orders of magnitude, and often achieves nearly-perfect load balance. This result translates into an improvement of up to 60% in throughput and up to 45% in latency when deployed on a real Storm cluster.

sted, utgiver, år, opplag, sider
IEEE conference proceedings, 2015. 137-148 s.
Emneord [en]
Balancing, Distributed parameter control systems, Engines, Distributed stream processing, Distributed streaming, Large datasets, Load balance, Load imbalance, Novel techniques, Orders of magnitude, Power-of-two, Network management
HSV kategori
Identifikatorer
URN: urn:nbn:se:kth:diva-176121DOI: 10.1109/ICDE.2015.7113279Scopus ID: 2-s2.0-84940858966ISBN: 9781479979639 (tryckt)OAI: oai:DiVA.org:kth-176121DiVA: diva2:874693
Konferanse
2015 31st IEEE International Conference on Data Engineering, ICDE 2015, 13 April - 17 April 2015
Merknad

QC 20151127

Tilgjengelig fra: 2015-11-27 Laget: 2015-11-02 Sist oppdatert: 2015-11-27bibliografisk kontrollert

Open Access i DiVA

Fulltekst mangler

Andre lenker

Forlagets fulltekstScopus

Personposter BETA

Nasir, M. Anis U.

Søk i DiVA

Av forfatter/redaktør
Nasir, M. Anis U.
Av organisasjonen

Søk utenfor DiVA

GoogleGoogle Scholar

doi
isbn
urn-nbn

Altmetric

doi
isbn
urn-nbn
Totalt: 21 treff
RefereraExporteraLink to record
Permanent link

Direct link
Referera
Referensformat
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Annet format
Fler format
Språk
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Annet språk
Fler språk
Utmatningsformat
  • html
  • text
  • asciidoc
  • rtf