kth.sePublications KTH
Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Aggregate Local, Sync Global: A Hierarchical Approach to Efficient Geo-Distributed LLM Training
Roma Tre University, Italy.
Roma Tre University, Italy.
RISE Research Institutes of Sweden.
Roma Tre University, Italy.
Show others and affiliations
2026 (English)In: INFOCOM 2026 - IEEE Conference on Computer Communications, Institute of Electrical and Electronics Engineers (IEEE) , 2026Conference paper, Published paper (Refereed)
Abstract [en]

Nowadays, the exponential growth of LLMs in size, increasingly stringent data sovereignty regulations, and growing power and energy constraints in modern datacenters are driving the adoption of multi-datacenter infrastructures to distribute the training process. However, standard training frameworks prove ill-suited for these heterogeneous environments, lacking awareness of inter-datacenter constraints or stretching communication patterns across sites without differentiating between network tiers. This approach causes severe bottlenecks on high-latency inter-datacenter links, hindering scalability. To address this challenge, we propose a hierarchical communication strategy designed to decouple intra-site aggregation from global synchronization through the election of "leader model replicas". Experimental results show that the proposed hierarchical strategy reduces training iteration time by up to 32% and exposed communication time by up to 65% compared to the baseline Megatron-LM strategy, providing an effective solution for scaling training workloads across geographically distributed datacenters.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE) , 2026.
Keywords [en]
Collective Communications Library, Datacenters, Large Language Models, Multi-Site Training
National Category
Computer Systems Communication Systems
Identifiers
URN: urn:nbn:se:kth:diva-386135DOI: 10.1109/INFOCOM59046.2026.11571620Scopus ID: 2-s2.0-105044497247OAI: oai:DiVA.org:kth-386135DiVA, id: diva2:2088454
Conference
2026 IEEE Conference on Computer Communications, INFOCOM 2026, Tokyo, Japan, May 18 2026 - May 21 2026
Note

Part of ISBN 979-8-3315-4961-9

QC 20260728

Available from: 2026-07-28 Created: 2026-07-28 Last updated: 2026-07-28Bibliographically approved

Open Access in DiVA

No full text in DiVA

Other links

Publisher's full textScopus

Authority records

Chiesa, Marco

Search in DiVA

By author/editor
Chiesa, Marco
By organisation
Computing and Learning Systems
Computer SystemsCommunication Systems

Search outside of DiVA

GoogleGoogle Scholar

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 20 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf