Aggregate Local, Sync Global: A Hierarchical Approach to Efficient Geo-Distributed LLM TrainingShow others and affiliations
2026 (English)In: INFOCOM 2026 - IEEE Conference on Computer Communications, Institute of Electrical and Electronics Engineers (IEEE) , 2026Conference paper, Published paper (Refereed)
Abstract [en]
Nowadays, the exponential growth of LLMs in size, increasingly stringent data sovereignty regulations, and growing power and energy constraints in modern datacenters are driving the adoption of multi-datacenter infrastructures to distribute the training process. However, standard training frameworks prove ill-suited for these heterogeneous environments, lacking awareness of inter-datacenter constraints or stretching communication patterns across sites without differentiating between network tiers. This approach causes severe bottlenecks on high-latency inter-datacenter links, hindering scalability. To address this challenge, we propose a hierarchical communication strategy designed to decouple intra-site aggregation from global synchronization through the election of "leader model replicas". Experimental results show that the proposed hierarchical strategy reduces training iteration time by up to 32% and exposed communication time by up to 65% compared to the baseline Megatron-LM strategy, providing an effective solution for scaling training workloads across geographically distributed datacenters.
Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE) , 2026.
Keywords [en]
Collective Communications Library, Datacenters, Large Language Models, Multi-Site Training
National Category
Computer Systems Communication Systems
Identifiers
URN: urn:nbn:se:kth:diva-386135DOI: 10.1109/INFOCOM59046.2026.11571620Scopus ID: 2-s2.0-105044497247OAI: oai:DiVA.org:kth-386135DiVA, id: diva2:2088454
Conference
2026 IEEE Conference on Computer Communications, INFOCOM 2026, Tokyo, Japan, May 18 2026 - May 21 2026
Note
Part of ISBN 979-8-3315-4961-9
QC 20260728
2026-07-282026-07-282026-07-28Bibliographically approved