In modern machine learning, the dominant approach to training models relies on centralized datasets. However, this paradigm is often impractical or even prohibited in real-world scenarios. Concerns about data privacy, ownership, and the ethical use of publicly available data are rapidly growing, especially with increasing scrutiny on how personal data is handled. Furthermore, collecting, storing, and managing large-scale datasets incurs substantial costs. For instance, millions of smartphone users generate vast amounts of data daily -- photos, sleep patterns, text messages, and more -- which is expensive and often infeasible to process centrally. In response, distributed machine learning has emerged as a promising alternative.

Distributed machine learning trains models across multiple users without centralizing data, addressing privacy concerns and logistical challenges. In this framework, data remains with clients, who train models locally and share model updates instead of data. A prominent example is federated learning, which uses a central server to coordinate training by aggregating and distributing updates. In contrast, decentralized learning removes the central server, enabling clients to communicate directly in a peer-to-peer network. However, significant data variability across clients -- data heterogeneity -- complicates model aggregation and reduces performance. This thesis proposes novel strategies to improve decentralized learning, focusing on client collaboration and data heterogeneity.

First, it introduces peer-selection and clustering techniques, enabling clients to collaborate selectively with peers whose data distributions are similar. This approach circumvents the limitations of a single global model, which may fail to generalize well across diverse clients. Second, the thesis develops privacy-preserving methods to estimate data similarity and strengthens user privacy using multi-armed bandits, enabling dynamic, adaptive collaboration among clients. Beyond addressing static data heterogeneity, the thesis also tackles the challenge of evolving data distributions. New algorithms are proposed to enable models to adapt over time, ensuring robust performance even as client data distributions change. The research further extends these methods to generative models, presenting a novel ensemble approach for training generative adversarial networks (GANs) in distributed settings.

Overall, the contributions of this thesis advance the scalability, efficiency, and privacy of distributed machine learning systems. By enhancing these systems' ability to manage diverse data environments, the work ensures more reliable and personalized model performance across clients, paving the way for broader applications of distributed machine learning.

I modern maskininlärning förlitar man sig på central lagring av data för att träna modeller. Detta sätt att träna modeller på är ofta opraktiskt, eller till och med otillåtet, i många realistiska sammanhang. Det finns många orosmoment gällande dataintegritet, ägarskap och etiskt användade av publika datamängder, som växer allt snabbare när mängden information växer. Dessutom medför insamling, lagring och hantering av storskaliga datamängder betydande kostnader. Mobiltelefoner är ett exempel där mycket data genereras dagligen -- användare tar foton, spelar in ljud, skriver textmeddelanden och mycket mer. Att samla in denna data är dyrt och ofta omöjligt att behandla centralt. I ljuset av detta har distribuerad maskininlärning dykt upp som ett lovande alternativ.

Distribuerad maskininlärning möjliggör modellträning för klienter utan något krav på att centralisera data. I detta ramverk stannar data hos klienterna, som tränar modeller lokalt och istället delar modelluppdateringar. Federerad inlärning är ett sådant exempel, som bygger på en central server för att koordinera modellträning genom att aggregera och distribuera modeller. Decentraliserad inlärning eliminerar däremot behovet av en central server helt och hållet, genom att klienter kommunicerar direkt i ett peer-to-peer-nätverk för att kollaborativt träna modeller. Men när data skiljer sig mellan klienter -- när dataheterogeniteten är stor -- försvårar det att träna modeller via aggregering. Denna avhandling föreslår därmed nya strategier för att förbättra träning i distribuerad maskininlärning i miljöer av betydande dataheterogenitet.

Först presenterar vi metoder för klientselektion och klustring, vilket möjliggör för klienter att samarbeta selektivt med klienter med liknande datafördelningar. Detta tillvägagångssätt kringgår begränsningarna med en global modell som inte nödvändigtvis kan generalisera över samtliga klienter. Vidare utvecklar vi integritetsbevarande metoder för att uppskatta datalikhet utan att kränka användarnas integritet genom att använda flerarmade banditer. Utöver att ta itu med stationär dataheterogenitet, tar avhandlingen också upp utmaningen med icke-stationära datamängder. En ny algoritm föreslås som gör det möjligt för modeller att anpassa sig över tid. Slutligen studerar vi generativa modeller och föreslår en ny metod för att träna generative adversarial networks (GANs) decentraliserat.

Sammantaget förbättrar bidragen från denna avhandling skalbarheten och prestandan hos distribuerade maskininlärningssystem. Genom att förbättra systemens förmåga att hantera heterogena datamiljöer skapar detta arbete ett mer robust ramverk, vilket banar väg för bredare användning av distribuerad maskininlärning.

Decentralized Learning (DL) enables privacy-preserving collaboration among organizations or users to enhance the performance of local deep learning models. However, model aggregation becomes challenging when client data is heterogeneous, and identifying compatible collaborators without direct data exchange remains a pressing issue. In this paper, we investigate the effectiveness of various similarity metrics in DL for identifying peers for model merging, conducting an empirical analysis across multiple datasets with distribution shifts. Our research provides insights into the performance of these metrics, examining their role in facilitating effective collaboration. By exploring the strengths and limitations of these metrics, we contribute to the development of robust DL methods.

Personalized decentralized learning is a promising paradigm for distributed learning, enabling each node to train a local model on its own data and collaborate with other nodes to improve without sharing any data. However, this approach poses significant privacy risks, as nodes may inadvertently disclose sensitive information about their data or preferences through their collaboration choices. In this paper, we propose Private Personalized Decentralized Learning (PPDL), a novel approach that combines secure aggregation and correlated adversarial multi-armed bandit optimization to protect node privacy while facilitating efficient node selection. By leveraging dependencies between different arms, represented by potential collaborators, we demonstrate that PPDL can effectively identify suitable collaborators solely based on aggregated models. Additionally, we show that PPDL surpasses previous non-private methods in model performance on standard benchmarks under label and covariate shift scenarios.

Federated Learning (FL) is a promising framework for distributed learning when data is private and sensitive. However, the state-of-the-art solutions in this framework are not optimal when data is heterogeneous and non-IID. We propose a practical and robust approach to personalization in FL that adjusts to heterogeneous and non-IID data by balancing exploration and exploitation of several global models. To achieve our aim of personalization, we use a Mixture of Experts (MoE) that learns to group clients that are similar to each other, while using the global models more efficiently. We show that our approach achieves an accuracy up to 29.78% better than the state-of-the-art and up to 4.38% better compared to a local model in a pathological non-IID setting, even though we tune our approach in the IID setting.

We study the problem of training personalized deep learning models in a decentralized peer-to-peer setting, focusing on the setting where data distributions differ between the clients and where different clients have different local learning tasks. We study both covariate and label shift, and our contribution is an algorithm which for each client finds beneficial collaborations based on a similarity estimate for the local task. Our method does not rely on hyperparameters which are hard to estimate, such as the number of client clusters, but rather continuously adapts to the network topology using soft cluster assignment based on a novel adaptive gossip algorithm. We test the proposed method in various settings where data is not independent and identically distributed among the clients. The experimental evaluation shows that the proposed method performs better than previous state-of-the-art algorithms for this problem setting, and handles situations well where previous methods fail.

Federated Learning (FL) is a promising framework for distributed learning whendata is private and sensitive. However, the state-of-the-art solutions in thisframework are not optimal when data is heterogeneous and non-Independent andIdentically Distributed (non-IID). We propose a practical and robust approachto personalization in FL that adjusts to heterogeneous and non-IID data bybalancing exploration and exploitation of several global models. To achieve ouraim of personalization, we use a Mixture of Experts (MoE) that learns to groupclients that are similar to each other, while using the global models moreefficiently. We show that our approach achieves an accuracy up to 29.78 % andup to 4.38 % better compared to a local model in a pathological non-IIDsetting, even though we tune our approach in the IID setting.

Decentralized machine learning tackles the problem of learning useful models when data is distributed among several clients. The most prevalent decentralized setting today is federated learning (FL), where a central server orchestrates the learning among clients. In this work, we contribute to the relatively understudied sub-field of generative modelling in the FL framework.We study the task of how to train generative adversarial networks (GANs) when training data is heterogeneously distributed (non-iid) over clients and cannot be shared. Our objective is to train a generator that is able to sample from the collective data distribution centrally, while the client data never leaves the clients and user privacy is respected. We show using standard benchmark image datasets that existing approaches fail in this setting, experiencing so-called client drift when the local number of epochs becomes to large and local parameters drift too far away in parameter space. To tackle this challenge, we propose a novel approach named EFFGAN: Ensembles of fine-tuned federated GANs. Being an ensemble of local expert generators, EFFGAN is able to learn the data distribution over all clients and mitigate client drift. It is able to train with a large number of local epochs, making it more communication efficient than previous works.