Ultra-reliable low latency communications (URLLC) service is envisioned to enable use cases with strict reliability and latency requirements in 5G. One approach for enabling URLLC services is to leverage Reinforcement Learning (RL) to efficiently allocate wireless resources. However, with conventional RL methods, the decision variables (though being deployed at various network layers) are typically optimized in the same control loop, leading to significant practical limitations on the control loop's delay as well as excessive signaling and energy consumption. In this paper, we propose a multi-agent Hierarchical RL (HRL) framework that enables the implementation of multi-level policies with different control loop timescales. Agents with faster control loops are deployed closer to the base station, while the ones with slower control loops are at the edge or closer to the core network providing high-level guidelines for low-level actions. On a use case from the prior art, with our HRL framework, we optimized the maximum number of retransmissions and transmission power of industrial devices. Our extensive simulation results on the factory automation scenario show that the HRL framework achieves better performance as the baseline single-agent RL method, with significantly less overhead of signal transmissions and delay compared to the one-agent RL methods.

Digital transformation within smart manufacturing presents new challenges for wireless communication, demanding stringent reliability and latency. One prominent approach to meet these requirements in 5G technology is to leverage spatial diversity techniques, such as the transmission of duplicated packets via independent user plane paths. While spatial diversity and hardware redundancy ensure high availability and reduced latency, they increase wireless resource utilization significantly. In this paper, we investigate a scenario where large industrial devices can access multiple user plane paths via multiple user equipment. To manage this effectively, we propose a deep Q-network-based reinforcement learning control framework that optimizes spatial diversity use to maximize communication service availability with minimized wireless resource usage. We implement our solution on a 3GPP-compliant simulator for a factory automation scenario. Our results show that our framework can adapt to varying delay bounds and greatly enhance communication service availability compared to the baselines. Remarkably, our method achieves these results more resource-efficiently, evading the baseline's need for double the bandwidth for comparable availability levels.

Communication systems supporting cyber-physical production applications should satisfy stringent delay and reliability requirements. Diversity techniques and power control are the main approaches to reduce latency and enhance the reliability of wireless communications at the expense of redundant transmissions and excessive resource usage. Focusing on the application layer reliability key performance indicators (KPIs), we design a deep reinforcement learning orchestrator for power control and hybrid automatic repeat request retransmissions to optimize these KPIs. Furthermore, to address the scalability issue that emerges in the per-device orchestration problem, we develop a new branching soft actor-critic framework in which a separate branch represents the action space of each industrial device. Our orchestrator enables near real-time control and can be implemented in the edge cloud. We test our solution with a 3GPP-compliant and realistic simulator for factory automation scenarios. Compared to the state-of-the-art, our solution offers significant scalability gains in terms of computational time and memory requirements. Our extensive experiments show significant improvements in our target KPIs, over the state-of-the-art, especially for 5th percentile user availability. To achieve these targets, our framework requires substantially less total energy or spectrum, thanks to our scalable RL solution.

Cyber-Physical Systems (CPSs) are becoming ubiquitous in modern society, enabling new applications that rely on the seamless interaction between computing, communication, and physical processes. In this context, ultra-reliable low-latency communications (URLLC) emerges as a crucial element, reliably allowing the real-time exchange of critical data.

In wireless networks, reliability is commonly evaluated based on the percentage of packets delivered successfully, with timeliness sometimes considered. Nevertheless, in CPSs, performance is typically assessed by operational metrics such as availability (as the ability to provide service at any given time) and reliability (as the ability to maintain consistent service over an extended period). To bridge the gap between these two domains, we study the CPSs performance in terms of wireless communications and derive a mapping function between the well-known network metrics (such as packet error ratio) and operational metrics (namely communication service availability and reliability) for deterministic traffic arrivals. This thesis then deals with wireless system orchestration techniques that aim to facilitate URLLC for CPSs, factoring in spectrum and energy efficiency. It investigates two scenarios: i) a single service, where the focus is only on URLLC, and ii) mixed services, where other services simultaneously run on the same network as URLLC.

In the first part, we assume that the impact of other nearby services on URLLC service is negligible. Accordingly, we concentrate on diversity techniques and power control as primary methods to enhance communication service availability and reliability at the cost of redundant transmissions and excessive resource usage. Thus, we devise a deep reinforcement learning (DRL) orchestrator that optimizes the number of hybrid automatic repeat request retransmissions and transmission power to enhance these metrics. We use a deep Q-network framework along with a branching soft actor-critic (BSAC) framework to address scalability issues in per-device orchestration. Our 3GPP-compliant simulations show that our approach achieves significant gains in computational time and memory requirements compared to the state-of-the-art. Besides, our approach requires substantially less energy or spectrum to achieve the target metrics. Additionally, we offer valuable insights into the practical implementation of DRL solutions for URLLC service in real-world wireless communication systems.

In the second part, we examine mixed services with an emphasis on distributed learning as a coexistent service. We consider 5G-NR's quality of service mechanisms to prioritize URLLC traffic and develop models to characterize distributed training workflow in terms of training delay, model size, and convergence. This leads to an optimization problem that uses device selection to minimize distributed learning convergence time, while meeting URLLC availability requirements. We transform this coexistence problem into a DRL problem and tackle it with our adjusted BSAC framework. Our simulations reveal that our approach achieves URLLC service availability performance comparable to the scenario where all communication resources are solely dedicated to URLLC service, and significantly higher than the performance achieved using a static slicing approach with unvarying dedicated resources to slices.  Finally, we propose a hierarchical reinforcement learning architecture for dynamic resource slicing on a large timescale, thereby enhancing network flexibility, scalability, and profitability.

Cyberfysiska system (CPS) blir alltmer relevanta och påtagliga i det moderna samhället. I sådana system sker en sömlös interaktion mellan datorberäkningar, kommunikation och fysiska processer. I detta sammanhang spelar system för kommunikation med extremt hög tillförlitlighet och låg fördröjning, s.k. URRLC (Ultra Reliable Low Latency Communication), en nyckelroll vid överföring av viktiga och tidskritiska data. I denna avhandling undersöks existerande tekniska lösningar för att erbjuda URLLC-tjänster i CPS-tillämpningar samt dessas prestanda och begränsningar. Vidare presenteras och utvärderas våra förslag till förbättringar för URRLC-tekniker inom ramen för 5G och inom andra nätarkitekturer.

Avhandlingen inleds med en översikt av de utmaningar som möter konstruktörer av CPS inom trådlösa nätverk, samt med en jämförelse av de vanligaste förekommande trådbundna och trådlösa kommunikationsstandarderna,  inklusive 5G. Därefter presenteras en omfattande översikt av olika prestandakriterier  för CPS samt för de senaste tekniska lösningar för URLLC-tjänster.

Tillförlitligheten för ett trådlösa nätverk utvärderas vanligen genom att mäta andelen datapaket som levereras i tid. Relevanta prestandamått för CPS är däremot systemets tillgänglighet (förmågan att tillhandahålla tjänster vid en godtycklig tidpunkt) och tillförlitlighet, som här definieras som förmågan att kunna bibehålla tjänsten över tid. I syfte att överbrygga klyftan mellan dessa två domäner studeras prestanda hos CPS som utnyttjar trådlösa nätverk som kommunikationsmedium. I avhandlingen härleds ett samband mellan väletablerade prestandamått för nätverk (t.ex. paketfel) och prestandamått för CPS (tillgänglighet och tillförlitlighet) för en deterministisk trafikmodell. Denna avhandling analyserar olika implementeringsstrategier för trådlösa system för att dessa skall uppfylla de prestandakrav som ställs på CPS. Kritiska parametrar är här spektrum- och energieffektivitet.

Två scenarier studeras:  i) ett där enbart en URLLC-tjänst erbjuds, och ii) i ett scenario där URRLC tjänsten tvingas dela resurser med andra tjänster. Antagandet i det första scenariot är att resurstillgången är så god, att påverkan från övriga tjänster på URLLC är försumbar. I detta scenario koncentrerar vi oss på diversitetssteknologier och effektreglering som primära metoder för att öka kommunikationstjänstens tillförlitlighet till priset av redundanta överföringar och överkonsumtion av resurser. En maskininlärningsteknik, s.k.Deep Reinforcement Learning (DRL)) används för att optimerar omsändningsförfarande (ARQ) och sändareffekt i syfte att förbättra tillgänglighet och tillförlitlighet. . Ett djupt Q-nätverk med en s.k. ”Branching Soft Actor-Critic” (BSAC) teknik,  har använts för att lösa skalbarhetsproblem.  Vår metod utvärderas med en 3GPP-kompatibel simulator för fabriksautomatiseringsscenarier. Vi kan visa påtagliga vinster vad avser våra prestandamått men även var gäller beräkningstid och minneskrav  i jämförelse med den tidigare arbeten. Den skalbara DRL metoden resulterar i betydligt lägre total energiförbrukning och spektrumbehov för att uppnå uppsatta prestandamål. Metoden erbjuder även viktiga insikter i den praktiska implementeringen av DRL lösningar för URLLC i verkliga trådlösa kommunikationssystem.

I det andra scenariot undersöker vi hur tjänster som konkurrerar med URRLC-tjänsten om resurserna, påverkar prestanda. Vi fokuserar på ett fall där ett system för distribuerad maskininlärning är en samexisterande tjänst. Den befintliga tjänsten “5G-NR quality of service” utnyttjas här för att prioritera URLLC-trafik. Vi utvecklar modeller för att karakterisera den trafik som arbetsflödet för maskininlärning (t.ex. träning av fördröjning, modellstorlek, konvergens och noggrannhet) ger upphov till, för att kunna analysera samspelet mellan denna trafik och URLLC-tillgängligheten. Eftersom systemet har begränsade resurser (när det gäller bandbredd och överföringskraft) och tillgänglighetskraven för URLLC är strikta, måste maskinlärningen stå tillbaka och endast låta en delmängd av sina enheter delta i varje iteration av den distribuerade träningen. Vi formulerar ett optimeringsproblem som minimerar den genomsnittliga träningstiden för maskinlärningstjänsten i syfte att uppnå en viss noggrannhet utan att kompromissa med URLLC -tillgängligheten. Vi omvandlar detta samexistensproblem till ett DRL problem och använder åter vår BSAC metod för att angripas problemet. Realistiska simuleringar av scenarier för fabriksautomation har använts för utvärdering av vår metod. Resultatet är en väsentlig minskning av träningstiden inom den distribuerade maskinlärningstjänsten samtidigt som den bibehåller URLLC-tillgängligheten över det krävda tröskelvärdet. Vår metod åstadkommer en högre prestandanivå för URLLC-tillgängligheten jämfört med de konventionella statiska “slicing” metoderna och prestanda snarlika med det första scenariot, där samtliga kommunikationsresurser var dedikerade till URLLC-tjänsten, uppnås. För att övervinna begränsningar i den statiska “slicing” metoden och åstadkomma mera flexibla, skalbara och lönsamma nätverk, föreslår vi en implementering av en hierarkisk RL-metod som dynamisk allokerar kommunikation och beräkningsresurser till tjänster över en större tidskala.

Industry 4.0 imposes strict requirements on the fifth generation of wireless systems (5G), such as high reliability, high availability, and low latency. Guaranteeing such requirements implies that system failures should occur with an extremely low probability. However, some applications (e.g., training a reinforcement learning algorithm to operate in highly reliable systems or rare event simulations) require access to a broad range of observed failures and extreme values, preferably in a short time. In this paper, we propose IL-GAN, an alternative training framework for generative adversarial networks (GANs), which leverages incremental learning (IL) to enable the generation to learn the tail behavior of the distribution using only a few samples. We validate the proposed IL-GAN with data from 5G simulations on a factory automation scenario and real measurements gathered from various video streaming platforms. Our evaluations show that, compared to the state-of-the-art, our solution can significantly improve the learning and generation performance, not only for the tail distribution but also for the rest of the distribution.

Distributed artificial intelligence (AI) has recently accomplished tremendous breakthroughs in various communication services, ranging from fault-tolerant factory automation to smart cities. When distributed learning is run over a set of wireless connected devices, random channel fluctuations, and the incumbent services simultaneously running on the same network affect the performance of distributed learning. In this paper, we investigate the interplay between distributed AI workflow and ultra-reliable low latency communication (URLLC) services running concurrently over a network. Using 3GPP compliant simulations in a factory automation use case, we show the impact of various distributed AI settings (e.g., model size and the number of participating devices) on the convergence time of distributed AI and the application layer performance of URLLC. Unless we leverage the existing 5G-NR quality of service handling mechanisms to separate the traffic from the two services, our simulation results show that the impact of distributed AI on the availability of the URLLC devices is significant. Moreover, with proper setting of distributed AI (e.g., proper user selection), we can substantially reduce network resource utilization, leading to lower latency for distributed AI and higher availability for the URLLC users. Our results provide important insights for future 6G and AI standardization.

Communication systems supporting cyber-physical production applications should satisfy stringent delay and reliability requirements. Violation of these requirements may result in faulty behavior of the system and cause significant economic losses. Although wireless communications enable mobility and easy maintenance to industrial networks, it introduces many challenges to high-performance control systems due to interference and harsh environments (e.g., vibrations and many metallic objects). Diversity techniques and power control are powerful approaches to reduce latency and enhance reliability at the expense of excessive resource usage due to redundant transmissions. In this paper, we adopt fundamental metrics from reliability literature to wireless communications and provide critical indicators to measure reliability key performance indicators (KPIs) of cyber-physical systems. Then, we design a deep reinforcement learning orchestrator for power control and hybrid automatic repeat request retransmissions to optimize our reliability KPIs. Our orchestrator enables near real-time control and can be implemented on the edge cloud. We implement our framework on 3GPP compliant simulator on a factory automation scenario. Our comprehensive experiments show that, compared to the state-of-the-art, our solution can substantially improve the performance, especially for 5th percentile availability.

The performance prediction of user equipment (UE) metrics has many applications in the 5G era and beyond. For instance, throughput prediction can improve carrier selection, adaptive video streaming's quality of experience (QoE), and traffic latency. Many studies suggest distributed learning algorithms (e.g., federated learning (FL)) for this purpose. However, in a multi-tier design, features are measured in different tiers, e.g., UE tier, and gNodeB (gNB) tier. On one hand, neglecting the measurements in one tier results in inaccurate predictions. On the other hand, transmitting the data from one tier to another improves the prediction performance at the expense of increasing network overhead and privacy risks. In this paper, we propose cascaded FL to enhance UE throughput prediction with minimum network footprint and privacy ramifications (if any). The idea is to introduce feedback to conventional FL, in multi-tier architectures. Although we use cascaded FL for UE prediction tasks, the idea is rather general and can be used for many prediction problems in multi-tier architectures, such as cellular networks. We evaluate the performance of cascaded FL by detailed and 3GPP compliant simulations of London's city center. Our simulations show that the proposed cascaded FL can achieve up to 54% improvement over conventional FL in the normalized gain, at the cost of 1.8 MB (without quantization) and no cost with quantization.

Cyber-physical control applications impose strict requirements on the reliability and latency of the underlying communication system. Hence, they have been mostly implemented using wired channels where the communication service is highly predictable. Nevertheless, fulfilling such stringent demands is envisioned with the fifth generation of mobile networks (5G). The requirements of such applications are often defined on the application layer. However, cyber-physical control applications can usually tolerate sparse packet loss, and therefore it is not at all obvious what configurations and settings these application level requirements impose on the underlying wireless network. In this paper, we apply the fundamental metrics from reliability literature to wireless communications and derive a mapping function between application level requirements and network level parameters for those metrics under deterministic arrivals. Our mapping function enables network designers to realize the end-to-end performance (as the target application observes it). It provides insights to the network controller to either enable more reliability enhancement features (e.g., repetition), if the metrics are below requirements, or to enable features increasing network utilization, otherwise. We evaluate our theoretical results by realistic and detailed simulations of a factory automation scenario. Our simulation results confirm the viability of the theoretical framework under various burst error tolerance and load conditions.