Network digital twin (NDT) models are virtual models that replicate the behavior of physical communication networks and are considered a key technology component to enable novel features and capabilities in future 6G networks. In this work, we focus on communication-aware control applications and study NDTs that model the communication quality properties of a multi-cell, dynamically changing wireless network over a workspace populated with multiple moving users. We propose an NDT modeled as a hybrid system, where each mode corresponds to a different base station and comprises sub-modes that correspond to areas of the workspace with similar network characteristics. The proposed hybrid NDT is identified and continuously improved through an annealing optimization-based learning algorithm, driven by online data measurements collected by the users. The advantages of the proposed hybrid NDT are studied with respect to memory and computational efficiency, data consumption, and the ability to timely adapt to network changes. Finally, we validate the proposed methodology on real experimental data collected from a two-cell 5G testbed.

We introduce a real-time identification method for discrete-time state-dependent switching systems in both the input–output and state-space domains. In particular, we design a system of adaptive algorithms running in two timescales; a stochastic approximation algorithm implements an online deterministic annealing scheme at a slow timescale and estimates the mode-switching signal, and a recursive identification algorithm runs at a faster timescale and updates the parameters of the local models based on the estimate of the switching signal. We first focus on piece-wise affine systems and discuss identifiability conditions and convergence properties based on the theory of two-timescale stochastic approximation. In contrast to standard identification algorithms for switched systems, the proposed approach gradually estimates the number of modes and is appropriate for real-time system identification using sequential data acquisition. The progressive nature of the algorithm improves computational efficiency and provides real-time control over the performance-complexity trade-off. Finally, we address specific challenges that arise in the application of the proposed methodology in identification of more general switching systems. Simulation results validate the efficacy of the proposed methodology.

We study the problem of approximating a model predictive controller (MPC) with learning models to facilitate real-time operation. In particular, we investigate how the use of a hybrid learning model can tighten the statistical learning bounds used for stability guarantees given by existing robust data-driven MPC approaches. We propose a hybrid learning framework with a finite set of state-dependent modes, each consisting of a supervised regression model. The mode-switching signal corresponds to a state space partition produced by solving a homotopy optimization problem that implicitly minimizes the Lipschitz constant of the regression model in each mode. The cardinality of the partition is decided by a bifurcation phenomenon, inducing a performance-complexity trade-off that is discussed. The proposed MPC approximation framework is validated on a nonlinear benchmark problem.

While reinforcement learning (RL) has shown experimental success in a number of applications, it is known to be sensitive to noise and perturbations in the parameters of the system, leading to high variability in the total reward amongst different episodes on slightly different environments. To introduce robustness, as well as sample efficiency, risk-sensitive RL methods are being thoroughly studied. In this work, we provide a definition of robust RL policies and formulate a risk-sensitive RL problem to approximate them, by solving an optimization problem with respect to a modified objective based on exponential criteria. In particular, we study a model-free risk-sensitive variation of the widely used Monte Carlo policy gradient algorithm, and introduce a novel risk-sensitive online Actor–Critic algorithm based on solving a multiplicative Bellman equation using stochastic approximation updates. Analytical results suggest that the use of exponential criteria generalizes commonly used ad-hoc regularization approaches, improves sample efficiency, and introduces robustness with respect to perturbations in the model parameters and the environment. The implementation, performance, and robustness properties of the proposed methods are evaluated in simulated experiments.

In this paper we propose an unpredictability-based jamming defense framework based on the principles of Moving Target Defense for a wireless communication problem. Taking advantage of the complex nature of large-scale cyber-physical systems, we consider a platform consisting of a single receiving component but multiple potential transmitting components, each equipped with a multi-antenna phased array. We formulate an optimization problem over the probability simplex that characterizes a randomized receiving angle which seeks to balance between the estimated performance of the transmission and an entropy-based unpredictability measure. Furthermore, we explore the effect of an intelligent adversary that has knowledge of the derived probabilities and optimally places a single-antenna jamming device to disrupt the communication links. Finally, simulation results showcase the efficacy of the proposed algorithm.

We introduce a constructive function approximation approach as a general tool, particularly useful in adaptive and data-driven methods for perception and control. The key idea is to estimate of a collection of simple local models as opposed to a single and complex regression model trained in the entire input space. We use principles from the Online Deterministic Annealing (ODA) optimization framework to construct an adaptive partition of the input space, which enables the introduction of local function approximation models within each subset of the partition. We show that both the partitioning and the local model training algorithms are stochastic approximation algorithms that operate online, and with the same observations, as part of a two-timescale stochastic approximation scheme. This process constitutes a heuristic method to gradually increase the complexity of the function approximation framework in a task-agnostic manner, giving emphasis to regions of the input space where the regression error is high. As a result this framework has inherent explainability properties, and is suitable for continuous learning applications where regression improvement without retraining from scratch is crucial. Simulation results illustrate the properties of the proposed approach.

We propose an online identification scheme for discrete-time piece-wise affine state-space models based on a system of adaptive algorithms running in two timescales. A stochastic approximation algorithm implements an online deterministic annealing scheme at a slow timescale, estimating the partition of the augmented state-input space that defines the switching signal. At the same time, an adaptive identification algorithm, running at a higher timescale, updates the parameters of the local models based on the estimate of the switching signal. Identifiability conditions for the switched system are discussed and convergence results are given based on the theory of two-timescale stochastic approximation. In contrast to standard identification algorithms for piece-wise affine systems, the proposed approach progressively estimates the number of modes needed and is appropriate for online system identification using sequential data acquisition. This progressive nature of the algorithm improves computational efficiency and provides real-time control over the performance-complexity trade-off, desired in practical applications. Experimental results validate the efficacy of the proposed methodology.

We investigate the problem of deceiving a malicious agent employing an identification method to estimate the closed-loop dynamics of a cyber-physical system. In particular, we propose a moving target defense mechanism that utilizes stochastic switching between linear closed-loop dynamics to drive a linear system identification process of a potential adversary to sub-optimal solutions with non-vanishing error. We provide a statistical analysis of the induced identification error and show that it is not possible for any linear system identification method to reconstruct the average dynamics of a stochastic switched linear system. Finally, we utilize the theory of Markov jump linear systems to guarantee asymptotic stability of the switching system, and formulate the switching control problem as an optimization problem that guarantees stability while taking into account the trade-off between security and switching effort. Simulation results showcase the efficacy of the proposed approach in inducing identification error for the adversary using minimal switching.