Spectral bias, the tendency of neural networks to learn low-frequency features first, is a well-known issue with many training algorithms for physics-informed neural networks (PINNs). To overcome this issue, we propose IFeF-PINN, an algorithm for iterative training of PINNs with Fourier-enhanced features. The key idea is to enrich the latent space using high-frequency components through Random Fourier Features. This creates a two-stage training problem: (i) estimate a basis in the feature space, and (ii) perform regression to determine the coefficients of the enhanced basis functions. For an underlying linear model, it is shown that the latter problem is convex, and we prove that the iterative training scheme converges. Furthermore, we empirically establish that Random Fourier Features enhance the expressive capacity of the network, enabling accurate approximation of high-frequency PDEs. Through extensive numerical evaluation on classical benchmark problems, the superior performance of our method over state-of-the-art algorithms is shown, and the improved approximation across the frequency domain is illustrated.

The use of conventional model-based frameworks for design of estimation and control algorithms is precluded by the increasing complexity of engineering systems. To address this challenge, in this thesis we study new paradigms for data-driven modelling of dynamical systems based on deep learning methods. Three contributions are developed on fundamental algorithms and on applications of the techniques.

Our first and main contribution is a method for learning continuous-time simulation models of control systems. The method uses a novel neural network architecture to approximate the solution operator of such systems. The resulting model enables efficient simulation of trajectories, and is differentiable, meaning one can easily compute gradients of trajectories with respect to initial conditions, system parameters, and external inputs. These properties are desirable for model-based optimisation tasks such as parameter estimation, and we show that our method is highly effective in such contexts. We consider surrogate modelling of excitable systems, provide an extension to spatiotemporal control systems, and prove that the architecture is a universal approximator of flows of a general class of control systems.

In the second contribution we develop physics-informed learning methods for partial differential equations. We propose a new method that addresses the spectral bias problem, with performance superior to the state of the art on several baselines. In an application to traffic systems, we provide a method to learn solutions and equation parameters when measurements are sparse.

The third contribution is on the use of learning-based simulation models in unmanned vehicle operations for ocean observation, specifically for mission planning, adaptive sampling, and trajectory planning, in which simulation of dynamical systems plays a key role. We develop algorithms for sampling ocean fronts using a single vehicle, showing in numerical and software-in-the-loop simulations that we are able to track fronts using prior data. We propose mission and trajectory optimisation methods for multi-stage missions with multiple vehicles, developing an efficient dynamic programming algorithm with an associated software architecture.

Den ständigt ökande komplexiteten hos moderna tekniska system utesluter användandet av konventionella modelleringsramverk för design, reglering och estimering. För att hantera dessa utmaningar studerar vi i denna avhandling nya metoder för modellering av dynamiska system baserade på djupinlärningsmetoder.

Vårt huvudsakliga bidrag är en ny arkitektur och metod för tidskontinuerlig modellering av styrsystem. De resulterande modellerna är mycket effektiva i jämförelse med traditionella numeriska integrationsmodeller. De är därtill differentierbara, vilket innebär att man enkelt och på ett tillförlitligt sätt kan beräkna gradienten av tillståndet med avseende på initialvärden, parametrar och insignaler. Dessa använder dessutom endast gängse maskininlärningskomponenter och är därav enkla att implementera. Vi tillämpar metoden på system med flerskaliga och högfrekventa tidssvar samt utvidgar den till spatial-temporala system och bevisar att arkitekturen kan approximera lösningsoperatorer för en allmän klass av styrsystem.

I ett andra bidrag utvecklar vi metoder för lärande av lösningar av partiella differentialekvationer. Vi föreslår en ny arkitektur och träningsmetod som förbättrar avbildningen av högfrekventa och flerskaliga komponenter i lösningen, samt tar fram en lösningsmetod för fall där mätningarna är glesa.

Vi förutser att sådana modeller kan användas i operationer med obemannade farkoster för havsobservation, samt betraktar tre utmaningar i samband med denna tillämpning, nämligen uppdragsplanering, banplanering och adaptiv sampling, där simulering av dynamiska system spelar en nyckelroll. I avhandlingens tredje bidrag utvecklar vi algoritmer för sampling av havsfronter, samt uppdragsplanering och banoptimering för komplexa uppdrag med flera havsfarkoster.

Learning flow functions of continuous-time control systems is considered in this thesis. The flow function is the operator mapping initial states and control inputs to the state trajectories, and the problem is to find a suitable neural network architecture to learn this infinite-dimensional operator from measurements of state trajectories. The main motivation is the construction of continuous-time simulation models for such systems. The contribution is threefold.

We first study the design of neural network architectures for this problem, when the control inputs have a certain discrete-time structure, inspired by the classes of control inputs commonly used in applications. We provide a mathematical formulation of the problem and show that, under the considered input class, the flow function can be represented exactly in discrete time. Based on this representation, we propose a discrete-time recurrent neural network architecture. We evaluate the architecture experimentally on data from models of two nonlinear oscillators, namely the Van der Pol oscillator and the FitzHugh-Nagumo oscillator. In both cases, we show that we can train models which closely reproduce the trajectories of the two systems.

Secondly, we consider an application to spiking systems. Conductance-based models of biological neurons are the prototypical examples of this type of system. Because of their multi-timescale dynamics and high-frequency response, continuous-time representations which are efficient to simulate are desirable. We formulate a framework for surrogate modelling of spiking systems from trajectory data, based on learning the flow function of the system. The framework is demonstrated on data from models of a single biological neuron and of the interconnection of two neurons. The results show that we are able to accurately replicate the spiking behaviour.

Finally, we prove an universal approximation theorem for the proposed recurrent neural network architecture. First, general conditions are given on the flow function and the control inputs which guarantee that the architecture is able to approximate the flow function of any control system with arbitrary accuracy. Then, we specialise to systems with dynamics given by a controlled ordinary differential equation, showing that the conditions are satisfied whenever the equation has a continuously differentiable right-hand side, for the control input classes of interest.

Denna avhandling studerar maskininlärningsmetoder för tidskontinuerliga reglersystem. Vi utgår från en abstrakt systemrepresentation med en lösningsoperator, som avbildar systemets initialtillstånd och insignal på motsvarande tillståndstrajektorian. Målet är att undersöka inlärning av tidskontinuerliga simuleringsmodeller utifrån tillståndsmätningar. Avhandlingen består av tre huvudbidrag.

Vi undersöker först arkitekturer baserade på neurala nätverk, för klasser av insignaler som är brukliga i tillämpningar och har en viss tidsdiskret struktur. Vi formulerar problemet matematiskt, och visar att lösningsoperatorn kan representeras exakt av ett tidsdiskret system. Detta leder till en arkitektur baserad på ett återkopplande neuralt nätverk (RNN), som vi utförligt beskriver, analyserar och validerar med hjälp av data från två modeller av icke-linjära oscillatorer, nämligen Van der Pol oscillatorn och FitzHugh-Nagumo oscillatorn. I båda fall visar vi att vi kan träna modeller som noggrant reproducerar systemens lösningsbanor.

Därefter studerar vi en tillämpning på system vars tillståndstrajektorier kännetecknas av förekomsten av snabba oscillationer i form av impulser, såsom modeller av biologiska neuroner. Denna klass av system karakteriseras av ett flerskaligt och högfrekvent tidssvar, vilket gör det önskvärt att ta fram tidskontinuerliga modeller som är lätta att simulera. Vi lägger fram ett ramverk för inlärning av surrogatmodeller av sådana system från data. Ramverket demonstreras med hjälp av data från en modell av en biologisk neuron och en modell av två kopplade biologiska neuroner, och resultaten visar att våra modeller noggrant reproducerar systemens beteende.

Slutligen tar vi fram ett bevis för ett approximationsteorem för inlärning av lösningsoperatorer av tidskontinuerliga system. Vi visar att den RNN- arkitektur som vi har tagit fram kan approximera godtyckliga reglersystem under vissa villkor som vi först formulerar abstrakt. Sedan bevisar att reglersystem som beskrivs av ordinära differentialekvationer uppfyller dessa villkor, vilket betyder att de kan approximeras av den studerade arkitekturen.

We propose a framework for surrogate modelling of spiking systems. These systems are often described by stiff differential equations with high-amplitude oscillations and multi-timescale dynamics, making surrogate models an attractive tool for system design and simulation. We parameterise the flow function of a spiking system using a recurrent neural network architecture, allowing for a direct continuous-time representation of the state trajectories. The spiking nature of the signals makes for a data-heavy and computationally hard training process; thus, we describe two methods to mitigate these difficulties. We demonstrate our framework on two conductance-based models of biological neurons, showing that we are able to train surrogate models which accurately replicate the spiking behaviour.

This paper proposes a method that uses satellite data to improve adaptive sampling missions. We find and track algal bloom fronts using an autonomous underwater vehicle (AUV) equipped with a sensor that measures the concentration of chlorophyll a. Chlorophyll a concentration indicates the presence of algal blooms. The proposed method learns the kernel parameters of a Gaussian process model using satellite images of chlorophyll a from previous days. The AUV estimates the chlorophyll a concentration online using locally collected data. The algal bloom front estimate is fed to the motion control algorithm. The performance of this method is evaluated through simulations using a real dataset of an algal bloom front in the Baltic. We consider a real-world scenario with sensor and localization noise and with a detailed AUV model.

We describe a recurrent neural network (RNN) based architecture to learn the flow function of a causal, time-invariant and continuous-time control system from trajectory data. By restricting the class of control inputs to piecewise constant functions, we show that learning the flow function is equivalent to learning the input-to-state map of a discrete-time dynamical system. This motivates the use of an RNN together with encoder and decoder networks which map the state of the system to the hidden state of the RNN and back. We show that the proposed architecture is able to approximate the flow function by exploiting the system's causality and time-invariance. The output of the learned flow function model can be queried at any time instant. We experimentally validate the proposed method using models of the Van der Pol and FitzHugh-Nagumo oscillators. In both cases, the results demonstrate that the architecture is able to closely reproduce the trajectories of these two systems. For the Van der Pol oscillator, we further show that the trained model generalises to the system's response with a prolonged prediction time horizon as well as control inputs outside the training distribution. For the FitzHugh-Nagumo oscillator, we show that the model accurately captures the input-dependent phenomena of excitability.

We consider the problem of approximating flow functions of continuous-time dynamical systems with inputs. It is well-known that continuous-time recurrent neural networks are universal approximators of this type of system. In this paper, we prove that an architecture based on discrete-time recurrent neural networks universally approximates flows of continuous-time dynamical systems with inputs. The required assumptions are shown to hold for systems whose dynamics are well-behaved ordinary differential equations and with practically relevant classes of input signals. This enables the use of off-the-shelf solutions for learning such flow functions in continuous-time from sampled trajectory data.

We consider the problem of tracking moving algal bloom fronts using an unmanned surface vehicle (USV) equipped with a sensor that measures the concentration of chlorophyll a. Chlorophyll a is a green pigment found in plants, and its concentration is an indicator of phytoplankton abundance. Our algal bloom front tracking mission consists of three stages: deployment, data collection, and front tracking. At the deployment stage, a satellite collects an image of the sea from which the location of the front, the reference value for the concentration at this front and, consequently, the appropriate initial position for the USV are determined. At the data collection stage, the USV collects data points to estimate the local algal gradient as it crosses the front. Finally, at the front tracking stage, an adaptive algorithm based on recursive least squares fitting using recent past sensor measures is executed. We evaluate the performance of the algorithm and its sensitivity to measurement noise through MATLAB simulations. We also present an implementation of the algorithm on the DUNE onboard software platform for marine robots and validate it using simulations with satellite model forecasts from Baltic sea data.

Marine pollution incidents can have a huge impact on different ecosystems, with unpredictable short- and long-term consequences. Once the pollutant is detected, it is critical to quickly understand its characteristics so that authorities can lay out an adequate response. In parallel to the time- and cost-constrained traditional operational means, this paper suggests the use of AUVs for the sampling procedures of marine pollution incidents, to increase the speed and efficiency of operations. A new software architecture is developed, integrating trajectory optimization for AUVs into a software toolchain that allows human operators to plan, simulate, and control multiple vehicles deployments. A method to optimize AUVs deployment position and time is also presented. The overall architecture is simulated using high-resolution hydrodynamic data from the Ria de Aveiro lagoon and the adjacent coastal area, in Aveiro, Portugal. 

We consider the problem of traffic density reconstruction using measurements from probe vehicles (PVs) with a low penetration rate. In other words, the number of sensors is small compared to the number of vehicles on the road. The model used assumes noisy measurements and a partially unknown first-order model. All these considerations make the use of machine learning to reconstruct the state the only applicable solution. We first investigate how the identification and reconstruction processes can be merged and how a sparse dataset can still enable a good identification. Secondly, we propose a pre-training procedure that aids the hyperparameter tuning, preventing the gradient descent algorithm from getting stuck at saddle points. Examples using numerical simulations and the SUMO traffic simulator show that the reconstructions are close to the real density in all cases.