In this article, an iterative deterministic policy gradient method for finding optimal strategies in the presence of market impact is introduced. The derivation of the policy gradient sheds light on a proper way of handling the market impact of trades in the context of reinforcement learning. Similar to many machine learning methods, the proposed deterministic policy gradient method is based on mini-batch stochastic gradient descent optimization. The method is demonstrated to consistently perform well for several different objectives and market dynamics when applied to the financial applications of order execution and option hedging.

Recent advancements in machine learning have opened up new possibilities for algorithmic trading, enabling the optimization of trading strategies in complex market environments. This thesis aims to improve algorithmic trading methods by developing machine learning models for the realistic simulation of limit order books and the learning of optimal strategies. Consisting of three papers, the thesis combines theoretical insights with practical applications.

The first paper presents a generative model for the dynamic evolution of a limit order book, using recurrent neural networks. The model captures the complete dynamics of the limit order book by decomposing the probability of each transition of the limit order book into a product of conditional probabilities for order type, price level, order size, and time delay. Each of these conditional probabilities is modeled by a recurrent neural network. Additionally, the paper introduces several evaluation metrics for generative models related to order execution. The generative model is trained on both synthetic data generated by a Markov model and real data from the Nasdaq Stockholm exchange.

The second paper proposes an iterative deterministic policy gradient method for stochastic control problems in finance, which incorporates both temporary and permanent market impact. The method is based on a derived policy gradient theorem and uses mini-batch stochastic gradient descent for optimization. It is applied to both order execution and option hedging, demonstrating consistently strong performance across several objectives and market dynamics. 

The third paper studies a policy gradient method with parameter-based exploration, where a single deterministic policy is sampled at the beginning of an episode and used throughout the whole episode. A marginal equivalence between parameter-based and action-based exploration is shown, facilitating the adaption of previously established convergence results for policy gradient methods with action-based exploration. Convergence rates to first-order stationary points are derived under mild assumptions, and global convergence is established under an introduced Fisher-non-degenerate condition for parameter-based exploration.

Den senaste tidens framsteg inom maskininlärning har skapat nya förutsättningar för algoritmisk handel och möjliggjort optimering av handelsstrategier i komplexa miljöer. Syftet med denna avhandling är att förbättra metoder för algorithmisk handel genom att utveckla modeller baserade på maskininlärning för realistisk simulering av orderböcker samt för lärande av optimala strategier. Avhandlingen består av tre artiklar och kombinerar teoretiska insikter med praktiska tillämpningar.

Den första artikeln utvecklar en generativ modell för den dynamiska utvecklingen av en orderbok baserad på rekurrenta neurala nätverk. Modellen fångar orderbokens fullständiga dynamik genom att bryta ner sannolikheten för varje förändring av orderboken i en produkt av betingade sannolikheter för ordertyp, prisnivå, orderstorlek och tidsfördröjning. Var och en av dessa betingade sannolikheter modelleras med ett rekurrent neuralt nätverk. Dessutom introducerar artikeln flera evalueringsmetoder för generativa modeller relaterade till orderexekvering. Den generativa modellen tränas framgångsrikt både för syntetisk data, genererad av en Markovmodell, och verklig data från Nasdaq Stockholm.

Den andra artikeln föreslår en iterativ deterministisk policygradientmetod för stokastiska kontrollproblem inom finans, som inkluderar både temporär och permanent marknadspåverkan. Metoden är baserad på ett härlett policy gradient teorem och använder stokastisk gradientnedstigning för optimering. Den tillämpas framgångsrikt på både orderexekvering och optionshedging och visar konsekvent bra resultat för varierande objektiv och marknadsdynamik.

Den tredje artikeln studerar en policygradientmetod med parameterbaserad utforskning, där en enda deterministisk policy väljs slumpmässigt i början av en episod och används under hela episoden. En ekvivalens mellan parameterbaserad och handlingsbaserad utforskning visas, vilket möjliggör anpassning av tidigare etablerade konvergensresultat för policygradientmetoder med handlingsbaserad utforskning. Konvergenshastigheter till första ordningens stationära punkter härleds under milda antaganden, och global konvergens etableras under ett introducerat villkor gällande Fisher-icke-degenerering för parameterbaserad utforskning.

In this work, a generative model based on recurrent neural networks for the complete dynamics of a limit order book is developed. The model captures the dynamics of the limit order book by decomposing the probability of each transition into a product of conditional probabilities of order type, price level, order size and time delay. Each such conditional probability is modelled by a recurrent neural network. Several evaluation metrics for generative models related to trading execution are introduced. Using these metrics, it is demonstrated that the generative model can be successfully trained to fit both synthetic and real data from the Nasdaq Stockholm exchange.

We argue that when comparing two graphs, the distribution of node structural features is more informative than global graph statistics which are often used in practice, especially to evaluate graph generative models. Thus, we present GraphDCA - a framework for evaluating similarity between graphs based on the alignment of their respective node representation sets. The sets are compared using a recently proposed method for comparing representation spaces, called Delaunay Component Analysis (DCA), which we extend to graph data. To evaluate our framework, we generate a benchmark dataset of graphs exhibiting different structural patterns and show, using three node structure feature extractors, that GraphDCA recognizes graphs with both similar and dissimilar local structure. We then apply our framework to evaluate three publicly available real-world graph datasets and demonstrate, using gradual edge perturbations, that GraphDCA satisfyingly captures gradually decreasing similarity, unlike global statistics. Finally, we use GraphDCA to evaluate two state-of-the-art graph generative models, NetGAN and CELL, and conclude that further improvements are needed for these models to adequately reproduce local structural features.

We analyze the convergence of the Policy Gradient algorithm with Parameter-based Exploration (PGPE). This algorithm employs a parameterized sampling procedure to select a deterministic policy at the beginning of each episode, which is then executed throughout the episode. The parameter is updated using stochastic gradient ascent. A key advantage of PGPE over other policy gradient methods is its ability to learn optimal deterministic policies without requiring any smoothness assumptions on these policies (e.g., it can learn optimal threshold-based policies). We investigate the convergence rate of PGPE to a stationary point under mild assumptions on the class of sampling procedures. This rate is illustrated for sampling procedures using specific types of parameterizations, including product, softmax, and Gaussian parameterizations. Finally, we extend the Fisher-non-degenerate condition, used in earlier work for action-based policies, to PGPE sampling procedures and demonstrate that under this condition, PGPE attains an ϵ-optimal stationary point with a sample complexity scaling as Õ(ϵ⁻³).