Learned iterative reconstruction methods have recently emerged as a powerful tool to solve inverse problems. These deep learning techniques for image reconstruction achieve remarkable speed and accuracy by combining hard knowledge about the physics of the image formation process, represented by the forward operator, with soft knowledge about how the reconstructions should look like, represented by deep neural networks. A diverse set of such methods have been proposed, and this chapter seeks to give an overview of their similarities and differences, as well as discussing some of the commonly used methods to improve their performance.

The paper considers the problem of performing a post-processing task defined on a model parameter that is only observed indirectly through noisy data in an ill-posed inverse problem. A key aspect is to formalize the steps of reconstruction and post-processing as appropriate estimators (non-randomized decision rules) in statistical estimation problems. The implementation makes use of (deep) neural networks to provide a differentiable parametrization of the family of estimators for both steps. These networks are combined and jointly trained against suitable supervised training data in order to minimize a joint differentiable loss function, resulting in an end-to-end task adapted reconstruction method. The suggested framework is generic, yet adaptable, with a plug-and-play structure for adjusting both the inverse problem and the post-processing task at hand. More precisely, the data model (forward operator and statistical model of the noise) associated with the inverse problem is exchangeable, e.g., by using neural network architecture given by a learned iterative method. Furthermore, any post-processing that can be encoded as a trainable neural network can be used. The approach is demonstrated on joint tomographic image reconstruction, classification and joint tomographic image reconstruction segmentation.

In this work, we consider methods for solving large-scale optimization problems with a possibly nonsmooth objective function. The key idea is to first parametrize a class of optimization methods using a generic iterative scheme involving only linear operations and applications of proximal operators. This scheme contains some modern primal-dual first-order algorithms like the Douglas-Rachford and hybrid gradient methods as special cases. Moreover, we show weak convergence of the iterates to an optimal point for a new method which also belongs to this class. Next, we interpret the generic scheme as a neural network and use unsupervised training to learn the best set of parameters for a specific class of objective functions while imposing a fixed number of iterations. In contrast to other approaches of "learning to optimize," we present an approach which learns parameters only in the set of convergent schemes. Finally, we illustrate the approach on optimization problems arising in tomographic reconstruction and image deconvolution, and train optimization algorithms for optimal performance given a fixed number of iterations.

Unpaired image-to-image translation has attracted significant interest due to the invention of CycleGAN, a method which utilizes a combination of adversarial and cycle consistency losses to avoid the need for paired data. It is known that the CycleGAN problem might admit multiple solutions, and our goal in this paper is to analyze the space of exact solutions and to give perturbation bounds for approximate solutions. We show theoretically that the exact solution space is invariant with respect to automorphisms of the underlying probability spaces, and, furthermore, that the group of automorphisms acts freely and transitively on the space of exact solutions. We examine the case of zero 'pure' CycleGAN loss first in its generality, and, subsequently, expand our analysis to approximate solutions for 'extended' CycleGAN loss where identity loss term is included. In order to demonstrate that these results are applicable, we show that under mild conditions nontrivial smooth automorphisms exist. Furthermore, we provide empirical evidence that neural networks can learn these automorphisms with unexpected and unwanted results. We conclude that finding optimal solutions to the CycleGAN loss does not necessarily lead to the envisioned result in image-to-image translation tasks and that underlying hidden symmetries can render the result utterly useless.

Model-based learned iterative reconstruction methods have recently been shown to outperform classical reconstruction algorithms. Applicability of these methods to large scale inverse problems is however limited by the available memory for training and extensive training times, the latter due to computationally expensive forward models. As a possible solution to these restrictions we propose a multi-scale learned iterative reconstruction scheme that computes iterates on discretisations of increasing resolution. This procedure does not only reduce memory requirements, it also considerably speeds up reconstruction and training times, but most importantly is scalable to large scale inverse problems with non-trivial forward operators, such as those that arise in many 3D tomographic applications. In particular, we propose a hybrid network that combines the multi-scale iterative approach with a particularly expressive network architecture which in combination exhibits excellent scalability in 3D. Applicability of the algorithm is demonstrated for 3D cone beam computed tomography from real measurement data of an organic phantom. Additionally, we examine scalability and reconstruction quality in comparison to established learned reconstruction methods in two dimensions for low dose computed tomography on human phantoms.

In this thesis on data-driven methods in inverse problems we introduce several new methods to solve inverse problems using recent advancements in machine learning and specifically deep learning. The main goal has been to develop practically applicable methods, scalable to medical applications and with the ability to handle all the complexities associated with them.

In total, the thesis contains six papers. Some of them are focused on more theoretical questions such as characterizing the optimal solutions of reconstruction schemes or extending current methods to new domains, while others have focused on practical applicability. A significant portion of the papers also aim to bringing knowledge from the machine learning community into the imaging community, with considerable effort spent on translating many of the concepts. The papers have been published in a range of venues: machine learning, medical imaging and inverse problems.

The first two papers contribute to a class of methods now called learned iterative reconstruction where we introduce two ways of combining classical model driven reconstruction methods with deep neural networks. The next two papers look forward, aiming to address the question of "what do we want?" by proposing two very different but novel loss functions for training neural networks in inverse problems. The final papers dwelve into the statistical side, one gives a generalization of a class of deep generative models to Banach spaces while the next introduces two ways in which such methods can be used to perform Bayesian inversion at scale.

Digital breast tomosynthesis is rapidly replacing digital mammography as the basic x-ray technique for evaluation of the breasts. However, the sparse sampling and limited angular range gives rise to different artifacts, which manufacturers try to solve in several ways. In this study we propose an extension of the Learned Primal Dual algorithm for digital breast tomosynthesis. The Learned Primal-Dual algorithm is a deep neural network consisting of several 'reconstruction blocks', which take in raw sinogram data as the initial input, perform a forward and a backward pass by taking projections and back-projections, and use a convolutional neural network to produce an intermediate reconstruction result which is then improved further by the successive reconstruction block. We extend the architecture by providing breast thickness measurements as a mask to the neural network and allow it to learn how to use this thickness mask. We have trained the algorithm on digital phantoms and the corresponding noise-free/noisy projections, and then tested the algorithm on digital phantoms for varying level of noise. Reconstruction performance of the algorithms was compared visually, using MSE loss and Structural Similarity Index. Results indicate that the proposed algorithm outperforms the baseline iterative reconstruction algorithm in terms of reconstruction quality for both breast edges and internal structures and is robust to noise.

Wasserstein Generative Adversarial Networks (WGANs) can be used to generate realistic samples from complicated image distributions. The Wasserstein metric used in WGANs is based on a notion of distance between individual images, which induces a notion of distance between probability distributions of images. So far the community has considered l(2) as the underlying distance. We generalize the theory of WGAN with gradient penalty to Banach spaces, allowing practitioners to select the features to emphasize in the generator. We further discuss the effect of some particular choices of underlying norms, focusing on Sobolev norms. Finally, we demonstrate a boost in performance for an appropriate choice of norm on CIFAR-10 and CelebA.

Energy-dispersive X-ray spectroscopy (EDS) tomography is an advanced technique to characterize compositional information for nanostructures in three dimensions (3D). However, the application is hindered by the poor image quality caused by the low signal-to-noise ratios and the limited number of tilts, which are fundamentally limited by the insufficient number of X-ray counts. In this paper, we explore how to make accurate EDS reconstructions from such data. We propose to augment EDS tomography by joining with it a more accurate high-angle annular dark-field STEM (HAADF-STEM) tomographic reconstruction, for which usually a larger number of tilt images are feasible. This augmentation is realized through total nuclear variation (TNV) regularization, which encourages the joint EDS and HAADF reconstructions to have not only sparse gradients but also common edges and parallel (or antiparallel) gradients. Our experiments show that reconstruction images are more accurate compared to the non-regularized and the total variation regularized reconstructions, even when the number of tilts is small or the X-ray counts are low.

We propose the Learned Primal-Dual algorithm for tomographic reconstruction. The algorithm accounts for a (possibly non-linear) forward operator in a deep neural network by unrolling a proximal primal-dual optimization method, but where the proximal operators have been replaced with convolutional neural networks. The algorithm is trained end-to-end, working directly from raw measured data and it does not depend on any initial reconstruction such as filtered back-projection (FBP). We compare performance of the proposed method on low dose computed tomography reconstruction against FBP, total variation (TV), and deep learning based post-processing of FBP. For the Shepp-Logan phantom we obtain >6 dB peak signal to noise ratio improvement against all compared methods. For human phantoms the corresponding improvement is 6.6 dB over TV and 2.2 dB over learned post-processing along with a substantial improvement in the structural similarity index. Finally, our algorithm involves only ten forward-back-projection computations, making the method feasible for time critical clinical applications.