How well can one expect transfer learning to work in a new setting where the domain is shifted, the task is different, and the architecture changes? Many transfer learning metrics have been proposed to answer this question. But how accurate are their predictions in a realistic new setting? We conducted an extensive evaluation involving over 42,000 experiments comparing 23 transferability metrics across 16 different datasets to assess their ability to predict transfer performance for image classification tasks. Our findings reveal that none of the existing metrics perform well across the board. However, we find that a simple k-nearest neighbor evaluation – as is commonly used to evaluate feature quality for self-supervision – not only surpasses existing metrics, but also offers better computational efficiency and ease of implementation.

The deep learning field is converging towards the use of general foundation models that can be easily adapted for diverse tasks. While this paradigm shift has become common practice within the field of natural language processing, progress has been slower in computer vision. In this paper we attempt to address this issue by investigating the transferability of various state-of-the-art foundation models to medical image classification tasks. Specifically, we evaluate the performance of five foundation models, namely Sam, Seem, Dinov2, BLIP, and OpenCLIP across four well-established medical imaging datasets. We explore different training settings to fully harness the potential of these models. Our study shows mixed results. Dinov2 consistently outperforms the standard practice of ImageNet pretraining. However, other foundation models failed to consistently beat this established baseline indicating limitations in their transferability to medical image classification tasks.

Artificial intelligence (AI) is progressively influencing business, science, and society, leading to major socioeconomic changes. However, its application in real-world problems varies significantly across different sectors. One of the primary challenges limiting the widespread adoption of AI in certain areas is data availability. Medical image analysis is one of these domains, where the process of gathering data and labels is often challenging or even infeasible due to legal and privacy concerns, or due to the specific characteristics of diseases. Logistical obstacles, expensive diagnostic methods and the necessity for invasive procedures add to the difficulty of data collection. Even when ample data exists, the substantial cost and logistical hurdles in acquiring expert annotations pose considerable challenges. Thus, there is a pressing need for the development of AI models that can operate in low-data settings.

In this thesis, we explore methods that improve the generalization and robustness of models when data availability is limited. We highlight the importance of model architecture and initialization, considering their associated assumptions and biases, to determine their effectiveness in such settings. We find that models with fewer built-in assumptions in their architecture need to be initialized with pre-trained weights, executed via transfer learning. This prompts us to explore how well transfer learning performs when models are initially trained in the natural domains, where data is abundant, before being used for medical image analysis where data is limited. We identify key factors responsible for transfer learning’s efficacy, and explore its relationship with data size, model architecture, and the distance between the target domain and the one used for pretraining. In cases where expert labels are scarce, we introduce the concept of complementary labels as the means to expand the labeling set. By providing information about other objects in the image, these labels help develop richer representations, leading to improved performance in low-data regimes. We showcase the utility of these methods by streamlining the histopathology-based assessment of chronic kidney disease in an industrial pharmaceutical setting, reducing the turnaround time of study evaluations by 97%. Our results demonstrate that AI models developed for low data regimes are capable of delivering industrial-level performance, proving their practical use in drug discovery and healthcare.

Artificiell intelligens (AI) påverkar gradvis allt fler domäner såsom affärsvärlden, vetenskapsvärlden och samhället i stort, vilket leder till stora socioekonomiska förändringar.Dock varierar dess tillämpning i verkliga problem avsevärt mellan olika sektorer.En av de främsta utmaningarna som begränsar den breda adoptionen av AI inom vissa områden är tillgången på data.Analys av medicinska bilder är en av dessa domäner, där möjligheten att samla data och annoteringar ofta är begränsad eller till och med omöjlig på grund av juridiska och integritetsmässiga skäl, eller på grund av specifika sjukdomskaraktäristiska problem.Logistiska hinder, dyra diagnostiska metoder och behovet av invasiva procedurer försvårar ytterligare datainsamling.Även när det finns gott om data utgör den betydande kostnaden och logistiska hinder för att skaffa expertannotationer betydande utmaningar.Således finns det ett tydligt behov för utvecklingen av AI-modeller som kan även fungera i med begränsade mängder data.

I denna avhandling utforskar vi metoder som förbättrar generaliseringen och robustheten hos modeller när tillgången på data är begränsad.Vi betonar vikten av modellarkitektur och initialisering, med fokus på aspekter som inbyggda antaganden, för att avgöra deras effektivitet under sådana förhållanden.Vi finner att modeller med färre inbyggda antaganden i sin arkitektur behöver initialiseras med förtränade vikter, genomfört via överföringsinlärning.Detta leder oss till att utforska hur väl överföringsinlärning presterar när modeller initialt tränas inom de naturliga domänerna, där data är rikligt tillgänglig, innan de används för analys av medicinska bilder där data är begränsad.Vi identifierar nyckelfaktorer som påverkar överföringsinlärningens effektivitet och utforskar påverkan som datasetsstorlek, modellarkitektur och avståndet mellan måldomänen och den som används för förträning.I fall där få expertannoteringar är tillgängliga introducerar vi konceptet kompletterande annoteringar, som en strategi för att utöka annoteringssättet.Genom att tillhandahålla information om andra objekt i bilden hjälper dessa annoteringar till att utveckla rikare representationer, vilket leder till förbättrad prestanda i domäner med begränsade mängder data.Vi visar användbarheten av dessa metoder genom att effektivisera histopatologi-baserad utvärderingen av kronisk njursjukdom i en industriell miljö, vilket reducerar tiden för studieutvärderingar med 97%.Våra resultat demonstrerar att AI-modeller utvecklade för förhållanden med små datamängder är kapabla att leverera effektivisering i industriell relevanta situationer, vilket visar på dess praktiska användbarhet inom läkemedelsupptäckt och hälsovård.

High Content Imaging (HCI) plays a vital role in modern drug discovery and development pipelines, facilitating various stages from hit identification to candidate drug characterization. Applying machine learning models to these datasets can prove challenging as they typically consist of multiple batches, affected by experimental variation, especially if different imaging equipment have been used. Moreover, as new data arrive, it is preferable that they are analyzed in an online fashion. To overcome this, we propose CODA, an online self-supervised domain adaptation approach. CODA divides the classifier’s role into a generic feature extractor and a task-specific model. We adapt the feature extractor’s weights to the new domain using cross-batch self-supervision while keeping the task-specific model unchanged. Our results demonstrate that this strategy significantly reduces the generalization gap, achieving up to a 300% improvement when applied to data from different labs utilizing different microscopes. CODA can be applied to new, unlabeled out-of-domain data sources of different sizes, from a single plate to multiple experimental batches.

The Soi crater region (0° to 60°N, 180°W to −110°W), which includes the well-preserved Soi crater in its center, spans a region from Titan's aeolian-dominated equatorial regions to fluvially-dominated high northern latitudes. This provides a rich diversity of landscapes, one that is also representative of the diversity encountered across Titan. Schoenfeld et al. (2023) mapped this region at 1:800,000 scale and produced a geomorphological map showing that the area consists of 22 types of geomorphological units. The Visual and Infrared Mapping Spectrometer (VIMS) coverage of the region enabled the detailed analysis of spectra of 261 different locations using a radiative transfer technique and a mixing model, yielding compositional constraints on Titan's optical surface layer. Additional constraints on composition on the near-surface substrate were obtained from microwave emissivity. We have derived combinations of top surface materials between dark materials, tholins, water-ice, and methane suggesting that dark mobile organic material at equatorial and high latitudes indicates “young” terrains and compositions, while tholin/water-ice mixtures that dominate areas around latitude 35°N show a material that is older plains deposits that we interpret to be the end stage of aeolian and fluvial transport and deposition. We found no spectral evidence of CO2, HC3N, and NH3 ice. We use the stratigraphic relations between the various mapping units and the relation between the geomorphology and the composition of the surface layers to build hypotheses on the origin and evolution of the regional geology. We suggest that sedimentary deposits, likely aeolian, are dominant in the region with fluvial activity and leaching changing the nature of the top surfaces of the midlatitude areas of the Soi crater region.

We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to 37× faster training and up to 26× reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.

High content imaging assays can capture rich phenotypic response data for large sets of compound treatments, aiding in the characterization and discovery of novel drugs. However, extracting representative features from high content images that can capture subtle nuances in phenotypes remains challenging. The lack of high-quality labels makes it difficult to achieve satisfactory results with supervised deep learning. Self-Supervised learning methods have shown great success on natural images, and offer an attractive alternative also to microscopy images. However, we find that self-supervised learning techniques underperform on high content imaging assays. One challenge is the undesirable domain shifts present in the data known as batch effects, which are caused by biological noise or uncontrolled experimental conditions. To this end, we introduce Cross-Domain Consistency Learning (CDCL), a self-supervised approach that is able to learn in the presence of batch effects. CDCL enforces the learning of biological similarities while disregarding undesirable batch-specific signals, leading to more useful and versatile representations. These features are organised according to their morphological changes and are more useful for downstream tasks – such as distinguishing treatments and mechanism of action.

Vision transformers have demonstrated the potential to outperform CNNs in a variety of vision tasks. But the computational and memory requirements of these models prohibit their use in many applications, especially those that depend on high-resolution images, such as medical image classification. Efforts to train ViTs more efficiently are overly complicated, necessitating architectural changes or intricate training schemes. In this work, we show that standard ViT models can be efficiently trained at high resolution by randomly dropping input image patches. This simple approach, PatchDropout, reduces FLOPs and memory by at least 50% in standard natural image datasets such as IMAGENET, and those savings only increase with image size. On CSAW, a high-resolution medical dataset, we observe a 5. savings in computation and memory using PatchDropout, along with a boost in performance. For practitioners with a fixed computational or memory budget, PatchDropout makes it possible to choose image resolution, hyperparameters, or model size to get the most performance out of their model.

Transfer learning is a standard technique to transfer knowledge from one domain to another. For applications in medical imaging, transfer from ImageNet has become the de-facto approach, despite differences in the tasks and image characteristics between the domains. However, it is unclear what factors determine whether - and to what extent transfer learning to the medical domain is useful. The longstanding assumption that features from the source domain get reused has recently been called into question. Through a series of experiments on several medical image benchmark datasets, we explore the relationship between transfer learning, data size, the capacity and inductive bias of the model, as well as the distance between the source and target domain. Our findings suggest that transfer learning is beneficial in most cases, and we characterize the important role feature reuse plays in its success.

Evidence suggests that networks trained on large datasets generalize well not solely because of the numerous training examples, but also class diversity which encourages learning of enriched features. This raises the question of whether this remains true when data is scarce - is there an advantage to learning with additional labels in low-data regimes In this work, we consider a task that requires difficult-To-obtain expert annotations: Tumor segmentation in mammography images. We show that, in low-data settings, performance can be improved by complementing the expert annotations with seemingly uninformative labels from non-expert annotators, turning the task into a multi-class problem. We reveal that these gains increase when less expert data is available, and uncover several interesting properties through further studies. We demonstrate our findings on CSAW-S, a new dataset that we introduce here, and confirm them on two public datasets.