Ovarian lesions are common and often incidentally detected. A critical shortage of expert ultrasound examiners has raised concerns of unnecessary interventions and delayed cancer diagnoses. Deep learning has shown promising results in the detection of ovarian cancer in ultrasound images; however, external validation is lacking. In this international multicenter retrospective study, we developed and validated transformer-based neural network models using a comprehensive dataset of 17,119 ultrasound images from 3,652 patients across 20 centers in eight countries. Using a leave-one-center-out cross-validation scheme, for each center in turn, we trained a model using data from the remaining centers. The models demonstrated robust performance across centers, ultrasound systems, histological diagnoses and patient age groups, significantly outperforming both expert and non-expert examiners on all evaluated metrics, namely F1 score, sensitivity, specificity, accuracy, Cohen’s kappa, Matthew’s correlation coefficient, diagnostic odds ratio and Youden’s J statistic. Furthermore, in a retrospective triage simulation, artificial intelligence (AI)-driven diagnostic support reduced referrals to experts by 63% while significantly surpassing the diagnostic performance of the current practice. These results show that transformer-based models exhibit strong generalization and above human expert-level diagnostic accuracy, with the potential to alleviate the shortage of expert ultrasound examiners and improve patient outcomes.

How well can one expect transfer learning to work in a new setting where the domain is shifted, the task is different, and the architecture changes? Many transfer learning metrics have been proposed to answer this question. But how accurate are their predictions in a realistic new setting? We conducted an extensive evaluation involving over 42,000 experiments comparing 23 transferability metrics across 16 different datasets to assess their ability to predict transfer performance for image classification tasks. Our findings reveal that none of the existing metrics perform well across the board. However, we find that a simple k-nearest neighbor evaluation – as is commonly used to evaluate feature quality for self-supervision – not only surpasses existing metrics, but also offers better computational efficiency and ease of implementation.

We investigate the role of uncertainty quantification in aiding medical decision-making. Existing evaluation metrics fail to capture the practical utility of joint human-AI decision-making systems. To address this, we introduce a novel framework to assess such systems and use it to benchmark a diverse set of confidence and uncertainty estimation methods. Our results show that certainty measures enable joint human-AI systems to outperform both standalone humans and AIs, and that for a given system there exists an optimal balance in the number of cases to refer to humans, beyond which the system’s performance degrades.

We investigate the role of uncertainty quantification in aiding medical decision-making. Existing evaluation metrics fail to capture the practical utility of joint human-AI decision-making systems. To address this, we introduce a novel framework to assess such systems and use it to benchmark a diverse set of confidence and uncertainty estimation methods. Our results show that certainty measures enable joint human-AI systems to outperform both standalone humans and AIs, and that for a given system there exists an optimal balance in the number of cases to refer to humans, beyond which the system's performance degrades.

In this work, we explore various computer vision techniques, with a focus on texture recognition approaches, for the task of plant species detection. We particularly emphasize the study of a challenging dataset consisting of 50 Brazilian plant species' leaf midrib cross-sections using microscope images. The research focuses on a recent method named Random Encoding of Aggregated Deep Activation Maps (RADAM) that leverages deep features from pre-trained Convolutional Neural Networks (CNNs) for improved plant species identification. This method demonstrates significant advancement over traditional texture analysis and deep learning approaches, showcasing the potential of combining deep feature engineering with texture analysis for accurate plant species recognition.

We introduce Learning from Offline Foundation Features with Tensor Augmentations (LOFF-TA), an efficient training scheme designed to harness the capabilities of foundation models in limited resource settings where their direct development is not feasible. LOFF-TA involves training a compact classifier on cached feature embeddings from a frozen foundation model, resulting in up to 37× faster training and up to 26× reduced GPU memory usage. Because the embeddings of augmented images would be too numerous to store, yet the augmentation process is essential for training, we propose to apply tensor augmentations to the cached embeddings of the original non-augmented images. LOFF-TA makes it possible to leverage the power of foundation models, regardless of their size, in settings with limited computational capacity. Moreover, LOFF-TA can be used to apply foundation models to high-resolution images without increasing compute. In certain scenarios, we find that training with LOFF-TA yields better results than directly fine-tuning the foundation model.

Artificial intelligence (AI) offers significant potential to enhance the accuracy and efficiency of medical diagnosis, monitoring, and treatment. In ovarian cancer, where 70% of cases are detected only at stage III or IV, AI-driven tools could enable earlier detection and improve patient outcomes.However, the safety-critical nature of medicine— where even minor errors can have serious consequences—has led to cautious adoption of AI technologies.To integrate AI  into clinical practice, it must not only demonstrate good performance, but also robustness and generalizability across diverse clinical settings.

This thesis investigates the development and evaluation of generalizable and robust AI systems, with a focus on medical image analysis. We begin by addressing key gaps in our understanding of how complexity influences generalization, exploring scaling laws across increasingly complex tasks and analyzing how the performance of foundation models is impacted. Foundation models are becoming vital for AI development in medical imaging, particularly in addressing data scarcity challenges. Adapting these models for medical applications often demands substantial computational resources, particularly due to their large size. To mitigate these computational demands, we propose an efficient method for adapting the robust representations of large foundation models trained on diverse datasets to specific medical tasks, aiming to make foundation models more accessible for medical use without compromising their effectiveness.

Using ovarian cancer as a case study, we develop and rigorously evaluate AI systems for ovarian tumor classification.Our systems demonstrate superior performance compared to both non-expert and expert doctors, with a strong emphasis on ensuring accuracy, generalizability across hospitals, and robustness across diverse patient subgroups.We implement a comprehensive evaluation strategy that tests the AI systems in varied clinical settings, ensuring that they maintain high performance.

Finally, we explore the integration of AI systems into clinical workflows, with a focus on the development of joint human-AI systems. By designing AI systems that collaborate effectively with healthcare professionals, we aim to enhance diagnostic accuracy, reduce doctors' workloads, and optimize the use of healthcare resources. Our collaborative human-AI system is designed to be generalizable across different clinical settings to improve patient care and advance the broader adoption of AI in medical practice, paving the way for more efficient and effective healthcare solutions.

Artificiell intelligens (AI) har stor potential att förbättra träffsäkerheten och effektiviteten inom medicinsk diagnostik, uppföljning och behandling. Inom äggstockscancer, där 70% av fallen upptäcks först i stadium III eller IV, kan AI-drivna verktyg möjliggöra tidigare upptäckt och förbättra patienternas utfall. Men den säkerhetskritiska karaktären av medicin—där även mindre fel kan få allvarliga konsekvenser—har lett till en försiktig användning av AI-teknologier. För att AI ska kunna integreras i klinisk praxis är god prestanda ej tillräckligt, utan den måste även uppvisa robusthet och generaliserbarhet till olika kliniska miljöer.

Denna avhandling undersöker utvecklingen och utvärderingen av generaliserbara och robusta AI-system, med fokus på medicinsk bildanalys. Vi börjar med att adressera viktiga luckor i vår förståelse av hur komplexitet påverkar generalisering, genom att utforska skalningslagar över allt mer komplexa uppgifter och analysera hur prestandan hos grundmodeller påverkas. Grundmodeller blir allt viktigare för AI-utveckling inom medicinsk bildbehandling, särskilt vad det gäller att hantera utmaningar med otillräckliga datamängder. Att anpassa dessa modeller för medicinska tillämpningar kräver ofta betydande beräkningsresurser, särskilt på grund av deras stora storlek. För att minska dessa krav föreslår vi en effektiv metod för att anpassa de robusta representationerna av stora grundmodeller som tränats på mångsidiga dataset till specifika medicinska uppgifter, med målet att göra grundmodeller mer tillgängliga för medicinsk användning utan att kompromissa med deras effektivitet.

Med äggstockscancer som ett fallstudie utvecklar och utvärderar vi AI-system för klassificering av äggstockstumörer. Våra system visar överlägsen prestanda jämfört med både icke-experter och expertläkare, och visar på god träffsäkerhet, generaliserbarhet över sjukhus och robusthet över olika patientgrupper. Vi implementerar en omfattande utvärderingsstrategi som testar AI-systemen i olika kliniska miljöer och säkerställer att de bibehåller hög prestanda.

Slutligen undersöker vi integrationen av AI-system i kliniska arbetsflöden, med fokus på utvecklingen av system där människa och AI samverkar. Genom att designa AI-system som samarbetar effektivt med vårdpersonal strävar vi efter att förbättra diagnostisk träffsäkerhet, minska läkarnas arbetsbelastning samt optimera användningen av vårdresurser. Vårt samverkanssystem mellan människa och AI är utformat för att vara generaliserbart över olika kliniska miljöer för att förbättra patientvården och främja en bredare användning av AI inom medicinsk praxis, vilket banar väg för mer effektiva och ändamålsenliga vårdlösningar.

Evidence suggests that networks trained on large datasets generalize well not solely because of the numerous training examples, but also class diversity which encourages learning of enriched features. This raises the question of whether this remains true when data is scarce – is there an advantage to learning with additional labels in low-data regimes? In this work, we consider a task that requires difficult-to-obtain expert annotations: tumor segmentation in mammography images. We show that, in low-data settings, performance can be improved by complementing the expert annotations with seemingly uninformative labels from non-expert annotators, turning the task into a multi-class problem. We reveal that these gains increase when less expert data is available, and uncover several interesting properties through further studies. We demonstrate our findings on CSAW-S, a new dataset that we introduce here, and confirm them on two public datasets.

In this preregistration submission, we propose an empirical study of how networks handle changes in complexity of the data. We investigate the effect of network capacity on generalization performance in the face of increasing data complexity. For this, we measure the generalization error for an image classification task where the number of classes steadily increases. We compare a number of modern architectures at different scales in this setting. The methodology, setup, and hypotheses described in this proposal were evaluated by peer review before experiments were conducted.

Ovarian lesions are common and often incidentally detected. A critical shortage of expert ultrasound examiners has raised concerns of unnecessary interventions and delayed cancer diagnoses. Deep learning has shown promising results in the detection of ovarian cancer in ultrasound images; however, external validation is lacking. In this international multicenter retrospective study, we developed and validated transformer-based neural network models using a comprehensive dataset of 17,119 ultrasound images from 3,652 patients across 20 centers in eight countries. Using a leave-one-center-out cross-validation scheme, for each center in turn, we trained a model using data from the remaining centers. The models demonstrated robust performance across centers, ultrasound systems, histological diagnoses and patient age groups, significantly outperforming both expert and non-expert examiners on all evaluated metrics, namely F1 score, sensitivity, specificity, accuracy, Cohen’s kappa, Matthew’s correlation coefficient, diagnostic odds ratio and Youden’s J statistic. Furthermore, in a retrospective triage simulation, artificial intelligence (AI)-driven diagnostic support reduced referrals to experts by 63% while significantly surpassing the diagnostic performance of the current practice. These results show that transformer-based models exhibit strong generalization and above human expert-level diagnostic accuracy, with the potential to alleviate the shortage of expert ultrasound examiners and improve patient outcomes.