Energy-resolving photon-counting CT promises improved material-separation capabilities compared to conventional CT. However, accurate separation of iodine and calcium, which is especially important for imaging atherosclerotic plaques, remains a challenge. In this proof-of-concept study, we present a deep-learning based method that takes a pair of basis images from a photon-counting CT as input and produces a map of the iodine distribution where the contamination from other materials such as calcium is minimized, and demonstrate its performance on clinical images of the carotid arteries. As training data, we used 13 pairs of image slices of the neck from a silicon-based photon-counting spectral CT, with one non-contrast and one contrast-enhanced slice in each pair. To generate a ground-truth iodine maps as training labels, 40 keV virtual monoenergetic non-contrast images were registered to align with the corresponding 40 keV contrast-enhanced images, and the difference between those two image slices was used to generate an iodine concentration map. We trained a ResUNet++ deep convolutional neural network using water-iodine basis image pairs resulting from a two-basis material decomposition as inputs and the iodine concentration map obtained from subtraction as label. The resulting method was evaluated on a previously unseen photon-counting CT image slice of the neck from the same patient. Our results show that the trained network correctly highlights the image features containing iodinated contrast agent and quantifies the concentration accurately. The contamination from calcium and other tissues is significantly reduced compared to the original iodine basis image. Our results demonstrate that the proposed method can successfully separate iodine from calcium and other tissues on clinical silicon-based photon-counting CT, with important potential implications for imaging of atherosclerosis.

Earth observation satellites play a crucial role in disaster response and management, offering timely and large-scale data for damage assessment. Recent studies have demonstrated the potential of deep learning techniques for automated building damage detection from satellite imagery, often based on the xBD dataset. This high-quality dataset features bi-temporal very-high-resolution image pairs of several disaster events. Notably, several studies have proposed new network architectures and demonstrated their improved performance on xBD. Although such highly engineered model-centric approaches achieve promising results on the original dataset split of xBD, we show that they underperform on a new event-based split, which evaluates them on unseen events. To reduce this generalization gap, we propose to follow a data-centric approach. For this, we first derive a simplified baseline method from the winning solution of the xView2 competition, with greatly reduced complexity. With a simple adjustment to this baseline method, we incorporate readily available disaster-type information, allowing it to account for disaster-specific damage characteristics. We evaluate the resulting disaster-adaptive model on the event-based split of xBD and demonstrate its improved ability to generalize to unseen events compared to several competing methods. These results highlight the potential of our data-centric approach for practical and robust building damage assessment in real-world disaster scenarios. Code including the strong baseline model is available at: https://github.com/SebastianHafner/DisasterAdaptiveNet.

Human 3D pose and motion capture have numerous applications in fields such as augmented and virtual reality, animation, robotics and sports. However, even the best capturing methods suffer from artifacts such as missed joints and noisy or inaccurate joint positions. To address this we propose the Cross-attention Masked Auto-Encoder (XMAE) for human 3D motion infilling and denoising. XMAE extends the original Masked Auto-Encoder design by introducing cross-attention in the decoder to deal with the train-test gap common in methods utilizing masking and mask tokens. Furthermore, we introduce joint displacement as an additional noise source during training, enabling XMAE to learn to correct incorrect joint positions. Through extensive experiments, we show XMAE's effectiveness compared to state-of-the-art approaches across three public datasets and its ability to denoise real-world data, reducing limb length standard deviation by 28\% when applied on our in-the-wild professional soccer dataset.

The Visible Infrared Imaging Radiometer Suite (VIIRS) onboard the Suomi National Polar-orbiting Partnership (Suomi-NPP) satellite has been used for the early detection and daily monitoring of active wildfires. How to effectively segment the active fire (AF) pixels from VIIRS image time-series in a reliable manner remains a challenge because of the low precision associated with high recall using automatic methods. For AF detection, multicriteria thresholding is often applied to both low-resolution and mid-resolution Earth observation images. Deep learning approaches based on convolutional neural networks (ConvNets) are also well-studied on mid-resolution images. However, ConvNet-based approaches have poor performance on low-resolution images because of the coarse spatial features. On the other hand, the high temporal resolution of VIIRS images highlights the potential of using sequential models for AF detection. Transformer networks, a recent deep learning architecture based on self-attention, offer hope as they have shown strong performance on image segmentation and sequential modeling tasks within computer vision. In this research, we propose a transformer-based solution to segment AF pixels from the VIIRS time-series. The solution feeds a time-series of tokenized pixels into a transformer network to identify AF pixels at each timestamp and achieves a significantly higher F1-score than prior approaches for AFs within the study areas in California, New Mexico, and Oregon in the U.S., and in British Columbia and Alberta in Canada, as well as in Australia, and Sweden.

We present a multi-temporal, multi-modal remote-sensing dataset for predicting how active wildfires will spread at a resolution of 24 hours.The dataset consists of 13 607 images across 607 fire events in the United States from January 2018 to October 2021.For each fire event, the dataset contains a full time series of daily observations, containing detected active fires and variables related to fuel, topography and weather conditions.The dataset is challenging due to: a) its inputs being multi-temporal, b) the high number of 23 multi-modal input channels, c) highly imbalanced labels and d) noisy labels, due to smoke, clouds, and inaccuracies in the active fire detection.The underlying complexity of the physical processes adds to these challenges.Compared to existing public datasets in this area, WILDFIRESPREADTS allows for multi-temporal modeling of spreading wildfires, due to its time series structure.Furthermore, we provide additional input modalities and a high spatial resolution of 375m for the active fire maps.We publish this dataset to encourage further research on this important task with multi-temporal, noise-resistant or generative methods, uncertainty estimation or advanced optimization techniques that deal with the high-dimensional input space.

The number of linear regions has been studied as a proxy of complexity for ReLU networks. However, the empirical success of network compression techniques like pruning and knowledge distillation, suggest that in the overparameterized setting, linear regions density might fail to capture the effective nonlinearity. In this work, we propose an efficient algorithm for discovering linear regions and use it to investigate the effectiveness of density in capturing the nonlinearity of trained VGGs and ResNets on CIFAR-10 and CIFAR-100. We contrast the results with a more principled nonlinearity measure based on function variation, highlighting the shortcomings of linear regions density. Furthermore, interestingly, our measure of nonlinearity clearly correlates with model-wise deep double descent, connecting reduced test error with reduced nonlinearity, and increased local similarity of linear regions.

We study the impact of using rich and diverse textual descriptions of classes for zero-shot learning (ZSL) on ImageNet. We create a new dataset ImageNet-Wiki that matches each ImageNet class to its corresponding Wikipedia article. We show that merely employing these Wikipedia articles as class descriptions yields much higher ZSL performance than prior works. Even a simple model using this type of auxiliary data outperforms state-of-the-art models that rely on standard features of word embedding encodings of class names. These results highlight the usefulness and importance of textual descriptions for ZSL, as well as the relative importance of auxiliary data type compared to the algorithmic progress. Our experimental results also show that standard zero-shot learning approaches generalize poorly across categories of classes.

In this paper we describe a probabilistic method for estimating the position of an object along with its covariance matrix using neural networks. Our method is designed to be robust to outliers, have bounded gradients with respect to the network outputs, among other desirable properties. To achieve this we introduce a novel probability distribution inspired by the Huber loss. We also introduce a new way to parameterize positive definite matrices to ensure invariance to the choice of orientation for the coordinate system we regress over. We evaluate our method on popular body pose and facial landmark datasets and get performance on par or exceeding the performance of non-heatmap methods. Our code is available at github.com/Davmo049/Public_prob_regression_with_huber_distributions.

Visual relationship detection is fundamental for holistic image understanding. However, the localization and classification of (subject, predicate, object) triplets remain challenging tasks, due to the combinatorial explosion of possible relationships, their long-tailed distribution in natural images, and an expensive annotation process. This paper introduces a novel weakly-supervised method for visual relationship detection that relies on minimal image-level predicate labels. A graph neural network is trained to classify predicates in images from a graph representation of detected objects, implicitly encoding an inductive bias for pairwise relations. We then frame relationship detection as the explanation of such a predicate classifier, i.e. we obtain a complete relation by recovering the subject and object of a predicted predicate. We present results comparable to recent fully- and weakly-supervised methods on three diverse and challenging datasets: HICO-DET for human-object interaction, Visual Relationship Detection for generic object-to-object relations, and UnRel for unusual triplets; demonstrating robustness to non-comprehensive annotations and good few-shot generalization.