Cross-modal hashing plays a pivotal role in large-scale remote sensing (RS) ship image retrieval. RS ship images often exhibit similar overall appearance with subtle differences. Existing hashing methods typically employ feature non-interaction strategies to generate common hash codes, which may not effectively capture the correlations between cross-modal ship images to reduce intermodality discrepancies. To address this issue, we propose a novel cross-modal hashing approach based on feature semi-interaction and semantic ranking (FSISR) for RS ship image retrieval. Our FSISR approach not only captures intricate correlations between different ship image modalities, but also enables the construction of hash tables for large-scale retrieval. FSISR comprises a feature semi-interaction module and a semantic ranking objective function. The semi-interaction module utilizes clustering centers from one modality to learn the correlations between two modalities and generate robust shared representations. The objective function optimizes these representations in a common Hamming space, consisting of a shared semantic alignment loss and a margin-free ranking loss. The alignment loss employs a shared semantic layer to preserve label-level similarity, while the ranking loss incorporates hard examples to establish a margin-free loss that captures similarity ranking relationships. We evaluate the performance of our method on benchmark datasets and demonstrate its effectiveness for cross-modal RS ship image retrieval. https://github.com/sunyuxi/FSISR.

Driven by the rapid growth in population, urbanization is progressing at an unprecedented rate in many places around the world. Earth observation (EO) has become a vital tool for monitoring urbanization on a global scale. Modern satellite missions, in particular, provide new opportunities for urban mapping and change detection (CD) through high-resolution imagery and frequent revisits. These missions have enabled multi-modal approaches by integrating data from different satellites, such as Sentinel-1 Synthetic Aperture Radar (SAR) and Sentinel-2 MultiSpectral Instrument (MSI). Concurrently, EO data analysis has evolved from traditional machine learning methods to deep learning (DL) models, particularly Convolutional Neural Networks (ConvNets). However, current DL methods for urban mapping and CD face several challenges, such as reliance on large labeled datasets for supervised training, the limited transferability of DL models across geographic regions, the effective integration of multi-modal EO data, and using satellite image time series (SITS) for CD. To address these challenges, this thesis aims to develop novel DL methods for robust urban mapping and CD using multi-source EO data.

First, a semi-supervised learning (SSL) method is introduced, leveraging multi-modal Sentinel-1 SAR and Sentinel-2 MSI data to improve the geographic transferability of urban mapping models. This method employs a dual stream ConvNet architecture to map built-up areas separately from SAR and optical images. By assuming consistent maps should be produced for both modalities, an unsupervised loss for unlabeled data is introduced to penalize discrepancies between them. Extensive evaluation using annotations from the SpaceNet 7 multi-temporal building monitoring dataset demonstrated that this SSL approach (F1 score 0.694) outperforms several supervised approaches (F1 scores ranging from 0.574 to 0.651). Furthermore, it produces built-up area maps that rival or surpass global human settlement maps like GHS-BUILT-S2 and WSF 2019.

For urban CD, a new network architecture is proposed for fusing bi-temporal Sentinel-1 SAR and Sentinel-2 MSI image pairs. This architecture uses a dual stream design to process each modality through separate ConvNets before combining the extracted features at a later stage. The proposed strategy outperforms other ConvNet-based approaches, both with uni-modal and multi-modal data. Additionally, it achieves state-of-the-art (SOTA) performance on the Onera Satellite CD dataset (F1 score 0.600).

Building on this, a second network architecture was developed to adapt the transferability improvement approach for urban CD. This approach uses bi-temporal Sentinel-1 SAR and Sentinel-2 MSI image pairs and outputs urban changes using a difference decoder while mapping built-up areas with a semantic decoder. Similar to the urban mapping method, inconsistencies in built-up area maps across modalities are penalized on unlabeled data. Evaluation on the SpaceNet 7 dataset, enhanced with Sentinel-1 SAR and Sentinel-2 MSI data, shows that the method performs well under limited label conditions, achieving an F1 score of 0.555 with all available labels, and delivering reasonable CD results (F1 score of 0.491) even with only 10 \% of the labeled data. In contrast, supervised multi-modal methods and SSL methods using optical data failed to exceed an F1 score of 0.402 under this condition.

A third urban CD method focuses on detecting changes in consecutive images of SITS (i.e., continuous urban CD). This method introduces a temporal feature refinement module that uses self-attention to enhance ConvNet-based multi-temporal representations of buildings. Additionally, a multi-task integration module employing Markov networks is proposed to generate optimal building map time series based on segmentation and dense change outputs. The proposed method effectively identifies urban changes in high-resolution SITS from PlanetScope (F1 score 0.551) and Gaofen-2 (F1 score 0.440), demonstrating superior performance compared to bi-temporal and multi-temporal urban CD and segmentation methods on two challenging datasets.

Finally, the thesis develops a baseline network for multi-hazard building damage detection using the xBD dataset, which contains bi-temporal images captured before and after natural disasters. The study examines model transferability across disaster types by employing a comprehensive dataset split and proposes incorporating disaster-specific information into the baseline model to account for disaster-specific damage characteristics. The disaster-adaptive model demonstrates improved generalization to unseen events compared to several competing methods.

This thesis addresses key challenges in urban mapping and urban CD, including multi-hazard building damage detection. By advancing methods that leverage multi-sensor EO data and DL techniques, this thesis makes major contributions to timely and reliable urban data production, thereby supporting sustainable urban planning and urban Sustainable Development Goal (SDG) indicators monitoring.

Urbaniseringen drivs på av den snabba befolkningstillväxten och går framåt i en aldrig tidigare skådad takt på många platser runt om i världen. Jordobservation (EO) har blivit ett viktigt verktyg för att övervaka urbaniseringen på global nivå. I synnerhet moderna satellituppdrag ger nya möjligheter till stadskartläggning och upptäckt av förändringar (CD) genom högupplösta bilder och frekventa återbesök. Dessa uppdrag har möjliggjort multimodala tillvägagångssätt genom att integrera data från olika satelliter, t.ex. Sentinel-1 Synthetic Aperture Radar (SAR) och Sentinel-2 MultiSpectral Instrument (MSI). Samtidigt har analysen av EO-data utvecklats från traditionella maskininlärningsmetoder till modeller för djupinlärning (DL), i synnerhet Convolutional Neural Networks (ConvNets). Nuvarande DL-metoder för stadskartläggning och CD står dock inför flera utmaningar, till exempel beroende av stora märkta dataset för övervakad träning, den begränsade överförbarheten av DL-modeller över geografiska regioner, effektiv integration av multimodala EO-data och användning av satellitbildstidsserier (SITS) för CD. För att ta itu med dessa utmaningar syftar denna avhandling till att utveckla nya djupinlärningsmetoder för robust stadskartläggning och förändringsdetektering med hjälp av EO-data från flera källor.

Först introduceras en SSL-metod (semi-supervised learning) som utnyttjar multimodala Sentinel-1 SAR- och Sentinel-2 MSI-data för att förbättra den geografiska överförbarheten av stadskartläggningsmodeller. Metoden använder en ConvNet-arkitektur med dubbla flöden för att kartlägga bebyggda områden separat från SAR- och optiska bilder. Genom att anta att konsekventa kartor ska produceras för båda modaliteterna införs en oövervakad förlust för omärkta data för att straffa avvikelser mellan dem. En omfattande utvärdering med hjälp av annoteringar från SpaceNet 7 multi-temporala dataset för byggnadsövervakning visade att denna SSL-metod (F1-poäng 0,694) överträffar flera övervakade metoder (F1-poäng från 0,574 till 0,651). Dessutom producerar den kartor över uppbyggda områden som konkurrerar med eller överträffar globala kartor över mänskliga bosättningar som GHS-BUILT-S2 och WSF 2019.

För CD i städer föreslås en ny nätverksarkitektur för sammanslagning av bi-temporala Sentinel-1 SAR- och Sentinel-2 MSI-bildpar. Denna arkitektur använder en dubbel strömdesign för att bearbeta varje modalitet genom separata ConvNets innan de extraherade funktionerna kombineras i ett senare skede. Den föreslagna strategin överträffar andra ConvNet-baserade metoder, både med uni-modal och multimodal data. Dessutom uppnår den toppmodern (SOTA) prestanda på Onera Satellite CD-dataset (F1-poäng 0,600).

På grundval av detta utvecklades en andra nätverksarkitektur för att anpassa metoden för förbättring av överförbarheten för CD i städer. Denna metod använder bi-temporala Sentinel-1 SAR- och Sentinel-2 MSI-bildpar och matar ut stadsförändringar med hjälp av en differensavkodare samtidigt som bebyggda områden kartläggs med en semantisk avkodare. I likhet med metoden för stadskartläggning straffas inkonsekvenser i kartor över bebyggda områden över modaliteter på omärkta data. Utvärdering på SpaceNet 7-datasetet, förbättrat med Sentinel-1 SAR och Sentinel-2 MSI-data, visar att metoden fungerar bra under begränsade etikettförhållanden, uppnår en F1-poäng på 0,555 med alla tillgängliga etiketter och levererar rimliga CD-resultat (F1-poäng på 0,491) även med endast 10 \% av de märkta data. Däremot lyckades inte övervakade multimodala metoder och SSL-metoder som använder optiska data överstiga en F1-poäng på 0,402 under detta villkor.

En tredje urban CD-metod fokuserar på att upptäcka förändringar i på varandra följande bilder av SITS (dvs. kontinuerlig urban CD). Denna metod introducerar en temporal funktionsförfiningsmodul som använder självupp-märksamhet för att förbättra ConvNet-baserade multitemporala representationer av byggnader. Dessutom föreslås en integrationsmodul med flera uppgifter som använder Markov-nätverk för att generera optimala tidsserier för byggnadskartor baserat på segmentering och täta förändringsutgångar. Den föreslagna metoden identifierar effektivt stadsförändringar i högupplösta SITS från PlanetScope (F1-poäng 0,551) och Gaofen-2 (F1-poäng 0,440), vilket visar överlägsen prestanda jämfört med bi-temporala och multi-temporala urbana CD- och segmenteringsmetoder på två utmanande dataset.

Slutligen utvecklar avhandlingen ett baslinjenätverk för detektering av byggnadsskador med flera faror med hjälp av xBD-datasetet, som innehåller bi-temporala bilder tagna före och efter naturkatastrofer. Studien undersöker modellens överförbarhet mellan olika katastroftyper genom att använda en omfattande datasetdelning och föreslår att katastrofspecifik information in-förlivas i baslinjemodellen för att ta hänsyn till katastrofspecifika skadeegenskaper. Den katastrofadaptiva modellen visar förbättrad generalisering till osedda händelser jämfört med flera konkurrerande metoder.

Denna avhandling behandlar viktiga utmaningar inom stadskartläggning och urban CD, inklusive detektering av byggnadsskador med flera faror. Genom att utveckla metoder som utnyttjar EO-data från flera sensorer och DL-tekniker ger den här avhandlingen viktiga bidrag till snabb och tillförlitlig produktion av stadsdata, vilket stöder hållbar stadsplanering och indikatorer för hållbara utvecklingsmål (SDG) i städer.

Urban areas are currently facing significant new and/or aggravated existing challenges due to the impacts of climate change, including increased frequency and intensity of extreme weather events, urban greenness loss, urban flash floods, air quality degradation and increased greenhouse gas emissions, geo-hazards, and urban heat fluxes among others. To enhance urban resilience and efficiently mitigate those impacts, sophisticated digital tools and Decision Support Systems (DSS) could play a determinant role in assisting decision-makers by means of providing access to pertinent data, analytical models as well as thorough insights for prioritizing the most effective mitigation strategies. The Horizon 2020 research and innovation project entitled "Development of a Support System for Improved Resilience and Sustainable Urban areas to cope with Climate Change and Extreme Events based on GEOSS and Advanced Modelling Tools - HARMONIA GA 101003517"introduces a series of innovative digital tools along with novel data services and products. This paper outlines the functioning of an urban-planning DSS that exploits the Harmonia multiparametric risk assessment methodology for a spectrum of different urban perils to eventually offer comprehensive and tangible urban recommendations for mitigating future hazard-driven adverse impacts. The proposed solution will be offered as a webbased application with a user-friendly interface, able to efficiently handle and visualize multidimensional (4D) geospatial information. The overall methodology and the capabilities of the DSS will be demonstrated in four different and diverse European urban environments, i.e., the cities of Milan, Piraeus, Sofia, and Ixelles.

Accurate urban maps provide essential information to support sustainable urban development. Recent urban mapping methods use multi-modal deep neural networks to fuse Synthetic Aperture Radar (SAR) and optical data. However, multi-modal networks may rely on just one modality due to the greedy nature of learning. In turn, the imbalanced utilization of modalities can negatively affect the generalization ability of a network. In this paper, we investigate the utilization of SAR and optical data for urban mapping. To that end, a dual-branch network architecture using intermediate fusion modules to share information between the uni-modal branches is utilized. A cutoff mechanism in the fusion modules enables the stopping of information flow between the branches, which is used to estimate the network's dependence on SAR and optical data. While our experiments on the SEN12 Global Urban Mapping dataset show that good performance can be achieved with conventional SAR-optical data fusion (F1 score = 0.682±0.014), we also observed a clear under-utilization of optical data. Therefore, future work is required to investigate whether a more balanced utilization of SAR and optical data can lead to performance improvements.

To better understand current trends of urban population growth in Sub-Saharan Africa, high-quality spatiotemporal population estimates are necessary. While the joint use of remote sensing and deep learning has achieved promising results for population distribution estimation, most of the current work focuses on fine-scale spatial predictions derived from single date census, thereby neglecting temporal analyses. In this work, we focus on evaluating how deep learning change detection techniques can unravel temporal population dynamics at short intervals. Since Post-Classification Comparison (PCC) methods for change detection are known to propagate the error of the individual maps, we propose an end-to-end population growth mapping method. Specifically, a ResNet encoder, pretrained on a population mapping task with Sentinel-2 MSI data, was incorporated into a Siamese network. The Siamese network was trained at the census level to accurately predict population change. The effectiveness of the proposed method is demonstrated in Kigali, Rwanda, for the time period 2016-2020, using bi-temporal Sentinel-2 data. Compared to PCC, the Siamese network greatly reduced errors in population change predictions at the census level. These results show promise for future remote sensing-based population growth mapping endeavors. Code is available on GitHub.

This paper proposes a novel multi-temporal urban mapping approach using multi-modal satellite data from the Sentinel-1 Synthetic Aperture Radar (SAR) and Sentinel-2 MultiSpectral Instrument (MSI) missions. In particular, it focuses on the problem of a partly missing optical modality due to clouds. The proposed model utilizes two networks to extract features from each modality separately. In addition, a reconstruction network is utilized to approximate the optical features based on the SAR data in case of a missing optical modality. Our experiments on a multi-temporal urban mapping dataset with Sentinel-1 SAR and Sentinel-2 MSI data demonstrate that the proposed method outperforms a multi-modal approach that uses zero values as a replacement for missing optical data, as well as a uni-modal SAR-based approach. Therefore, the proposed method is effective in exploiting multi-modal data, if available, but it also retains its effectiveness in case the optical modality is missing.

Urbanization is progressing at an unprecedented rate in many places around the world. The Sentinel-1 synthetic aperture radar (SAR) and Sentinel-2 MultiSpectral Instrument (MSI) missions, combined with deep learning, offer new opportunities to accurately monitor urbanization at a global scale. Although the joint use of SAR and optical data has recently been investigated for urban change detection, existing data fusion methods rely heavily on the availability of sufficient training labels. Meanwhile, change detection methods addressing label scarcity are typically designed for single-sensor optical data. To overcome these limitations, we propose a semi-supervised urban change detection method that exploits unlabeled Sentinel-1 SAR and Sentinel-2 MSI data. Using bitemporal SAR and optical image pairs as inputs, the proposed multi-modal Siamese network predicts urban changes and performs built-up area segmentation for both timestamps. Additionally, we introduce a consistency loss, which penalizes inconsistent built-up area segmentation across sensor modalities on unlabeled data, leading to more robust features. To demonstrate the effectiveness of the proposed method, the SpaceNet 7 dataset, comprising multi-temporal building annotations from rapidly urbanizing areas across the globe, was enriched with Sentinel-1 SAR and Sentinel-2 MSI data. Subsequently, network performance was analyzed under label-scarce conditions by training the network on different fractions of the labeled training set. The proposed method achieved an F1 score of 0.555 when using all available training labels, and produced reasonable change detection results (F1 score of 0.491) even with as little as 10% of the labeled training data. In contrast, multi-modal supervised methods and semi-supervised methods using optical data failed to exceed an F1 score of 0.402 under this condition. Code and data are made publicly available.

Urban population distribution maps are vital elements for monitoring the Sustainable Development Goals, appropriately allocating resources such as vaccination campaigns, and facilitating evidence-based decision making. Typically, population distribution maps are derived from census data from the region of interest. Nevertheless, in several low-and middle-income countries, census information may be unreliable, outdated or unsuitable for spatial analysis at the intra-urban level, which poses severe limitations in the development of urban population maps of adequate quality. To address these shortcomings, we deploy a novel framework utilizing multisource Earth Observation (EO) information such as Sentinel-2 and very-high-resolution Pleiades imagery, openly available building footprint datasets, and deep learning (DL) architectures, providing end -to-end solutions to the production of high quality intra-urban population distribution maps in data scarce contexts. Using several case studies in Sub-Saharan Africa, namely Dakar (Senegal), Nairobi (Kenya) and Dar es Salaam (Tanzania), our results emphasize that the combination of DL and EO data is very potent and can successfully capture relationships between the retrieved image features and population counts at fine spatial resolutions (100 meter). Moreover, for the first time, we used state-of-the-art domain adaptation methods to predict population distributions in Dar es Salaam and Nairobi (R2 = 0.39, 0.60) that did not require national census or survey data from Kenya or Tanzania, but only a sample of training locations from Dakar. The DL architecture is based on a modified ResNet-18 model with dual-streams to analyze multi-modal data. Our findings have strong implications for the development of a new generation of urban population products that are an output of end-to-end solutions, can be updated frequently and rely completely on open data.

Rapid urbanization in developing countries often results in uncontrolled urban growth. In order to support sustainable urban development, reliable and up-to-date information on urban land cover changes and their environmental impact is needed. In this study, we aim at evaluating the potential of Sentinel-2 (S-2) Multi-spectral Instrument (MSI) data for urban land cover change monitoring and for analyzing resulting impacts on Ecosystem Services (ES) provision in Kigali, Rwanda. Land cover classification into eight distinct urban classes (84% overall accuracies, 0.8 Kappa) was performed on data from 2016 and 2021 using a hybrid approach combining Random Forest with a U-Net-based impervious surface segmentation that improved the delineation of urban areas. The bi-temporal land cover maps were then analyzed regarding landscape structure using Landscape Metrics (LM). Ecosystem service bundles were derived for both years and their changes were summarized. Service providing areas were further evaluated in terms of changes in spatial attributes and structure of patches. ES were aggregated into eight bundles and grouped into provisioning, regulating and supporting services. The bundles were further analyzed using a matrix spatially linking landscape units with service supply and demand budgets. The results illustrated that three urban development scenarios can be distinguished including infill through housing and infrastructures development in core urban areas, urban sprawl in fringe zones and the development of urban patches at distant locations intercepted by cropland. The results revealed that the changes in LM negatively affected ES supply mainly through a decrease in cropland and forest. The expansion of built-up areas resulted in a high demand for provisioning and regulating services, especially food and water provision, surface runoff mitigation and erosion control. This is the first study demonstrating that detailed monitoring of urbanization and resulting environmental impacts can be performed with open access S-2 MSI data in Sub-Saharan Africa. Moreover, the framework developed in this study has the potential to be transferred to other Sub-Saharan cities.

Urbanization is progressing rapidly around the world. With sub-weekly revisits at global scale, Sentinel-1 synthetic aperture radar (SAR) and Sentinel-2 multispectral imager (MSI) data can play an important role for monitoring urban sprawl to support sustainable development. In this letter, we proposed an urban change detection (CD) approach featuring a new network architecture for the fusion of SAR and optical data. Specifically, a dual stream concept was introduced to process different data modalities separately, before combining extracted features at a later decision stage. The individual streams are based on U-Net architecture that is one of the most popular fully convolutional networks used for semantic segmentation. The effectiveness of the proposed approach was demonstrated using the Onera Satellite CD (OSCD) dataset. The proposed strategy outperformed other U-Net-based approaches in combination with unimodal data and multimodal data with feature level fusion. Furthermore, our approach achieved state-of-the-art performance on the urban CD problem posed by the OSCD dataset. Our Sentinel-1 SAR data and code are available on https://github.com/SebastianHafner/DS_UNet.