kth.sePublications KTH
Change search
Link to record
Permanent link

Direct link
Publications (10 of 19) Show all publications
Zhou, Y., Wang, Y., Su, J., Wen, Z., Zhang, P. & Zhang, W. (2025). EMSNet: Efficient Multimodal Symmetric Network for Semantic Segmentation of Urban Scene From Remote Sensing Imagery. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 18, 5878-5892
Open this publication in new window or tab >>EMSNet: Efficient Multimodal Symmetric Network for Semantic Segmentation of Urban Scene From Remote Sensing Imagery
Show others...
2025 (English)In: IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, ISSN 1939-1404, E-ISSN 2151-1535, Vol. 18, p. 5878-5892Article in journal (Refereed) Published
Abstract [en]

High-resolution remote sensing imagery (RSI) plays a pivotal role in the semantic segmentation (SS) of urban scenes, particularly in urban management tasks such as building planning and traffic flow analysis. However, the dense distribution of objects and the prevalent background noise in RSI make it challenging to achieve stable and accurate results from a single view. Integrating digital surface models (DSM) can achieve high-precision SS. But this often requires extensive computational resources. It is essential to address the tradeoff between accuracy and computational cost and optimize the method for deployment on edge devices. In this article, we introduce an efficient multimodal symmetric network (EMSNet) designed to perform SS by leveraging both optical and DSM images. Unlike other multimodal methods, EMSNet adopts a dual encoder-decoder structure to build a direct connection between DSM data and the final result, making full use of the advanced DSM. Between branches, we propose a continuous feature interaction to guide the DSM branch by RGB features. Within each branch, multilevel feature fusion captures low spatial and high semantic information, improving the model's scene perception. Meanwhile, knowledge distillation (KD) further improves the performance and generalization of EMSNet. Experiments on the Potsdam and Vaihingen datasets demonstrate the superiority of our method over other baseline models. Ablation experiments validate the effectiveness of each component. Besides, the KD strategy is confirmed by comparing it with the segment anything model (SAM). It enables the proposed multimodal SS network to match SAM's performance with only one-fifth of the parameters, computation, and latency.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2025
Keywords
Optical sensors, Optical imaging, Remote sensing, Feature extraction, Accuracy, Buildings, Biomedical optical imaging, Decoding, Computational modeling, Remote sensing image interpretation, segment anything, semantic segmentation, symmetric multimodal
National Category
Computer graphics and computer vision
Identifiers
urn:nbn:se:kth:diva-361297 (URN)10.1109/JSTARS.2025.3531422 (DOI)001432389100006 ()2-s2.0-85216074232 (Scopus ID)
Note

QC 20250317

Available from: 2025-03-17 Created: 2025-03-17 Last updated: 2025-03-17Bibliographically approved
Zhang, P., Hu, X., Ban, Y., Nascetti, A. & Gong, M. (2024). Assessing Sentinel-2, Sentinel-1, and ALOS-2 PALSAR-2 Data for Large-Scale Wildfire-Burned Area Mapping: Insights from the 2017–2019 Canada Wildfires. Remote Sensing, 16(3), Article ID 556.
Open this publication in new window or tab >>Assessing Sentinel-2, Sentinel-1, and ALOS-2 PALSAR-2 Data for Large-Scale Wildfire-Burned Area Mapping: Insights from the 2017–2019 Canada Wildfires
Show others...
2024 (English)In: Remote Sensing, E-ISSN 2072-4292, Vol. 16, no 3, article id 556Article in journal (Refereed) Published
Abstract [en]

Wildfires play a crucial role in the transformation of forest ecosystems and exert a significant influence on the global climate over geological timescales. Recent shifts in climate patterns and intensified human–forest interactions have led to an increase in the incidence of wildfires. These fires are characterized by their extensive coverage, higher frequency, and prolonged duration, rendering them increasingly destructive. To mitigate the impact of wildfires on climate change, ecosystems, and biodiversity, it is imperative to conduct systematic monitoring of wildfire progression and evaluate their environmental repercussions on a global scale. Satellite remote sensing is a powerful tool, offering precise and timely data on terrestrial changes, and has been extensively utilized for wildfire identification, tracking, and impact assessment at both local and regional levels. The Canada Centre for Mapping and Earth Observation, in collaboration with the Canadian Forest Service, has developed a comprehensive National Burned Area Composite (NBAC). This composite serves as a benchmark for curating a bi-temporal multi-source satellite image dataset for change detection, compiled from the archives of Sentinel-2, Sentinel-1, and ALOS-2 PALSAR-2. To our knowledge, this dataset is the inaugural large-scale, multi-source, and multi-frequency satellite image dataset with 20 m spatial resolution for wildfire mapping, monitoring, and evaluation. It harbors significant potential for enhancing wildfire management strategies, building upon the profound advancements in deep learning that have contributed to the field of remote sensing. Based on our curated dataset, which encompasses major wildfire events in Canada, we conducted a systematic evaluation of the capability of multi-source satellite earth observation data in identifying wildfire-burned areas using statistical analysis and deep learning. Our analysis compares the difference between burned and unburned areas using post-event observation solely or bi-temporal (pre- and post-event) observations across diverse land cover types. We demonstrate that optical satellite data yield higher separability than C-Band and L-Band Synthetic Aperture Radar (SAR), which exhibit considerable overlap in burned and unburned sample distribution, as evidenced by SAR-based boxplots. With U-Net, we further explore how different input channels influence the detection accuracy. Our findings reveal that deep neural networks enhance SAR’s performance in mapping burned areas. Notably, C-Band SAR shows a higher dependency on pre-event data than L-Band SAR for effective detection. A comparative analysis of U-Net and its variants indicates that U-Net works best with single-sensor data, while the late fusion architecture marginally surpasses others in the fusion of optical and SAR data. Accuracy across sensors is highest in closed forests, with sequentially lower performance in open forests, shrubs, and grasslands. Future work will extend the data from both spatial and temporal dimensions to encompass varied vegetation types and climate zones, furthering our understanding of multi-source and multi-frequency satellite remote sensing capabilities in wildfire detection and monitoring.

Place, publisher, year, edition, pages
MDPI AG, 2024
Keywords
ALOS-2 PALSAR-2, burned area mapping, change detection, data fusion, dataset, deep learning, multi-frequency, multi-source, SAR, Sentinel-1, Sentinel-2, siamese networks, wildfire
National Category
Earth Observation
Identifiers
urn:nbn:se:kth:diva-343666 (URN)10.3390/rs16030556 (DOI)001160514200001 ()2-s2.0-85184671536 (Scopus ID)
Note

QC 20240222

Available from: 2024-02-22 Created: 2024-02-22 Last updated: 2025-02-10Bibliographically approved
Huang, J. & Zhang, P. (2023). Beyond Pixel-Wise Unmixing: Spatial-Spectral Attention Fully Convolutional Networks for Abundance Estimation. Remote Sensing, 15(24), Article ID 5694.
Open this publication in new window or tab >>Beyond Pixel-Wise Unmixing: Spatial-Spectral Attention Fully Convolutional Networks for Abundance Estimation
2023 (English)In: Remote Sensing, E-ISSN 2072-4292, Vol. 15, no 24, article id 5694Article in journal (Refereed) Published
Abstract [en]

Spectral unmixing poses a significant challenge within hyperspectral image processing, traditionally addressed by supervised convolutional neural network (CNN)-based approaches employing patch-to-pixel (pixel-wise) methods. However, such pixel-wise methodologies often necessitate image splitting into overlapping patches, resulting in redundant computations and potential information leakage between training and test samples, consequently yielding overoptimistic outcomes. To overcome these challenges, this paper introduces a novel patch-to-patch (patch-wise) framework with nonoverlapping splitting, mitigating the need for repetitive calculations and preventing information leakage. The proposed framework incorporates a novel neural network structure inspired by the fully convolutional network (FCN), tailored for patch-wise unmixing. A highly efficient band reduction layer is incorporated to reduce the spectral dimension, and a specialized abundance constraint module is crafted to enforce both the Abundance Nonnegativity Constraint and the Abundance Sum-to-One Constraint for unmixing tasks. Furthermore, to enhance the performance of abundance estimation, a spatial-spectral attention module is introduced to activate the most informative spatial areas and feature maps. Extensive quantitative experiments and visual assessments conducted on two synthetic datasets and three real datasets substantiate the superior performance of the proposed algorithm. Significantly, the method achieves an impressive RMSE loss of 0.007, which is at least 4.5 times lower than that of other baselines on Urban hyperspectral images. This outcome demonstrates the effectiveness of our approach in addressing the challenges of spectral unmixing.

Place, publisher, year, edition, pages
MDPI AG, 2023
Keywords
hyperspectral unmixing, abundance estimation, patch-wise unmixing, fully convolutional networks, spatial-spectral attention
National Category
Computer graphics and computer vision
Identifiers
urn:nbn:se:kth:diva-342305 (URN)10.3390/rs15245694 (DOI)001130637400001 ()2-s2.0-85180616050 (Scopus ID)
Note

QC 20240124

Available from: 2024-01-24 Created: 2024-01-24 Last updated: 2025-02-07Bibliographically approved
Hu, X., Zhang, P., Ban, Y. & Rahnemoonfar, M. (2023). GAN-based SAR and optical image translation for wildfire impact assessment using multi-source remote sensing data. Remote Sensing of Environment, 289, 113522, Article ID 113522.
Open this publication in new window or tab >>GAN-based SAR and optical image translation for wildfire impact assessment using multi-source remote sensing data
2023 (English)In: Remote Sensing of Environment, ISSN 0034-4257, E-ISSN 1879-0704, Vol. 289, p. 113522-, article id 113522Article in journal (Refereed) Published
Abstract [en]

Despite the popularity and success in burned area detection and assessment, multispectral satellite images are often affected by poor sunlight-illumination conditions, particularly at high latitudes. Given that Synthetic Aperture Radar (SAR) can effectively penetrate clouds and collect images in all-weather conditions during day and night, the complementary use of optical and SAR data can be helpful for remote-sensing measurements and assessments of burned sites. Nevertheless, the widely used burn-sensitive spectral indices hardly help analyze SAR data due to the inherent difference between optical and SAR sensors in physical imaging mechanisms. In this study, we aim to leverage multi-source data for burned area mapping and burn severity assessment by translating SAR images into optical images using ResNet-based Pix2Pix model. Experiments were performed on 8669 pairs of bitemporal Sentinel-1 SAR and Sentinel-2 optical patches over 304 large wildfire events in Canada with a wide range of land covers. The translated optical images from SAR data occupied similar spectral properties to real optical observations and the corresponding generated spectral indices (i.e., delta Normalized Burn Ratio (dNBR), relative dNBR, and Relativized Burn Ratio) also showed high agreement with real optical indices. In terms of burned area detection using the generated indices, their medium values of the area under the receiver operating characteristics curve (AUC) were over 85%, achieving promising performance that outperformed the SAR-based index. Burn severity maps derived from multi-source data achieved a relatively high Kappa coefficient of 0.77. Results showed the feasibility and effectiveness of GAN-based SAR-to-optical translation for wildfire impact assessment, paving the way for the combined use of optical and SAR data.

Place, publisher, year, edition, pages
Elsevier BV, 2023
Keywords
Wildfire, Burned area, Burn severity, Deep learning, Image translation, Generative adversarial networks (GAN), Sentinel-1, Sentinel-2
National Category
Earth Observation
Identifiers
urn:nbn:se:kth:diva-325593 (URN)10.1016/j.rse.2023.113522 (DOI)000953270300001 ()2-s2.0-85149311571 (Scopus ID)
Note

QC 20230412

Available from: 2023-04-12 Created: 2023-04-12 Last updated: 2025-02-10Bibliographically approved
Hu, X., Zhang, P. & Ban, Y. (2023). Large-scale burn severity mapping in multispectral imagery using deep semantic segmentation models. ISPRS journal of photogrammetry and remote sensing (Print), 196, 228-240
Open this publication in new window or tab >>Large-scale burn severity mapping in multispectral imagery using deep semantic segmentation models
2023 (English)In: ISPRS journal of photogrammetry and remote sensing (Print), ISSN 0924-2716, E-ISSN 1872-8235, Vol. 196, p. 228-240Article in journal (Refereed) Published
Abstract [en]

Nowadays Earth observation satellites provide forest fire authorities and resource managers with spatial and comprehensive information for fire stabilization and recovery. Burn severity mapping is typically performed by classifying bi-temporal indices (e.g., dNBR, and RdNBR) using thresholds derived from parametric models incorporating field-based measurements. Analysts are currently expending considerable manual effort using prior knowledge and visual inspection to determine burn severity thresholds. In this study, we aim to employ highly automated approaches to provide spatially explicit damage level estimates. We first reorganize a large-scale Landsat-based bi-temporal burn severity assessment dataset (Landsat-BSA) by visual data cleaning based on annotated MTBS data (approximately 1000 major fire events in the United States). Then we apply state-of-the-art deep learning (DL) based methods to map burn severity based on the Landsat-BSA dataset. Experimental results emphasize that multi-class semantic segmentation algorithms can approximate the threshold-based techniques used extensively for burn severity classification. UNet-like models outperform other region-based CNN and Transformer-based models and achieve accurate pixel-wise classification results. Combined with the online hard example mining algorithm to reduce class imbalance issue, Attention UNet achieves the highest mIoU (0.78) and the highest Kappa coefficient close to 0.90. The bi-temporal inputs with ancillary spectral indices work much better than the uni-temporal multispectral inputs. The restructured dataset will be publicly available and create opportunities for further advances in remote sensing and wildfire communities.

Place, publisher, year, edition, pages
Elsevier BV, 2023
Keywords
Landsat data, Burn severity dataset, Deep learning, Semantic segmentation, Burn severity assessment
National Category
Earth Observation
Identifiers
urn:nbn:se:kth:diva-326865 (URN)10.1016/j.isprsjprs.2022.12.026 (DOI)000974595100001 ()2-s2.0-85146056049 (Scopus ID)
Note

QC 20230515

Available from: 2023-05-15 Created: 2023-05-15 Last updated: 2025-02-10Bibliographically approved
Zhang, P., Ban, Y. & Nascetti, A. (2023). Total-variation regularized U-Net for wildfire burned area mapping based on Sentinel-1 C-Band SAR backscattering data. ISPRS journal of photogrammetry and remote sensing (Print), 203, 301-313
Open this publication in new window or tab >>Total-variation regularized U-Net for wildfire burned area mapping based on Sentinel-1 C-Band SAR backscattering data
2023 (English)In: ISPRS journal of photogrammetry and remote sensing (Print), ISSN 0924-2716, E-ISSN 1872-8235, Vol. 203, p. 301-313Article in journal (Refereed) Published
Abstract [en]

Previous studies have shown that Synthetic Aperture Radar (SAR) is able to detect burned areas, serving as a key data source for monitoring active wildfires in situations where optical sensors are hindered by dense smoke or cloud cover. Radar remote sensing provides rich and useful data on historical burned areas, which are critical for large-scale wildfire burned area mapping. This study aims to unveil the potential inherent in Sentinel-1 C-Band SAR data and to investigate the impact of reference masks on wildfire burned area mapping. SAR images frequently exhibit disruptive speckle noise, which manifests as isolated pixels or small regions, thereby hindering the seamless identification of burned areas through SAR data. To mitigate noise and enhance the connectivity of SAR-based burned area delineation, we propose a novel approach: a Total-Variation (TV) regularized U-Net model, tailored to learn from noisy pseudo masks derived from Sentinel-1 C-Band SAR backscattering data. To validate its efficacy, we assembled a dataset encompassing 16 geographically diverse wildfire events, incorporating SAR-based kMap, a SAR-based binary label (designated as SARREF), and an optical-based binary label (referred to as OptREF). The kMap, serving as a relative indicator of SAR backscattering change, is defined as the divergence between post-fire SAR backscattering and the temporal average, normalized by the temporal standard deviation of pre-fire SAR time series. Utilizing SAR-based kMap as input and SARREF or OptREF as labels, we systematically explored the impact of TV regularization on SAR-based burned area mapping. Our experimental findings highlight several key insights: 1) Vanilla U-Net considerably elevates the F1 score from 0.61 to 0.66 when compared to threshold-based approaches; 2) The TV-regularized U-Net exhibits a notable capacity for enhancing both accuracy and the connectivity of SAR-based burned areas. When trained with noisy SARREF, it achieves a substantial F1 score of 0.69 and an IoU score of 0.53. 3) In scenarios involving OptREF, the vanilla U-Net attains peak performance with the highest F1 (0.75) and IoU (0.61) scores, with the expected outcome of TV regularization yielding no additional enhancement.

Place, publisher, year, edition, pages
Elsevier BV, 2023
Keywords
Burned area, Deep learning, SAR, Segmentation, Sentinel-1, Total variation, U-Net, Wildfire
National Category
Earth Observation
Identifiers
urn:nbn:se:kth:diva-334942 (URN)10.1016/j.isprsjprs.2023.07.024 (DOI)001064785200001 ()2-s2.0-85168409481 (Scopus ID)
Note

QC 20230830

Available from: 2023-08-30 Created: 2023-08-30 Last updated: 2025-02-10Bibliographically approved
Zhang, P. & Ban, Y. (2023). Unsupervised Geospatial Domain Adaptation for Large-Scale Wildfire Burned Area Mapping Using Sentinel-2 MSI and Sentinel-1 SAR Data. In: IGARSS 2023: 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings. Paper presented at 2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023, Pasadena, United States of America, Jul 16 2023 - Jul 21 2023 (pp. 5742-5745). Institute of Electrical and Electronics Engineers Inc.
Open this publication in new window or tab >>Unsupervised Geospatial Domain Adaptation for Large-Scale Wildfire Burned Area Mapping Using Sentinel-2 MSI and Sentinel-1 SAR Data
2023 (English)In: IGARSS 2023: 2023 IEEE International Geoscience and Remote Sensing Symposium, Proceedings, Institute of Electrical and Electronics Engineers Inc. , 2023, p. 5742-5745Conference paper, Published paper (Refereed)
Abstract [en]

Satellite remote sensing provides a cost-effective way for monitoring wildfires on a large scale, and the continuous observations and measurements have made remote sensing a primary source of unlabelled big data. Supervised deep learning has shown great success in various remote sensing applications, but it heavily relies on high-quality labels. However, burned area labels are only available for a small part of the world, supervised deep learning from limited labelled data has poor generalization performance across geographical regions and climate zones. Different satellite sensors represent the same physical objects in various ways, while multi-source satellite data often exhibits a combination of common and complementary information, such as optical and radar data. The common information makes it possible to exploit huge amounts of unlabelled multi-source data in model training through consistency regularization between multi-source predictions. In this work, we adopted an unsupervised geospatial domain adaptation (GDA) framework based Dual Stream U-Net model, which combines the supervised loss and unsupervised multi-modal consistency regularization to exploit both labelled and unlabelled multi-model data for model training in a semi-supervised learning manner. The experimental results demonstrate that unsupervised GDA has better generalization performance across geographical regions than fully supervised learning.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers Inc., 2023
Keywords
burned area, change detection, domain adaptation, segmentation, Sentinel-1, Sentinel-2, Wildfire
National Category
Earth Observation Computer graphics and computer vision
Identifiers
urn:nbn:se:kth:diva-340803 (URN)10.1109/IGARSS52108.2023.10281548 (DOI)001098971605221 ()2-s2.0-85178354899 (Scopus ID)
Conference
2023 IEEE International Geoscience and Remote Sensing Symposium, IGARSS 2023, Pasadena, United States of America, Jul 16 2023 - Jul 21 2023
Note

Part of ISBN 9798350320107

QC 20231214

Available from: 2023-12-14 Created: 2023-12-14 Last updated: 2025-02-10Bibliographically approved
Hu, X., Zhang, P. & Ban, Y. (2022). Gan-Based Sar To Optical Image Translation In Fire-Disturbed Regions. In: 2022 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2022): . Paper presented at IEEE International Geoscience and Remote Sensing Symposium (IGARSS), JUL 17-22, 2022, Kuala Lumpur, MALAYSIA (pp. 1456-1459). Institute of Electrical and Electronics Engineers (IEEE)
Open this publication in new window or tab >>Gan-Based Sar To Optical Image Translation In Fire-Disturbed Regions
2022 (English)In: 2022 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2022), Institute of Electrical and Electronics Engineers (IEEE) , 2022, p. 1456-1459Conference paper, Published paper (Refereed)
Abstract [en]

Climate change by anthropogenic warming leads to increases in dry fuels and promotes forest fires. Multispectral images' quality is easily affected by poor atmospheric conditions. SAR satellite sensors can penetrate through clouds and image day and night. However, the burned area mapping methods widely used for optical data are not feasible to be applied for SAR data owing to the differences in imaging mechanisms. Recent advances in deep image translation can fill this gap by using Generative Adversarial Networks (GAN). In this research, we apply a GAN-based model for SAR to optical image translation over fire-disturbed regions. Specifically, Sentinel-1 SAR images are translated into Sentinel-2 images using the ResNet-based Pix2Pix model, which is trained on 281 large fire events and tested on the other 23 events in Canada. The generated images preserve the spectral characteristics well and show high similarity to the real images with Structure Similarity Index Measure (SSIM) over 0.59.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
Series
IEEE International Symposium on Geoscience and Remote Sensing IGARSS, ISSN 2153-6996
Keywords
Sentinel-2 MSI, Sentinel-1 C band SAR, image Translation, Generative Adversarial Network (GAN)
National Category
Computer graphics and computer vision
Identifiers
urn:nbn:se:kth:diva-326624 (URN)10.1109/IGARSS46834.2022.9884234 (DOI)000920916601172 ()2-s2.0-85140393425 (Scopus ID)
Conference
IEEE International Geoscience and Remote Sensing Symposium (IGARSS), JUL 17-22, 2022, Kuala Lumpur, MALAYSIA
Note

QC 20230508

Available from: 2023-05-08 Created: 2023-05-08 Last updated: 2025-02-07Bibliographically approved
Zhang, P., Hu, X. & Ban, Y. (2022). Wildfire-S1S2-Canada: A Large-Scale Sentinel-1/2 Wildfire Burned Area Mapping Dataset Based On The 2017-2019 Wildfires In Canada. In: 2022 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2022): . Paper presented at IEEE International Geoscience and Remote Sensing Symposium (IGARSS), JUL 17-22, 2022, Kuala Lumpur, MALAYSIA (pp. 7954-7957). Institute of Electrical and Electronics Engineers (IEEE)
Open this publication in new window or tab >>Wildfire-S1S2-Canada: A Large-Scale Sentinel-1/2 Wildfire Burned Area Mapping Dataset Based On The 2017-2019 Wildfires In Canada
2022 (English)In: 2022 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM (IGARSS 2022), Institute of Electrical and Electronics Engineers (IEEE) , 2022, p. 7954-7957Conference paper, Published paper (Refereed)
Abstract [en]

Wildfires vary across space and time, precisely and timely mapping on the wildfire affected areas is critical for wildfire management, population and property protection, and environmental impact assessment. In this study, we established a large-scale annotated wildfire burned area dataset based on freely available Sentinel-1 SAR and Sentinel-2 multispectral instrument (MSI) data and Canada Wildfire Burned Area Database. This dataset includes bi-temporal Sentinel-1 and Sentinel-2 images, which allows users to exploit remotely sensed data acquired in both optical and microwave domains. On the proposed dataset, we achieved the highest IoU score of 0.86 on the Sentinel-2 data with Siamese U-Net, and the highest IoU score of 0.80 on the Sentinel-1 data using U-Net with early fusion. The combined use of Sentinel-1 and Sentinel-2 failed to bring significant improvement compared to Sentinel-2 based results, but this dataset may have the potential to boost Sentinel-1 based results with Sentinel-2 data for near real-time wildfire progression mapping.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
Series
IEEE International Symposium on Geoscience and Remote Sensing IGARSS, ISSN 2153-6996
Keywords
Wildfire Dataset, Change Detection, Burned Area Mapping, Deep Learning, U-Net, Siamese Network, Multi-source Data, Sentinel-1, Sentinel-2
National Category
Earth Observation
Identifiers
urn:nbn:se:kth:diva-326621 (URN)10.1109/IGARSS46834.2022.9884275 (DOI)000920916607212 ()2-s2.0-85140391453 (Scopus ID)
Conference
IEEE International Geoscience and Remote Sensing Symposium (IGARSS), JUL 17-22, 2022, Kuala Lumpur, MALAYSIA
Note

QC 20230508

Available from: 2023-05-08 Created: 2023-05-08 Last updated: 2025-02-10Bibliographically approved
Zhang, P. (2021). Deep Learning for Wildfire Progression Monitoring Using SAR and Optical Satellite Image Time Series. (Doctoral dissertation). Stockholm: KTH Royal Institute of Technology
Open this publication in new window or tab >>Deep Learning for Wildfire Progression Monitoring Using SAR and Optical Satellite Image Time Series
2021 (English)Doctoral thesis, comprehensive summary (Other academic)
Alternative title[sv]
Djupinlärning för övervakning av skogsbränders utveckling med hjälp av tidsserier av SAR- och optiska satellittidsserier
Abstract [en]

Wildfires have coexisted with human societies for more than 350 million years, always playing an important role in affecting the Earth's surface and climate. Across the globe, wildfires are becoming larger, more frequent, and longer-duration, and tend to be more destructive both in lives lost and economic costs, because of climate change and human activities. To reduce the damages from such destructive wildfires, it is critical to track wildfire progressions in near real-time, or even real-time.  Satellite remote sensing enables cost-effective, accurate, and timely monitoring on the wildfire progressions over vast geographic areas. The free availability of global coverage Landsat-8 and Sentinel-1/2 data opens the new era for global land surface monitoring, providing an opportunity to analyze wildfire impacts around the globe. The advances in both cloud computing and deep learning empower the automatic interpretation of spatio-temporal remote sensing big data on a large scale.

The overall objective of this thesis is to investigate the potential of modern medium resolution earth observation data, especially Sentinel-1 C-Band synthetic aperture radar (SAR) data, in wildfire monitoring and develop operational and effective approaches for real-world applications. This thesis systematically analyzes the physical basis of earth observation data for wildfire applications, and critically reviews the available wildfire burned area mapping methods in terms of satellite data, such as SAR, optical, and SAR-Optical fusion. Taking into account its great power in learning useful representations, deep learning is adopted as the main tool to extract wildfire-induced changes from SAR and optical image time series. On a regional scale, this thesis has conducted the following four fundamental studies that may have the potential to further pave the way for achieving larger scale or even global wildfire monitoring applications. 

To avoid manual selection of temporal indices and to highlight wildfire-induced changes in burned areas, we proposed an implicit radar convolutional burn index (RCBI), with which we assessed the roles of Sentinel-1 C-Band SAR intensity and phase in SAR-based burned area mapping. The experimental results show that RCBI is more effective than the conventional log-ratio differencing approach in detecting burned areas. Though VV intensity itself may perform poorly, the accuracy can be significantly improved when phase information is integrated using Interferometric SAR (InSAR). On the other hand, VV intensity also shows the potential to improve VH intensity-based detection results with RCBI. By exploiting VH and VV intensity together, the proposed RCBI achieved an overall mapping accuracy of 94.68% and 94.17% on the 2017 Thomas Fire and the 2018 Carr Fire.

For the scenario of near real-time application, we investigated and demonstrated the potential Sentinel-1 SAR time series for wildfire progression monitoring with Convolutional Neural Networks (CNN). In this study, the available pre-fire SAR time series were exploited to compute temporal average and standard deviation for characterizing SAR backscatter behaviors over time and highlighting the changes with kMap. Trained with binarized kMap time series in a progression-wise manner, CNN showed good capability in detecting wildfire burned areas and capturing temporal progressions as demonstrated on three large and impactful wildfires with various topographic conditions. Compared to the pseudo masks (binarized kMap), CNN-based framework brought an 0.18 improvement in F1 score on the 2018 Camp Fire, and 0.23 on the 2019 Chuckegg Creek Fire. The experimental results demonstrated that spaceborne SAR time series with deep learning can play a significant role for near real-time wildfire monitoring when the data becomes available at daily and hourly intervals.

For continuous wildfire progression mapping, we proposed a novel framework of learning U-Net without forgetting in a near real-time manner. By imposing a temporal consistency restriction on the network response, Learning without Forgetting (LwF) allows the U-Net to learn new capabilities for better handling with newly incoming data, and simultaneously keep its existing capabilities learned before. Unlike the continuous joint training (CJT) with all available historical data, LwF makes U-Net learning not dependent on the historical training data any more. To improve the quality of SAR-based pseudo progression masks, we accumulated the burned areas detected by optical data acquired prior to SAR observations. The experimental results demonstrated that LwF has the potential to match CJT in terms of the agreement between SAR-based results and optical-based ground truth, achieving a F1 score of 0.8423 on the Sydney Fire (2019-2020) and 0.7807 on the Chuckegg Creek Fire (2019). We also found that the SAR cross-polarization ratio (VH/VV) can be very useful in highlighting burned areas when VH and VV have diverse temporal change behaviors.

SAR-based change detection often suffers from the variability of the surrounding background noise, we proposed a Total Variation (TV)-regularized U-Net model to relieve the influence of SAR-based noisy masks. Considering the small size of labeled wildfire data, transfer learning was adopted to fine-tune U-Net from pre-trained weights based on the past wildfire data. We quantified the effects of TV regularization on increasing the connectivity of SAR-based areas, and found that TV-regularized U-Net can significantly increase the burned area mapping accuracy, bringing an improvement of 0.0338 in F1 score and 0.0386 in IoU score on the validation set. With TV regularization, U-Net trained with noisy SAR masks achieved the highest F1 (0.6904) and IoU (0.5295), while U-Net trained with optical reference mask achieved the highest F1 (0.7529) and IoU (0.6054) score without TV regularization. When applied on wildfire progression mapping, TV-regularized U-Net also worked significantly better than vanilla U-Net with the supervision of noisy SAR-based masks, visually comparable to optical mask-based results.

On the regional scale, we demonstrated the effectiveness of deep learning on SAR-based and SAR-optical fusion based wildfire progression mapping. To scale up deep learning models and make them globally applicable, large-scale globally distributed data is needed. Considering the scarcity of labelled data in the field of remote sensing, weakly/self-supervised learning will be our main research directions to go in the near future.

Abstract [sv]

Skogsbränder har funnits tillsammans med mänskliga samhällen i mer än350 miljoner år och har alltid spelat en viktig roll när det gäller att påverkajordens yta och klimat. Över hela världen blir skogsbränderna allt större, vanligareoch mer långvariga, och tenderar att bli mer destruktiva både när detgäller förlorade liv och ekonomiska kostnader, på grund av klimatförändringaroch mänsklig verksamhet. För att minska skadorna från sådana destruktivaskogsbränder är det viktigt att spåra utvecklingen av skogsbränder i nära realtid,eller till och med i realtid. Satellitbaserad fjärranalys gör det möjligtatt kostnadseffektivt, exakt och i rätt tid övervaka hur skogsbränder utvecklasöver stora geografiska områden. Den fria tillgången till data från Landsat-8och Sentinel-1/-2 som täcker hela världen öppnar en ny era för global övervakningav markytan och ger möjlighet att analysera effekterna av skogsbränderrunt om i världen. Framstegen inom både molntjänster och djupinlärning gördet möjligt att automatiskt tolkningen av rumslig och tidsmässig stora datafrån fjärranalys i stor skala.

Det övergripande målet med denna avhandling är att undersöka potentialenhos moderna jordobservationsdata med medelhög upplösning, särskiltSentinel-1 C-bands syntetisk aperturradar (SAR), för övervakning av skogsbränderoch att utveckla operativa och effektiva metoder för tillämpningar iverkligheten. I denna avhandling analyseras systematiskt den fysiska grundenför jordobservationsdata för tillämpningar vid skogsbränder, och de tillgängligametoderna för kartläggning av brända områden vid skogsbrändergranskas kritiskt med avseende på satellitdata, t.ex. SAR, optisk och SARoptiskfusion. Med hänsyn till dess stora förmåga att lära sig användbararepresentationer används djup inlärning som huvudverktyg för att extraheraförändringar orsakade av skogsbränder från tidsserier av SAR- och optiskabilder. På regional nivå har följande fyra grundläggande studier genomförts idenna avhandling, som kan ha potential att ytterligare bana väg för att uppnåmer omfattande eller till och med globala tillämpningar för övervakningav skogsbränder.

För att undvika manuellt urval av tidsmässiga index och för att belysaförändringar i brända områden som orsakats av skogsbränder, föreslog vi ettimplicit radarkonvolutionellt brännindex (RCBI), med vilket vi utvärderaderollerna för Sentinel-1 C-Band SAR-intensitet och fas i SAR-baserad kartläggningav brända områden. De experimentella resultaten visar att RCBI äreffektivare än den konventionella metoden med logförhållandedifferentieringnär det gäller att upptäcka brända områden. Även om VV-intensiteten i sigkan ge dåliga resultat kan noggrannheten förbättras avsevärt när fasinformationintegreras med hjälp av interferometrisk SAR (InSAR). Å andra sidanvisar VV-intensitet också potential att förbättra VH-intensitetsbaseradedetektionsresultat med RCBI. Genom att utnyttja VH- och VV-intensitettillsammans uppnådde den föreslagna RCBI en övergripande kartläggningsnoggrannhetpå 94,68

När det gäller scenariot för tillämpning i nära realtid undersökte och demonstrerade vi den potentiella Sentinel-1 SAR-tidsserien för övervakning av skogsbränders utveckling med hjälp av Convolutional Neural Networks(CNN). I den här studien utnyttjades de tillgängliga SAR-tidsserierna fråntiden före branden för att beräkna tidsmässigt medelvärde och standardavvikelseför att karaktärisera SAR-bakspridningsbeteenden över tiden och belysaförändringarna med kMap. CNN, som tränades med binäriserade kMaptidsserierpå ett progressivt sätt, visade god förmåga att upptäcka brändaområden i samband med skogsbränder och att fånga upp tidsmässiga förändringar,vilket visades på tre stora och påverkanfulla skogsbränder med olikatopografiska förhållanden. Jämfört med pseudomaskerna (binäriserad kMap)gav CNN-baserade ramverk en förbättring av F1-poängen med 0,18 för CampFire 2018 och 0,23 för Chuckegg Creek Fire 2019. De experimentella resultatenvisade att rymdburna SAR-tidsserier med djupinlärning kan spela en viktigroll för övervakning av vilda bränder i nära realtid när data blir tillgängligamed dagliga och timvisa intervaller.

För kontinuerlig kartläggning av skogsbränders utveckling föreslog vi ettnytt ramverk för inlärning av U-Net utan att glömma i nära realtid. Genomatt införa en begränsning av den tidsmässiga konsistensen för nätverksresponsengör inlärning utan glömska (LwF) det möjligt för U-Net att lära sig nyafunktioner för att bättre hantera ny inkommande data och samtidigt behållasina befintliga funktioner som man lärt sig tidigare. Till skillnad från kontinuerliggemensam träning (CJT) med alla tillgängliga historiska data gör LwFatt inlärningen av U-Net inte längre är beroende av historiska träningsdata.För att förbättra kvaliteten på SAR-baserade pseudoprogressionsmaskerackumulerade vi de brända områden som upptäcktes med optiska data somförvärvades före SAR-observationerna. De experimentella resultaten visadeatt LwF har potential att matcha CJT när det gäller överensstämmelsen mellanSAR-baserade resultat och optisk-baserad grundsanning, och uppnådde enF1-poäng på 0,8423 för Sydneybranden (2019-2020) och 0,7807 för ChuckeggCreek-branden (2019). Vi fann också att SAR:s korspolarisationsförhållande(VH/VV) kan vara mycket användbart för att lyfta fram brända områden närVH och VV har olika tidsmässiga förändringsbeteenden.

SAR-baserad förändringsdetektering lider ofta av variabiliteten hos detomgivande bakgrundsbruset, vi föreslog en Total Variation (TV)-reglerad UNet-modell för att lindra inflytandet av SAR-baserade bullriga masker. Medtanke på den lilla storleken på märkta uppgifter om skogsbränder användesöverföringsinlärning för att finjustera U-Net från förtränade vikter baseradepå tidigare uppgifter om skogsbränder. Vi kvantifierade effekternaav TV-regularisering för att öka konnektiviteten hos SAR-baserade områdenoch fann att TV-regulariserade U-Net kan öka noggrannheten i kartläggningenav brända områden avsevärt, vilket ger en förbättring på 0,0338i F1-poäng och 0,0386 i IoU-poäng på valideringsuppsättningen. Med TVregulariseringuppnådde U-Net som tränats med brusiga SAR-masker denhögsta F1- (0,6904) och IoU- (0,5295), medan U-Net som tränats med optiskreferensmask uppnådde den högsta F1- (0,7529) och IoU- (0,6054) poängenutan TV-regularisering. Vid tillämpning på kartläggning av skogsbrandsutvecklingfungerade TV-regulariserad U-Net också betydligt bättre än vanillaU-Net med övervakning av bullriga SAR-baserade masker, visuellt jämförbara med optiska maskbaserade resultat.

På regional nivå visade vi effektiviteten av djupinlärning på SAR-baseradoch SAR-optisk fusionsbaserad kartläggning av skogsbrandsutveckling. Föratt skala upp modeller för djupinlärning och göra dem globalt tillämpbarabehövs storskaliga globalt distribuerade data. Med tanke på bristen på märktadata inom fjärranalysområdet kommer svagt/självövervakad inlärning attvara vår huvudsakliga forskningsinriktning inom den närmaste framtiden.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2021. p. 83
Series
TRITA-ABE-DLT ; 2129
Keywords
Remote Sensing, Deep Learning, Wildfire, Burned Area Mapping, Synthetic Aperture Radar, Change Detection, Segmentation, Optical and Radar Image Analysis, Sentinel-1, Sentinel-2, fjärranalys, djup inlärning, skogsbrand, kartläggning av brända områden, Synthetic Aperture Radar, upptäckt av förändringar, segmentering, analys av optiska och radarbilder, Sentinel-1, Sentinel-2
National Category
Earth and Related Environmental Sciences
Research subject
Geodesy and Geoinformatics, Geoinformatics
Identifiers
urn:nbn:se:kth:diva-295725 (URN)978-91-7873-935-6 (ISBN)
Public defence
2021-06-15, Videolänk:https://kth-se.zoom.us/j/65483992232, Du som saknar dator /datorvana kontakta yifang@kth.se / Use the e-mail address if you need technical assistance, Stockholm, 14:30 (English)
Opponent
Supervisors
Note

QC 210526

Available from: 2021-05-26 Created: 2021-05-25 Last updated: 2025-02-07Bibliographically approved
Organisations
Identifiers
ORCID iD: ORCID iD iconorcid.org/0000-0001-9907-0989

Search in DiVA

Show all publications