Neural Transcoding Vision Transformers for EEG-to-fMRI SynthesisShow others and affiliations
2025 (English)In: Computer Vision-Eccv 2024 Workshops, Pt Xx / [ed] Canton, C Pont-Tuset, J DelBue, A Tommasi, T, Springer Nature , 2025, Vol. 15642, p. 53-70Conference paper, Published paper (Refereed)
Abstract [en]
This paper introduces the Neural Transcoding Vision Transformer (NT-ViT), a generative model designed to estimate highresolution functional Magnetic Resonance Imaging (fMRI) samples from simultaneous Electroencephalography (EEG) data. A key feature of NT-ViT is its Domain Matching (DM) sub-module which effectively aligns the latent EEG representations with those of fMRI volumes, enhancing the model's accuracy and reliability. Unlike previous methods that tend to struggle with fidelity and reproducibility of images, NT-ViT addresses these challenges by ensuring methodological integrity and higher-quality reconstructions which we showcase through extensive evaluation on two benchmark datasets; NT-ViT outperforms the current state-of-the-art by a significant margin in both cases, e.g., achieving a 10x reduction in RMSE and a 3.14x increase in SSIM on the Oddball dataset. An ablation study also provides insights into the contribution of each component to the model's overall effectiveness. This development is critical in offering a new approach to lessen the time and financial constraints typically linked with high-resolution brain imaging, thereby aiding in the swift and precise diagnosis of neurological disorders. Although it is not a replacement for actual fMRI but rather a step towards making such imaging more accessible, we believe that it represents a pivotal advancement in clinical practice and neuroscience research. Code is available at https://github.com/rom42pla/ntvit.
Place, publisher, year, edition, pages
Springer Nature , 2025. Vol. 15642, p. 53-70
Series
Lecture Notes in Computer Science, ISSN 0302-9743
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:kth:diva-374157DOI: 10.1007/978-3-031-91907-7_4ISI: 001544994800004Scopus ID: 2-s2.0-105014461109OAI: oai:DiVA.org:kth-374157DiVA, id: diva2:2022922
Conference
18th European Conference on Computer Vision (ECCV), Sep 29-OCT 04, 2024, Milan, Italy
Note
Part of ISBN 978-3-031-91906-0; 978-3-031-91907-7
QC 20251218
2025-12-182025-12-182025-12-18Bibliographically approved