NeuroBOLT: Resting-state EEG-to-fMRI Synthesis with Multi-dimensional Feature Mapping

We sincerely appreciate your valuable suggestions, which truly help us enhance the quality and readability of our paper from multiple aspects. Responses to your concerns are presented as follows:

Consistency and presentation details (W3-5, W8, W12):

• W3: We restructured the model overview in the intro and abstract, and included summarizing sentences to ensure a coherent and clear presentation of our method throughout the manuscript.
• W4: We have carefully reviewed the manuscript and ensured that we use a capital "F" only when "fMRI" appears at the beginning of a sentence, following standard conventions.
• W5: We have now moved the MSE metric to the appendix to create more space for displaying the mean and standard deviation of R values. We will include an interactive table that allows readers to easily zoom in/out on our project page.
• W8: We have now reviewed all equations and symbols throughout the manuscript to ensure they are standardized. In the revised version of the manuscript, we will include a notation section (table) in the appendix that defines all symbols and terms used in the equations to help readers better understand the mathematical content and ensure consistency throughout the manuscript.
• W12: The expression in question is intended to define the set of patch embeddings (i.e., to define the notation), rather than to present an equation.

Meaning of the model abbreviation (W1):

“NeuroBOLT” stands for “Neuro-to-BOLD-Translation.” We have added clarification of this abbreviation both in the abstract and introduction.

Related work (W2):

We have incorporated additional relevant literature and discussed their relevance to our work in the Introduction section of our revised manuscript, including the works from Calhas et al. [1-2] and Liu et al. [3-4].

Data and code availability (W6):

We will release all datasets, code, and model weights upon acceptance. This will provide the necessary resources for the community to validate, further explore, and build upon our models.

Training details of baselines (W7):

We ensured that all training hyperparameters (like batch size) were consistent with those used for our model across all the baselines. Our baseline models consist of (1) SOTA EEG encoders mentioned in [5-6] and (2) EEG-to-fMRI frameworks [7-8], as detailed in Appendix B.2. We strictly adhered to the model settings provided in their original papers when training. We have clarified this and added a table in our revised manuscript that specifies these parameter settings to ensure the results' replicability.

Statistical tests (W9):

We have now added statistical tests for all comparisons with other baseline models (please see General Response).

Theoretical contribution (W10):

Our study aims to address a key limitation in previous methods for learning the effective EEG representation useful for EEG-fMRI synthesis. We not only incorporate the SOTA EEG encoding framework but also emphasize the importance of more comprehensive frequency representations, which are often neglected in previous methods [1-4,6-8]. We propose a multi-level spectral representation learning approach that captures and aggregates a range of rapid to smooth fluctuations with adaptive windows. As demonstrated in Table 2 (ablation study), NeuroBOLT benefits from this design by effectively capturing dynamic representations and improving synthesis accuracy and fidelity. Furthermore, additional experiments on an auditory task dataset (please see the General Response) validate the robustness and applicability of our learned representations across neural data from different sites and scanners.

We believe our approach not only facilitates the translation between two neuroimaging modalities but also highlights the need for multi-scale spectral features for more effective EEG encoding in future research within this community. We will include more detailed theoretical derivations and explanations of spectral representations in the appendix to strengthen the theoretical foundation of our work.

Significance of 26 channels (W11):

For EEG, having fewer than 32 electrodes (26 in our case) is typically considered a small channel set, which presents challenges in accurately estimating neural activity in cortical and subcortical regions using traditional methods like EEG source localization, which often requires >64 channels [9]. However, our experiments show that even with fewer electrodes, we were able to reconstruct the fMRI signals across multiple brain regions in both intra- and inter-subject prediction scenarios. This finding suggests that our method is practical and effective, potentially broadening its applicability to real-world scenarios, such as clinical applications, where electrode count may be limited.

We genuinely appreciate the suggestions, and believe our paper will be improved with your feedback. Please let us know if you have additional questions!

[1] Calhas, David, et al. "Eeg to fmri synthesis for medical decision support: A case study on schizophrenia diagnosis."

[2] Calhas, David, et al. "Eeg to fmri synthesis benefits from attentional graphs of electrode relationships."

[3] Liu, Xueqing, et al. "Latent neural source recovery via transcoding of simultaneous EEG-fMRI."

[4] Liu, Xueqing, et al. "A convolutional neural network for transcoding simultaneously acquired EEG-fMRI data."

[5] Jiang, Wei-Bang, et al. "Large brain model for learning generic representations with tremendous EEG data in BCI."

[6] Yang, Chaoqi, et al. "Biot: Biosignal transformer for cross-data learning in the wild."

[7] Li, Yamin, et al. "Leveraging sinusoidal representation networks to predict fMRI signals from EEG."

[8] Kovalev, Alexander, et al. "fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships."

[9] Michel, Christoph M., et al. "EEG source imaging."

Here are the responses to <Point 3, Point 4 and references> (For responses to Point 1 and Point 2, please see the comment block above)

Point 3:

If we understand correctly, the reviewer suggests running a grid search for each baseline model to find the parameters that perform optimally on the current dataset and with the current evaluation metrics. In our study, we had adopted the parameter settings recommended by the authors. When evaluating baselines, it is common practice to use the parameters suggested by the original authors, due to the computational burden associated with a comprehensive parameter search across the search space for each application. Moreover, we would like to clarify that we used the same evaluation metrics as in $7,8$ , i.e., correlation coefficient (i.e., R values). For the other EEG encoding models, the tasks in the original papers are different (mostly classification tasks like detection of seizures, event type classification, etc.), so metrics like accuracy, AUC, Cohen’s Kappa, and weighted F1 were used in these papers.

To further address your concern regarding differences in dataset, we conducted additional experiments on the same dataset as in refs. $7,8$ (eyes open/close task fMRI), predicting the same brain regions, using the same settings and objective (i.e., intra-subject prediction). The results (R ± S.D) are shown below, with the best values in bold. We find that our model still outperforms $7,8$ .

Point 4:

• 4.1): We apologize for the confusion, but our intention here for saying "validate the robustness and applicability of our learned representations across neural data from different sites and scanners" was to convey that our model shows promising applicability to another dataset, even if it is collected in a different condition (task) and on a different scanner (see Global response and Part 1 in the one-page PDF). We now notice that the word ”robustness” could cause confusion so we will omit this word in our revised manuscript. Evaluating the influence of noise requires simulating different noise levels, which is a bit out of the current scope of this paper, but it is a very interesting future direction.

• 4.2): We had planned to provide further details on the building blocks of our model, including specific details of the multi-scale spectral representation learning module. Multiscale temporal representations and Short-Time-Fourier-Transform (STFT) are proving to be highly effective in representing neural data $9,10$ . While the components of this module have well-established theoretical foundations in signal processing, our key contribution lies in integrating these components to work effectively together, resulting in a novel method and application (shown through ablation studies in Section 4.4 Table 2). Our planned additions include the following:

1. Formally describe (i.e. provide mathematical notations) how we utilized STFT to convert EEG time sequences into time-frequency features.
2. Discuss how the dimensions of frequency features depend on the time-window length with equations illustrating this relationship, and emphasize the rationale for selecting specific window lengths.
3. Introduce the trade-off between temporal and frequency resolution, further clarifying the reasoning behind using an approach that considers multiple window lengths.

Thank you once again for your valuable comments. We hope our responses fully address your concerns.

References:

[1-6]: see refs. [1-6] in the Reviewer’s previous comment

[7] Li et al. “Leveraging sinusoidal representation networks to predict fMRI signals from EEG”

[8] Kovalev et al. “fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships.” arXiv preprint.

[9] Van De Ville et al. “When makes you unique: Temporality of the human brain fingerprint."

[10] Samiee et al. “Epileptic seizure classification of EEG time-series using rational discrete short-time Fourier transform”

Thank you very much for your additional comments. Please note that this year, authors are not permitted to upload the revised full manuscript or include links to external pages during this rebuttal period. We were only allowed to upload a one-page PDF (please kindly refer to our Global Response). Therefore, we are unable to provide the revised manuscript here (in response to your Points 1 and 4.2), but will upload it when permitted. Responses to your other new concerns are below:

Point 2:

We appreciate the reviewers' thorough examination of related work. Our rationale for including particular baselines, and further discussion of your reference list ([1-6]), is provided below:

**Ref. $2$ ** is an excellent review article on simultaneous EEG-fMRI, and we will add it to the other simultaneous EEG-fMRI articles that we have cited in our original manuscript (including Ritter et al., 2006; Laufs et al., 2003, 2006; Chang et al. 2013; de Munck et al., 2009). With regard to the DCM model in **Ref. $4$ **, we had not selected it as a baseline since it requires specifying stimulus onsets along with the neuroimaging data. Since no stimuli are presented during resting state, this model can not operate on resting-state data without major modifications. In discussing their future directions, the authors of **Ref. $4$ ** state: “Finally, the expansion of the current task-based analysis to the corresponding resting-state methodology, where an equivalent canonical microcircuit formulation for cross spectral data features, will be needed.” **Ref. $6$ **, aims to infer the effective (directed) connectivity of a brain network based on the complementary information provided by EEG and fMRI. This is interesting work too, but does not appear to provide a framework for inferring fMRI time courses from EEG.

As we mentioned in our first rebuttal, we indeed plan to cite **Ref. $3$ ** (which is the preprint version of * $3$

• in the previous response) and **Ref. $5$ ** (which uses the same model as * $1,2$
• in the previous response). For **Ref. $3$ **, we were unable to find the code, and some experimental settings are not specified in their paper, which would make a direct comparison potentially unfair and unreliable, so we did not include it as a baseline. For **Ref. $5$ **, the model only accepts 64-channel EEG as input (according to the authors’ GitHub repo and PyPi sites). Since we are using EEG data from 32-channel caps, we did not include this model as a baseline here, but look forward to doing so in future studies with 64-channel data. **Ref $1$ ** explored three conventional architectures: FCNN, CNN, and Transformer. However, the paper did not provide code/model parameters and offered limited information about the training process. In our baselines, we have included models that are similar to or more advanced versions of these architectures (see Appendix B.2 in the original manuscript: CNNs * $19, 23, 24, 32$
• and Transformers * $16, 44, 36, 32$ *).

We hope that our response helps to clarify the rationale behind the baseline models selected for this submission. While the simultaneous EEG-fMRI field is indeed large, the area of EEG-to-fMRI synthesis is currently a niche but rapidly emerging subfield. We will also include the referenced papers in our revised manuscript, which will appear on this forum when we are permitted to submit.

Please see our responses to <Point 3, Point 4 and references> in our next comment block

We thank the reviewer for the thoughtful review and encouraging feedback on our manuscript! We are delighted that the reviewer found our work novel and of interest to the community. We are very encouraged by the reviewers’ evaluation on this work opening up opportunities for multimodal neuroimaging analysis. We greatly appreciate the positive comments on the soundness of our model, the clarity of our manuscript, and the promise of our results. We have addressed their specific questions below and included additional details in our global response.

Evaluation on task fMRI:

Our motivation for focusing on resting-state fMRI data stems from the rich information that can be gained from naturally evolving brain dynamics. We sought to address the question of reconstructing resting-state fMRI signals since the spontaneous nature of the signals could present challenges beyond reconstructing block- or event-related fMRI task data. But we strongly agree with the reviewer that it is important to investigate the fidelity of our proposed model on fMRI data that contains a task. In this rebuttal, we now include an auditory task EEG-fMRI dataset for an in-depth evaluation, please kindly refer to the global response part 1 for details.

Data and code availability:

We will release all datasets on OSF, code on GitHub, and model weights on HuggingFace upon acceptance. This will provide the necessary resources for the community to validate, further explore, and build upon our models.

Statistical analyses:

Please kindly refer to the global rebuttal response, where we provide additional results from our statistical analyses.

Question 1:

“In-scan” stands for within-scan (i.e., intra-scan) prediction, where part of the scan is used for training and another part is used for testing. We clarified this terminology in the revised version.

Question 2:

All methods can be applied to both inter-subject and in-scan predictions. We would like to clarify that we evaluated BIOT for both inter-subject and in-scan (intra-subject) frameworks (as shown in Table 1 in our original manuscript). For the in-scan evaluation, we originally selected the state-of-the-art EEG encoders (LaBraM [1] and BIOT [2]), as representative baselines for comparison. However, in our analysis for this rebuttal, we have also evaluated the intra-subject prediction performance on the two EEG-to-fMRI baselines (Li et al.[3] and BERIA [4]) to provide a more comprehensive analysis. Please see the comparison below, which shows the means and S.D. of correlation between prediction and ground truth ( full MSE table will be included in the revised manuscript). The significance of the paired t-test between our model and other baselines is indicated as follows: *:p<0.05; **: p<0.01; ***: p<0.001.

Cu: Cuneus; He: Heschl’s gyrus; MF: Middle Frontal Gyrus Anterior; PA: Precuneus Anterior; Pu: Putamen; Th: Thalamus; GLB: Global Signal

Again, we genuinely appreciate the reviewer’s encouraging feedback and constructive suggestions! Please feel free to let us know if you have any further questions or comments.

[1] Jiang, Wei-Bang, et al. "Large brain model for learning generic representations with tremendous EEG data in BCI." arXiv preprint arXiv:2405.18765 (2024).

[2] Yang, Chaoqi, et al. "Biot: Biosignal transformer for cross-data learning in the wild." NeurIPS 36 (2024).

[3] Li, Yamin, et al. "Leveraging sinusoidal representation networks to predict fMRI signals from EEG." Medical Imaging 2024: Image Processing. Vol. 12926. SPIE, 2024.

[4] Kovalev, Alexander, et al. "fMRI from EEG is only Deep Learning away: the use of interpretable DL to unravel EEG-fMRI relationships." arXiv preprint arXiv:2211.02024 (2022).

We truly appreciate your excellent suggestions! We address specific concerns below. Please see the PDF in the general response for additional details and figures.

Evaluation on task fMRI

Our motivation for focusing on resting-state fMRI stems from the rich information that can be gained from naturally evolving brain dynamics. We sought to address the question of reconstructing resting-state fMRI signals since the spontaneous nature of the signals could present challenges beyond reconstructing block- or event-related fMRI task data. But we strongly agree that it would be more compelling if our proposed model is evaluated on task data, in addition to rest. In this rebuttal, we now include an auditory task EEG-fMRI dataset for additional evaluation, please kindly refer to the global response part 1 for details.

Lack of error bars/standard deviations

We have now revised our work to include error bars in figures and standard deviations in tables, and statistical significance. Please see the global response and the PDF attachment for details.

Model initialization and seeds

To ensure consistency during model training, we set a fixed seed. This clarification has been added to our revised manuscript. While this is certainly an important robustness test, we plan to focus on this point in a future study due to the large scope of the current analyses. We will raise this important point in the Discussion of the revised manuscript.

Single-ROI training

We now realize that the wording "generalize to other brain areas" can imply that a single trained model can predict fMRI signals from different brain areas. In fact, we had meant to convey that our modeling framework can be trained to reconstruct the fMRI signal from an arbitrary brain region (but, indeed, ROI-specific models are trained). We have now clarified the wording in an effort to avoid this misunderstanding.

Small N

We acknowledge the small sample size and agree that it is important to make this point clear to readers. We have carefully reviewed the entire manuscript and made sure this limitation is discussed more clearly in the main text.

Regarding pretraining, we would like to clarify that we do leverage the pre-trained weights from the EEG foundation model (LaBraM [1], trained on 2500 hours of various types of EEG data from around 20 datasets) for our “Spatiotemporal Representation Learning module.” In the original manuscript, we had briefly discussed pretraining in the last paragraph of the introduction section and in section B.1 of the appendix. We have now added clarifications across the manuscript (specifically in the Methods and in Section 4, Experiments: Implementation Detail), describing the pre-training procedure and its importance.

Moreover, the additional results on the task-based fMRI collected at a different site demonstrate that training our model on a relatively small sample of resting-state data shows promising potential to generalize to different task conditions and hardware.

[1] Jiang, Wei-Bang, et al. "Large brain model for learning generic representations with tremendous EEG data in BCI." arXiv preprint arXiv:2405.18765 (2024).

Question 1

Please also see our response to “Single-ROI training” above. The current framework takes multi-channel EEG as input and predicts the fMRI ROI signal from the region on which the model is trained. We would therefore expect the model trained for one ROI would tend to reconstruct other fMRI ROIs if they share correlated temporal fluctuations. In future work, we will extend the model to predict multiple fMRI ROIs signals at once. We are revising the main text accordingly.

Question 2

While our new task analysis presents an initial probe into this question, showing promising performance on task data acquired with different hardware/parameters, answering this question directly would require future work that systematically investigates altering scan parameters (while keeping the task condition, and ideally the subject, consistent) and would involve collecting more data. We believe this is a useful avenue for future work.

Question 3

This is a great point! Source localization is indeed a valuable method. However, we did not include source localization for the following reasons:

(1) Limited number of EEG electrodes: Our EEG data consist of 32 electrodes. After excluding the ECG, EOG, and EMG channels from the model input, this is reduced to 26 channels. However, larger numbers of channels (e.g., >64) may be needed for sufficiently accurate source localization [1-2].

(2) Hemodynamic filter between source electrical signal and fMRI signal: Even if we can map the EEG into source space, the output signals would be neural electrical signals. Our present study focuses on predicting fMRI hemodynamic signals, which are blurred/delayed relative to electrical signals. Although fMRI signals can be estimated by convolving neural signals with the hemodynamic response function (HRF), the shape and peak of the HRF vary across different brain regions and individuals, which would pose challenges for aligning between fMRI and source-reconstructed EEG signals.

However, we do think that source localization is an excellent future direction.

[1] Michel, Christoph M., and Denis Brunet. "EEG source imaging: a practical review of the analysis steps." Frontiers in neurology 10 (2019): 325.

[2] Michel, Christoph M., et al. "EEG source imaging." Clinical neurophysiology 115.10 (2004): 2195-2222.

We genuinely appreciate your commitment to ensuring the rigor of our paper and the constructive suggestions. These insights have significantly improved this work. We hope our rebuttal addresses your concerns and please let us know if you have any additional questions/comments!

The rebuttal provided by the authors addresses much of the concerns raised here. The requested error bars somewhat lower the impact of the model compared to baselines, as there is high variance and many baselines perform fairly comparably; however, there is demonstrative improvement. I have raised my score to a borderline accept.