MindSimulator: Exploring Brain Concept Localization via Synthetic fMRI

Dear Reviewer Atef,

Thanks for your constructive comments and valuable advice. We address your concerns one by one as below.

W1:

We have carefully read the two papers [1, 2] you mentioned and find that they are unsuitable to be used as our baseline methods for the following reasons:

• The inconsistency of stimuli category. In our study, we focus on the localization of visual concepts (elicited by visual stimuli). However, the data collected by Neurosynth primarily involve fMRI obtained under text-based stimuli or real-world physical stimuli. Such differences make the two completely incomparable.

• Subject differences. A more significant issue lies in the difference in subjects. Note that the regions of interest (ROIs) vary between individuals. Our method and fLoc share the same subjects in the NSD dataset, whereas the localization from Neurosynth/Text2Brain is unrelated to the NSD subjects, making comparisons infeasible.

W2:

Note that evaluating the semantic accuracy of synthetic fMRI data poses significant challenges. As you mentioned, using generated images from trained decoding models (MindEye2) introduces complexity and strong dependencies. However, our evaluation approach has two advantages:

• Interpretability: The generated images provide a more intuitive visualization of the visual semantic content derived from the fMRI data, facilitating further analysis and identifying potential deficiencies. In contrast, relying on fMRI latent representations reduces computation costs, but, falls short in intuitive visualization and explanations.
• High recognition and reproducibility: The trained MindEye2 models are widely recognized for their ability to extract and visualize visual information with fMRI data, capably offering a more authoritative and reproducible evaluation compared to a specific custom-trained voxel encoder.

Additionally, the method you proposed is also an efficient and lightweight strategy for fMRI semantic-level evaluation. Following your suggestion, we use the two-way classification accuracy of fMRI latent representations for evaluation. The results presented as follows show the best semantic accuracy of our MindSimulator. However, it can be observed that all values of these results are very high, resulting in insignificant differentiation and comparison. We have included these results in Appendix B.2.

Q1:

We must clarify that the resting-state fMRI is NOT the output of our Diffusion Estimator; in contrast, it is used as the input of our model. We replace pure Gaussian noise with the noised resting-state fMRI representation, which is designed to simulate the transition from the previous brain state (resting state) to the current brain state.

The approach of utilizing resting-state brain activity fMRI as the inference initialization is introduced as an exploration to incorporate the temporal dependencies of fMRI. Recent research [3] in fMRI encoding has shown that considering a subject's prior fMRI (referred to as memory state) can enhance the accuracy of encoding current fMRI data. Consequently, we attempted to leverage prior fMRI representations (resting states), alongside image embeddings, as conditions to guide the synthesis of fMRI data, allowing MindSimulator to simulate the brain's activity transitioning from resting states to activated states. We have provided more explanations in the revised version to provide a better understanding.

Q2:

The primary reason is that our encoding model cannot effectively generalize to fMRI data from other subjects, due to the substantial difference between subjects’ brains. Obviously, the experiment subjects in the NSD dataset and the THING-fMRI dataset are quite different. Our encoding model trained on NSD subjects is only capable of synthesizing fMRI for NSD subjects. When applied to an out-of-distribution dataset, such as CIFAR-10 or CIFAR-100, the synthetic fMRI also corresponds to NSD subjects. Consequently, the image-fMRI pairs from other subjects, such as those from THING-fMRI subjects, cannot be used as ground truth for evaluation.

References

[1] Tal Yarkoni et al. Large-scale automated synthesis of human functional neuroimaging data. Nature methods, 2011, 8(8): 665-670.

[2] Gia H. Ngo et al. Text2Brain: Synthesis of Brain Activation Maps from Free-Form Text Query. MICCAI 2021.

[3] Huzheng Yang et al. Memory Encoding Model. arXiv:2308.01175v1.

We sincerely appreciate your insightful feedback and thoughtful comments. We hope our response effectively addresses your concerns. Feel free to engage in further discussions, and your comments are greatly welcomed.

Best wishes,

All authors of Submission 180.

Thanks very much for your feedback. We would like to further clarify your concerns.

W1:

Thanks for pointing out the Grad-CAM-based method for comparison. The Grad-CAM-based method is unsuitable as a baseline for voxel localization for three primary reasons.

First of all, we briefly introduce the distinction between fLoc and Grad-CAM-based approaches concerning the localization of concept-selective voxels.

• fLoc involves conducting statistical analysis (a voxel-wise one-sample t-test) on a set of fMRI corresponding to natural stimulus (e.g., images) of a particular concept to locate concept-selective voxels.

• In contrast, Grad-CAM-based methods calculate the impact of voxels in single fMRI input on decoding a particular concept. This kind of approach generally involves text output or concept classification in fMRI decoding models to calculate concept-related gradients.

Then, we provide the reasons as below:

1. Since our work performs concept localization based on the synthetic fMRI generated by our MindSimulator, fLoc is introduced as an evaluation (rather than a comparative method) to verify the functional accuracy of the synthetic fMRI. Specifically, we treat the concept localization results of fLoc as ground truth to evaluate the concept localization accuracy from our synthetic fMRI, thereby reflecting the effectiveness of the synthetic fMRI. Note that there is no recognized concept localization of brain regions, that is, there is actually no ground truth concept localization for evaluation. From this point, our method and Grad-CAM-based methods are incomparable. This is why the work you mentioned presents heatmap visualization and visual reconstruction results instead of quantitative results to validate the concept localization.
2. Additionally, the Grad-CAM-based method uses single fMRI for concept localization, which may be coarse and questionable. According to the paper you mentioned, it locates the concept-selective voxels in the early visual cortex, which contradicts the consensus in neuroscience. Typically, the early visual cortex is responsible for processing basic features such as color and shape, rather than specific concepts. Anyway, since there is no recognized concept localization of brain regions, we cannot say our concept localization is better than theirs. From the heatmap visualization, its concept localization results are quite different from ours, indicating the comparison between our method and Grad-CAM-based methods is meaningless and unfair.
3. Finally, the number of selective concepts of fLoc is limited by the given concepts in practical subject stimulation experiments, that is limited experimental data. Our work, similar to recent work on concept localization, e.g., Shen’s work, aims to leverage generative models to extend concept localization to arbitrary concepts (ideally), addressing the issue of limited experimental data. Differently, our work builds an image-to-fMRI generation model, aiming to simulate the human brain to generate quantitatively and functionally accurate fMRI. Then, we can utilize the generation models to synthesize fMRI corresponding to an arbitrary concept leveraging images of the concept. Grad-CAM-based methods generally adopt fMRI as input and learn to align fMRI to the semantic text or labels (i.e., concepts) of visual stimulus. Then, these methods capably visualize the “correlation” between fMRI and various concepts within the visual stimuli. Grad-CAM-based methods are also subject to the limited concepts in all visual stimuli. Accordingly, from the methodology of concept localization, our method is quite different from Grad-CAM-based methods, confirming it is unsuitable to compare ours with Grad-CAM-based methods.

Given the aforementioned reasons, we did not compare our work with Grad-CAM-based methods. It is conceivable that in the future, these two approaches may converge towards a unified concept localization. However, at present, they remain distinctly incomparable.

Q2:

Thanks for your comments and suggestions. We regret not well introducing and explaining the out-of-distribution (OOD) generalization setting in our previous discussion, which may have led to your confusion and misunderstanding. Here, we introduce the motivation and setting of our OOD generalization experiments for your reference.

Motivation of the OOD experiments:  As we discussed above, our work aims to simulate the human brain to generate quantitatively and functionally accurate fMRI for any visual stimulus and extend concept localization to arbitrary concepts. Accordingly, we introduce other image datasets (i.e., CIFAR used in the experiments) to verify the capability of our MindSimulator in synthesizing high-quality fMRI for the visual stimulus of OOD concepts. The OOD experiment is to explore the question: since the images used to train MindSimulator come from MSCOCO, can MindSimulator also accurately synthesize fMRI when images come from other datasets?

Setting of OOD experiments: Based on the above motivation, we can outline the setting for OOD generalization experiments. First, we need an image-fMRI pair dataset $D_A$ to train the MindSimulator. Then, we synthesize fMRI using the images from dataset $D_B$ and evaluate the corresponding synthesis accuracy. This process is similar to zero-shot generalization. In our experiment, dataset $D_A$ is NSD, and $D_B$ is CIFAR-10/100. In the evaluation of OOD experiments, we verify the semantic-level accuracy of synthetic fMRI (Table 2) and visualize the reconstructed images from fMRI for validation. Additionally, note that our subject-specific MindSimulator is trained based on the image-fMRI pairs of a single subject. We did not intentionally select image-fMRI datasets for evaluation due to the fact that the substantial difference between subjects renders fMRI for other subjects are meaningless for comparison, which is the primary point we focused on in the last round of discussion.

Regarding the dataset generalization you mentioned:

• We first explain the reason for our settings. Since NSD is highly representative, many representative studies [1-6] have conducted experiments solely on the NSD dataset. Following previous studies, we also conduct experiments only on the NSD dataset.

• THINGS-fMRI and NSD have a comparable number of image samples (8640 vs. 10000). Intuitively, our MindSimulator has the potential to effectively generalize to the THINGS-fMRI dataset.

• Since we have not worked with this dataset before, it is difficult to complete the dataset generalization experiment within one week, especially considering the complicated fMRI preprocessing. However, we will implement it in the future and include it in the final version of our paper.

References

[1] Andrew F. Luo et al. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. NeurIPS 2023.

[2] Andrew F. Luo et al. Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers. arXiv:2410.05266.

[3] Paul S. Scotti et al. Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. NeurIPS 2023.

[4] Paul S. Scotti et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICML 2024.

[5] Huzheng Yang et al. Memory Encoding Model. arXiv:2308.01175v1.

[6] Hossein Adeli et al. Predicting brain activity using transformers. bioRxiv.

Q1:

The misuse of the comma is our oversight, and we have corrected the error in the revised version.

Q2:

We used noised resting-state fMRI representations as initialization for iterative denoising. This design was introduced to allow MindSimulator to simulate the brain's activity transitioning from resting states to activated states. Recent research [1] in fMRI encoding has shown that considering a subject's prior fMRI (referred to as memory state) can enhance the accuracy of encoding current fMRI data. Consequently, we attempted to leverage prior fMRI representations (resting states), alongside image embeddings, as conditions to guide the synthesis of fMRI data.

Note that subjects viewed different images consecutively in the NSD experiment. We have also experimented with utilizing noised prior fMRI representations instead of pure noise as a starting point of the iterative denoising process. Regrettably, these approaches proved ineffective, which may be caused by the complexity of fMRI data collection experiments or the intricate mechanisms involved in the transition of brain activity. Finally, we chose to use noised resting-state fMRI as the inference initialization. We have provided more explanations in the revised version to provide a better understanding. Although it lacks further exploration and sufficient explanations of the effect of resting-state brain activity fMRI, it is intriguing and worthwhile to explore the transition of brain activity in brain encoding/decoding analysis.

Q3:

Certainly, we have put the ROI boundaries on our flat maps. Please refer to the Figure 8 of our revised version. Thanks for your valuable suggestion.

Q4:

In Section 6.1, we used the MSCOCO images in the NSD experiment (a total of 73,000 images, including 9,000 images unique to each subject and 1,000 images shared among subjects). Note that each subject had fMRI data corresponding to only 10,000 out of 73,000 images. We have clarified this in Appendix A.4 of our revised version.

Q5:

In Appendix A.4, we provide the t-test threshold, set to 2 following the settings in [2]. We will further provide the t-test results in our code repository. Moreover, the results reported in our paper are averaged values across 3 comparisons under different random seeds and we will clarify it in Table 4 of the revised version.

References

[1] Huzheng Yang et al. Memory Encoding Model. arXiv:2308.01175v1.

[2] Andrew F. Luo et al. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. NeurIPS 2023.

[3] Hossein Adeli et al. Predicting brain activity using transformers. bioRxiv.

[4] Alessandro T. Gifford et al. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. arXiv:2301.03198.

We sincerely appreciate your insightful feedback and thoughtful comments. We hope that our response effectively addresses your concerns. Feel free to engage in further discussions, and your comments are greatly appreciated.

Best wishes,

All authors of Submission 180.

Regarding the performance:

• Quantitative results: Note that, after ablating key components, the corresponding performance shows a significant decrease in terms of voxel-level metrics such as MSE and Pearson. However, semantic-level metrics have relatively high values but low differences owing to their calculation method. For instance, the four metrics—Alex(2), Alex(5), Incep, and CLIP—are based on binary classification accuracy, with a minimum value of 50%. That is, as long as the images contain certain semantic information, the classification accuracy will reach a relatively high level.

• Baselines: Due to the limited research focusing on fMRI encoding, there are fewer suitable baseline methods for comparison. Our linear model baseline essentially represents encoding models widely used in neuroscience research. The Transformer encoding model we adopted follows the structure and training process in reference [3]. Notably, the encoding model from [3] achieves desirable performance (2nd place) in the Algonauts Challenge [4], signifying its superiority to the majority of methods employed in the fMRI encoding challenge. As a result, the baselines selected for the experiments are representative, yielding convincing experimental results.

• Better fMRI representation: The goal of our paper is to enhance the synthesis quality of fMRI. To achieve this, we have made every effort to find suitable baseline methods and comprehensive metrics for evaluation. We have also explored diverse approaches to enhance the performance of our generative encoding model. Accordingly, we argue our extensive experiments verify the effectiveness of our encoding model.

Regarding the missing references:

Thank you for pointing out the key reference. Neglecting to cite the original fLoc paper is indeed an oversight on our part. We have cited the reference in the revised version.

Regarding the claims in our Introduction:

Our wording in the introduction section may lead to the misunderstanding. Utilizing the fLoc experiment for conceptual localization is accurate and effective. What we refer to as "insufficient" pertains to the number of concepts. Since fLoc relies on specially curated visual stimuli that require manual processing, such datasets are rare. Therefore, when attempting to localize a less-studied concept, it is highly likely that there will be few or no directly available images. Similarly, our reference to "generalizability" stems from this issue: without directly available data, fLoc is difficult to generalize to the localization of new concepts. In contrast, natural scenes offer rich visual stimuli in terms of both quantity and semantics, potentially providing better generalizability. Regarding the effectiveness of natural stimuli, we have discussed in the above.

In addition, our proposed strategy of using MindSimulator to synthesize fMRI data for concept-selective localization is not intended to replace fLoc. Instead, we regard it as a complementary and auxiliary approach to fLoc, particularly in scenarios where manually constructing numerous visual stimuli and collecting corresponding fMRI data pose challenges.

Dear Reviewer SRDC,

Thanks for your constructive comments and valuable advice. We will address your concerns one by one.

W1:

Thank you for your suggestions, we have included appropriate captions for each table and figure to enhance the clarity and comprehensiveness of their descriptions.

W2:

We have carefully revised the statement of the claims you mentioned. Please refer to our response Regarding our claims in Introduction below.

W3:

Thank you for pointing out the citation formatting issue. We have carefully examined all citations and made corresponding revisions in the revised version.

W4:

We have modified the presentation of the "Introduction" section to make it more rigorous. If there are any other shortcomings, please do not hesitate to point them out.

W5:

Thank you for your suggestion. We have made a careful proofreading to improve the quality of the paper writing.

Regarding the random noise as input:

I have carefully read the two papers you provided.

• Prof. Kamitani's paper does not mention that visual semantics can be decoded from pure noise. He argues that current research on neural decoding has overstated its reconstruction performance, and the generalization, authenticity, and stability of decoding models need further improvement.
• The second paper does report that linguistic information can be decoded from noise. We speculate this is because its EEG-to-text decoding model essentially fine-tunes pre-trained large language models using small datasets. This fine-tuning process can somehow overfit limited noise data. In contrast, similar issues might not exist in fMRI visual decoding models like MindEye2, since fMRI-to-image embedding generation is trained from scratch.

Based on the literature you provided, we acknowledge that our statement, "If it can be reconstructed, then the fMRI contains the information," lacks the necessary rigor. Indeed, we should provide some premises or constraints. To clarify, successful decoding can only suggest that the fMRI holds the relevant visual information when the testing data aligns with the generalization capacity of the trained visual decoding model. We have refined the statement in the revised manuscript.

To further address your concerns, we conducted experiments replacing fMRI with pure noise as the input to the trained fMRI visual decoding model. The results, detailed in our Appendix B.3, demonstrate that no clear visual semantics can be reconstructed from noise.

Regarding the natural stimuli:

Thank you for sharing your insightful perspective. Your concerns are valid and thought-provoking. We would like to gently point out that you may overlook the benefits brought by the large-scale fMRI available. Indeed, when dealing with a limited amount of stimulus and corresponding fMRI data, the intricacies of natural scenes can introduce irrelevant factors that might impede the accuracy and efficacy of concept localization. In such scenarios, manually excluding these irrelevant factors could prove advantageous. However, when utilizing the vast fMRI data generated by MindSimulator, the potential exists to mitigate the influence of the irrelevant factors. Taking food images as an example, due to the large amount of food images, the image collection covers a variety of colors. Also, the probability of repetitive activations of voxels in the lower-level visual cortex associated with specific colors is low, while all activations are associated with specific food concept. Subsequent to the t-test and threshold filtering, only the consistently activated food-selective voxels are selected and localized, instead of the color-related voxels. Therefore, leveraging large-scale synthetic fMRI could effectively minimize the impact of irrelevant factors in natural scenes. Based on these considerations, we argue that visual stimuli of natural scenes can indeed be utilized to explore the localization of concepts. Certainly, the above considerations require further solid evidence and exploration, which is an emerging research field attracting increasing attention and precisely our future research direction.

I am one of the authors of BrainDiVE.

I wanted to quickly provide my thoughts for MindSimulator: Exploring Brain Concept Localization via Synthetic fMRI.

I have carefully read this paper and I will summarize it as follows:

1. The authors train an fMRI autoencoder with a latent space that is regularized by CLIP
2. They train an diffusion transformer which synthesizes fMRI responses conditioned on the CLIP image embedding
3. They take the expectation of the fMRI response by sampling from the diffusion model multiple times, and then taking an average.

Pros:

1. I think the approach overall of a diffusion fMRI encoder is quite novel. Indeed the fMRI response is stochastic, so this design does make sense to me.
2. Without the use of gradients, computational tests of selectivity can be done much more quickly.

Questions & weaknesses:

1. I'm a bit unsure about the proposed evaluation metrics for fMRI encoding performance. Using pearson R or R^2 is standard in fMRI encoder literature. Using a decoder as part of the evaluation process introduces additional complications.
2. Using Resting-State Brain Activity fMRI as the inference initialization is a bit strange, in my view this is not well justified
3. Using Correlated Gaussian Noise as the multi-trial seed is not well justified.
4. The fMRI beta pre-processing stage is a bit unclear. Do the authors use all three repeats of the same image individually? Or do the authors average the beta values?

Minor issues:

1. Typo in Figure 4 left text? Trial = 1 seems to be repeated twice

If I were the reviewer of this paper, I would give this paper a 6 (weak accept) prior to rebuttal.

Based on the discussions presented, along with the points raised by the author of the referenced paper and the explanations provided by the current authors, I acknowledge my initial oversight and have revised the score accordingly. I appreciate the concept explored in this paper and find it both timely and relevant. However, I remain uncertain about the extent of its technical novelty compared to the previous work.

Dear Reviewer KVAs,

There are distinct differences between our paper and BrainDIVE. We outline the key disparities in terms of methodology, evaluation, and exploration of neuroscience for your reference.

Regarding the methodology:

Typically, to generate a synthetic fMRI, three key components are required: an input image, an image embedder like ViT, and an encoding model that projects the image embedding to fMRI data.

• BrainDIVE concentrates on refining the input images, enhancing their quality by fine-tuning Stable Diffusion conditioned on specific brain regions with maximum activation.
• In contrast, our focus lies on optimizing the encoding model. We introduce a generative encoding model instead of BrainDIVE's Linear model, aiming to enhance accurate encoding. Intuitively, our model can collaborate with BrainDIVE to enhance the overall quality of fMRI encoding.

Regarding the evaluation:

Overall, the evaluations of our work and BrainDIVE are significantly different.

• BrainDIVE evaluates images to check if the generated images can notably activate specific brain regions.
• In contrast, our focus lies on evaluating fMRIs to compare the accuracy of synthetic fMRIs generated by our MindSimulator against baselines at both voxel and semantic levels.

Regarding the neuroscience exploration:

While our work and BrainDIVE share the goal of locating concept-selective regions in the brain, our strategies differ fundamentally.

• BrainDIVE localizes concept-selective regions by generating images that activate specific brain areas, followed by analyzing image semantics to deduce the related concepts.
• In contrast, our method initiates by selecting images according to predetermined concepts. Subsequently, we employ MindSimulator to simulate functional localizer experiments, thereby identifying the concept-selective regions.

We hope that our response addresses your concerns. We are eager to receive your feedback. If you have any further questions or concerns, please do not hesitate to inform us. Thanks for your comments.

Best wishes,

All authors of Submission 180.

We greatly appreciate your feedback and raising the rating. Regarding your remaining concerns about the novelty of our work, we would like to provide further clarification from three aspects.

1. Novelty of the encoding model:

Our proposed fMRI encoding model is an innovative generative model and essentially distinct from previous fMRI encoding models.

First, to the best of our knowledge, previous existing methods are discriminative regression encoding models, where the regression model only yields a unique certain fMRI for a given image. However, we argue such an encoding process is different from the nature of human brains. As we mentioned in Section 3.1, observations show that there are noticeable differences in the individual’s brain activity fMRI recordings even when receiving the same visual stimuli. Accordingly, we utilize a generative model to estimate the conditional distribution of p(fMRI|image), guaranteeing the uncertainty of brain activity encoding.

Additionally, we have introduced special designs in our generative encoding model, as outlined below:

• We innovatively trained an fMRI autoencoder to construct the fMRI representation space. Based on this, we were able to learn the conditional distribution in the latent space, which made the training process more stable and resulted in better results.

• We proposed a novel sampling strategy, multi-trials enhancement, which enhances the voxel reproducibility by averaging the fMRI obtained from multiple encoding processes. We further introduced correlated noise, which increases the correlation between multiple generated fMRI and reduces their variability, thereby achieving more stable fMRI synthesis.

2. Novelty of the evaluation techniques:

We introduce a novel and effective semantic evaluation method leveraging well-trained fMRI-to-image decoding models. This approach assumes that the trained decoding model effectively reconstructs semantic information from (synthetic) fMRI data. Specifically, we utilize the decoding model to reconstruct images from synthetic fMRI and employ visual semantic metrics to measure the semantic quality of the reconstructions. The evaluation technique leverages image semantic assessment to indirectly validate and assess the semantic content embodied within synthetic fMRI data, offering a compelling approach to visualize the semantics embedded in synthetic fMRI. Notable, the new evaluation approach is effective and insightful, contributing to the field of brain activity encoding.

3. Novelty of the concept localization techniques:

Our proposed encoding-based concept localization technique is powerful and distinct from previous ones. Note that conventional functional localizers (fLoc) that rely on practical experiments of natural stimulus can only localize concepts given by the available stimulus set. In contrast, our approach leveraging the synthetic fMRI capably utilizes arbitrary natural scene images for localization. It has the potential to allow for the localization of any given concept and multi-granular concepts.

Thanks for your constructive response and valuable suggestion. We address each point in detail and make corresponding updates in the revised version.

Q1

According to your suggestion, we have changed the colors used in the $R^2$ visualization in Figure 10.

Q2

In our experiments, the timestep ratio $\bar{\alpha_T}$ of resting-state fMRI was set to 0.01. We have conducted ablation experiments on the resting-state fMRI, with the results presented below and detailed further in Appendix B.6.

As you expected, the results indeed show that the utilization of resting state initialization does not show a significant improvement. This may be attributed that

1. the resting state fMRI is not actually the prior state of each sample during the practical image-fMRI data collection;   

2. the generation process during inference does not align exactly with the brain's activity transition from resting states to activated states.

Naturally, it is interesting yet exceedingly difficult to uncover the transition aligning with natural stimulus. Additionally, while our work tries an initial exploration into resting state initialization, it does not constitute the core novelty or contribution of our research.

Q3

Our objective is to estimate the conditional distribution p(brain∣image), which is, however, highly idealized. In the NSD dataset, each image has only three associated fMRI scans, making accurate estimation challenging due to the limited data. Consequently, the trained diffusion estimator in this setting remains imperfect and requires the addition of correlated noise for further refinement.

We further ablate the correlated noise with 5 independent noises, and the results are shown in the following table. It can be seen that the introduction of correlated noise is beneficial for fMRI encoding. The results confirm our expectation that generating multi-trial synthetic fMRI that are highly correlated with each other can better enhance voxel-wise reproducibility.

We have also added the corresponding results in Appendix B.7.

Reference

[1] Zixuan Gong et al. NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction. NeurIPS 2024.

[2] Andrew F. Luo et al. Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers. arXiv:2410.05266.

[3] Paul S. Scotti et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICML 2024.

[4] Zixuan Gong et al. Lite-Mind: Towards Efficient and Robust Brain Representation Learning. ACM MM 2024.

[5] Shizun Wang et al. MindBridge: A Cross-Subject Brain Decoding Framework. CVPR 2024.

[6] Ruijie Quan et al. Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity. CVPR 2024.

[7] Weihao Xia et al. DREAM: Visual Decoding from Reversing Human Visual System. WACV 2024.

[8] Andrew F. Luo et al. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. NeurIPS 2023.

[9] Paul S. Scotti et al. Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. NeurIPS 2023.

[10] Guobin Shen et al. Neuro-Vision to Language: Enhancing Visual Reconstruction and Language Interaction through Brain Recordings. NeurIPS 2024.

[11] Zijiao Chen et al. CinematicMindscapes: High-quality Video Reconstruction from Brain Activity. NeurIPS 2023.

[12] Huzheng Yang et al. Memory Encoding Model. arXiv:2308.01175v1.

[13] Hossein Adeli et al. Predicting brain activity using transformers. bioRxiv.

[14] Alessandro T. Gifford et al. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. arXiv:2301.03198.

We sincerely appreciate your insightful feedback and thoughtful comments. We hope that our response effectively addresses your concerns. Feel free to engage in further discussions, and your comments are greatly appreciated.

Best wishes,

All authors of Submission 180.

W3:

Thanks for your suggestions.

• Regarding specific baselines. The Transformer encoding model is a specific baseline. It has the same structure and training process as the method presented in the paper [13]. Notably, the encoding model achieves desirable performance (2nd place) in the Algonauts Challenge [14], signifying its superiority to the majority of methods employed in the fMRI encoding challenge. Therefore, the baselines selected for the experiments are representative and could yield convincing experimental results.

• Regarding connectivity-based metrics. Following your suggestions, we performed further experiments based on the connectivity-based evaluation metrics. First, we separately computed the functional connectivity maps for both the synthetic and real fMRI data. Specifically, we adopted the ROIs (words, faces, bodies, places, others) provided by NSD and performed dimensionality reduction on the voxels within each ROI using principal component analysis to guarantee the same length of fMRI. Subsequently, we calculated the Pearson correlation among the different regions to generate the functional connectivity maps. The similarity between the functional connectivity maps of the synthetic and real fMRI data was then evaluated using the mean absolute error. The results reported in the following table show that our MindSimulator achieves the best performance, confirming the superiority of our MindSimulator. We have included the results in Appendix B.2 of the revised version. We will further complete relevant experiments and incorporate these results and corresponding discussion into the main text.

W4:

All of the training was done on NVIDIA Tesla V100 32GB GPU. For a single subject, the training is about 12 GPU hours for the fMRI autoencoder and 20 GPU hours for the diffusion estimator. During the inference phase, it takes only an average ~300ms to synthesize an fMRI. We have included the details in Section 4.2 of the revised version.

Q1:

1. Regarding spatial gradients:

We compute the spatial gradients of each sample in the test set across all directions and then calculate the absolute difference between the ground-truth fMRI and the synthetic fMRI. The average results of each voxel are reported as follows:

It can be observed that in terms of spatial gradients, the results of different methods are similar. The performance of MindSimulator is not optimal since we do not impose a special constraint on the spatial gradient. We have included the results in Appendix B.2 of the revised version. We will further complete relevant experiments and incorporate these results and corresponding discussion into the main text.

2. Regarding the temporal gradients:

In the NSD dataset, different time points correspond to different image-fMRI pairs. Therefore, the gradient along the temporal dimension does not make sense in evaluating the fMRI encoding performance.

Q2:

Our task focuses on the synthetic quality of fMRI, however different timepoints correspond to different fMRI samples. Exploring the similarity of functional connectivity maps across the temporal dimension does not make sense for evaluating the encoding accuracy. Certainly, comparing the similarity of functional connectivity maps between synthetic fMRI and ground-truth fMRI is highly significant, and we have provided the corresponding evaluation results in the response to W3.

Dear Reviewer YLHx,

Thanks for your constructive comments and valuable advice. We address your concerns point by point.

W1:

First of all, we need to point out that the NSD dataset in the experiments does NOT contain temporal information. 1) The image-fMRI pairs in the NSD dataset are collected when participants view a static image. 2) The collected fMRI corresponding to each image is averaged along the temporal dimension and reduced to 3D, to enhance voxel reproducibility.

Although spatial dependencies in fMRI may provide information to enhance fMRI representation, the spatial structure in fMRI is intricate and brings extreme difficulties to brain encoding/decoding, especially fMRI reconstruction. Accordingly, the latest related studies [1, 2, 3] generally adopt flattened fMRI as input and focus on improving the design of generative models.

In addition, we have made some important initial explorations to incorporate temporal and spatial dependencies in the experiments.

• Regarding spatial dependencies: We have explored extracting voxel tokens (patches) with padding and 3D-convolution on the original un-flattened fMRI and added them with cosine/learnable position encoding. However, such a design does NOT lead to performance improvement on the fMRI autoencoding task. Instead, it introduces additional computational overhead because the computational complexity of Transformer architecture used for voxel tokens is higher than that of MLP used for 1D voxel sequences.

• Regarding temporal dependencies: We have considered the temporal dependence of fMRI in our Diffusion Estimator. Specifically, we have used previous fMRI representations, together with image embedding, as conditions to guide the synthesis of the current fMRI. We have also tried to utilize noised previous fMRI representations instead of pure noise as a starting point for the iterative denoising process. However, these attempts do NOT work, introducing additional computational overhead and increasing the instability of the convergence process.

Based on the above attempts, our auto-encoding model still flattens fMRI without incorporating spatial and temporal factors. Certainly, integrating temporal and spatial dependencies is extremely challenging in brain encoding/decoding tasks. Nevertheless, it holds great promise and stands to greatly benefit comprehensive brain analysis and our future research focus.

W2:

First, we clarify our voxel selection and masking process as follows:

The standardized fMRI sample is a 3D matrix. We used nsdgeneral, which is a manually delineated ROI mask by NSD, represented as a 0/1 matrix, to dot-multiply with the raw fMRI data. After masking, the retained voxels are irregularly shaped but possess connectivity in 3D space. Subsequently, we flattened the 3D masked fMRI into 1D in the order of length-width-height and then removed the zero values (i.e. masked voxel position).

Indeed, the above fMRI preprocessing considers less-to-none spatial context information. However, our process is widely adopted by plenty of previous work [1-9] and has been verified effective to yield good results in brain encoding/decoding. Additionally, we explored the method you mentioned (please refer to W1), but it failed to achieve desirable performance in the fMRI auto-encoding task. It may be attributed that spatial dependencies in fMRI are intricate and bring extreme difficulties to fMRI reconstruction. Accordingly, we adopt the above fMRI preprocessing and pay more attention to the design of the generative encoding model and the semantic evaluation of synthetic fMRI in this work. In our recent work, we are exploring the contribution of the temporal and spatial dependence in fMRI to brain encoding/decoding and will also consider involving more elaborated fMRI embedding methods such as [10-12] to enhance fMRI auto-encoding performance.