Toward Generalizing Visual Brain Decoding to Unseen Subjects

We thank the reviewer for the detailed feedback on our paper. We would like to clarify that the purpose of our study is to explore methods to meet a major challenge — model’s generalization capabilities to unseen subjects.

1.Q: fMRIs for each image may not accurately capture the brain's response.

R: Accurately matching each video frame to brain response is nearly impossible. fMRI measures changes in the brain's blood oxygen levels, which do not correspond in a real-time one-to-one manner. However, there are many approved fMRI-video frame retrieval studies [1][2][3][4] which either use a delayed fMRI response (due to hemodynamic delay) or average fMRI voxels over a few seconds. These methods are widely accepted in the academic community, so we followed them [1]. Regarding the validation experiments, our stable training results across different subjects confirm the reliability of our data.

2.Q: The notable performance disparity observed between the NSD dataset and the paper's dataset. (Table 2, row 1, compared to Table 6, row 2)

R: This appears to be a misunderstanding. The large performance disparity likely arises from two factors: (1) The NSD experiments in Table 6 test on 300 images (see line 487), while the experiments in Table 2 retrieve from 100 images (line 350). This inherently leads to differences in performance. (2) The NSD dataset presents a more challenging task with more scenarios, so a lower retrieval performance is reasonable. This performance disparity does not indicate that our dataset is problematic.

3.Q: MindEye1's top-1 accuracy is mismatched, and our implementation is unclear. R: This appears to be a misunderstanding. Our 84% accuracy aligns with the MindEye1 [5] training using CLIP loss (please see the second row of Table 3: 83.7% in the MindEye1 paper, also refer to the settings in line 346 of our paper). Additionally, when we reproduced MindEye1 with BiMixCo + SoftCLIP, the result was 93%, which is perfectly aligned with the original result. We do not report this in the paper, as our main experiments use CLIP loss. The reason for using CLIP loss in our experiments is that we want to verify a specific phenomenon, not to achieve SOTA results (see lines 463-465 of our paper).

4.Q: Hand-picking is not ideal for data collection, yet it is good for model performance.

R: We completely agree and acknowledge in lines 232-237 that hand-labeled data yields better performance, as it is directly linked to specific brain regions responsible for visual processing. Further, it is not ideal for research generalization. When working with hundreds or thousands of subjects, hand-labeling becomes impractical. Our approach prioritizes the scalability of data to explore generalization across a large number of subjects, rather than focusing solely on performance improvements. Therefore, hand-picking is not ideal for our work.

5.Q: Table 6, row 7, indicates extremely poor performance.

R: The poor performance here is expected. It reflects the challenge of generalizing when the number of subjects is small. The model was trained on Subj1, 3, 4, 6, 7 and tested on Subj2, 5, which demonstrates the difficulty of generalizing to unseen subjects with a limited number of training subjects.

As discussed in Questions 3, 4, and 5, our research focuses on finding generalization capabilities to unseen subjects and exploring the possibility of training foundational large brain decoding models. Understanding the distinction between "seen" and "unseen" subjects is necessary. Previous work has typically focused on training and testing with seen subjects, unlike our setting. Prior to this work, researchers were unsure whether generalizing to unseen subjects was even possible. We do not focus on achieving SOTA results, as this is not the goal of our paper.

—————————Because of the character number limit, we split the responses into two comments.——————————

For all the concerns raised by Reviewer cDXj, we have provided detailed responses. Do the responses and results address your concerns adequately? Please note that the discussion period ends on Dec. 2. Kindly let us know if you have any unresolved issues or additional concerns. Thank you.

We thank the reviewer for the thoughtful review and the constructive feedback on our paper. We appreciate your recognition of our work. Your questions are all valuable. Please find our responses to the points you raised.

1.Q: Is the data uniformly sampled from the video?

R: The images are uniformly sampled, and we will consider some suitable visualization methods. We will release both the data and code.

2.Q: There is a risk of overfitting to video scenes?

R: The test images are uniformly sampled as well. As the reviewer noted, our videos contain only dozens of scenes and many similar frames, so it remains challenging for our models to generalize to more semantic scenarios. However, this possible overfitting does not undermine our conclusion that more subjects and greater similarity among subjects improve generalization capabilities. Below is the summary of an overfitting experiment about this concern:

In this experiment, we train models on 1,000 public images (seen by all subjects) from the NSD test set and test them on those same images. The models are completely overfitted to those images. As shown in the table, the model trained on Subj1,2,5 achieves nearly 100% accuracy on the seen subject Subj5, but performs poorly on the unseen subject Subj7 (1% accuracy). We believe that generalization occurs in two dimensions: subject generalization and semantic space generalization. NSD has a broad semantic space that supports semantic generalization, while our dataset contains a sufficient number of subjects to facilitate subject generalization. Thus, even if the semantic space is overfitted, it will not affect our conclusion that generalization across subjects is possible.

3.Q: Different averaging windows or other pooling strategies.

R: We have experimented with several pooling strategies (e.g., pooling next 2-6 voxels). While these strategies affect the absolute performance metrics to a small extent, they do not impact the generalization conclusions. Therefore, we adopted the current approach.

4.Q: Model parameters.

R: Currently, the networks are designed to align with the parameters of MindEye1 [1], which has around 900M parameters. However, our backbone is replaceable. We have also trained a 100M parameter transformer in previous experiments, and the generalization conclusions remain unchanged.

5.Q: Upsampling methodology.

R: The upsampling method we use is a simple bicubic interpolation, which does not require training. Its purpose is solely for size alignment, and the generalization capabilities depends on the network itself.

6.Q: Do you use betas or BOLD signals for NSD dataset?

R: We use beta values for both NSDGeneral and NSD “whole brain” data. NSDGeneral data are directly related to brain regions involved in visual processing, which generally results in better visual decoding performance (see line 232 in the paper). We also explain why we use “whole brain” data: although the decoding performance is not as high, it does not require manual labeling. Therefore, it can be scaled up to produce larger datasets, making it more practical and suitable for studying generalization.

7.Q: Demo images selection.

R: There are no quantifiable criteria for the demo image selection. We select a few from different scenes, showing both good and poor retrieval cases.

We will provide a revised version and incorporate these points with more details as requested by the reviewer. We look forward to the reviewer’s continued positive rating of our paper.

References:

[1] Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. (NIPS)

We thank the reviewer for the thoughtful review and constructive feedback on our paper. We appreciate your recognition of our work, including the timely motivation and the practical method. Please find our responses to your points.

1.Q: How does this approach address the alignment problem?

R: The study is not designed to solve the alignment problem, and our approach does not involve alignment (except for data sizes alignment). The alignment problem typically refers to aligning a trained model to a specific subject (or vice versa). It requires identical input-output pairs across subjects, as the reviewer mentioned. However, our method focuses on testing subjects without any prior information. As for the source of generalization capabilities, we can describe it as "including" rather than "aligning". In other words, our method works well for unseen subjects because training samples may include the feature mapping patterns of unseen testing subjects. Experiments also support this explanation. When the number of subjects is the same, a more similar training group to a testing subject (such as sharing the same gender) will lead to better generalization. Moreover, when there are more training subjects, the generalization abilities becomes better, because more training subjects provide more possibilities to include the pattern of the unseen testing subjects. Our main contribution is to highlight this new approach: generalization abilities can be generated by “including” enough subjects without alignment (fine-tuning).

2.Q: Provide the comparison with the alignment methods.

R: As mentioned above, we do not directly compare to alignment or fine-tune methods because all such methods require information from aligned subjects (e.g., identical input-output pairs or training data from aligned subjects). This makes alignment inherently a "seen" problem. Our approach, however, is designed for zero-shot generalization, where no information is available about the unseen subjects. In this case, traditional alignment or fine-tune methods are not feasible. Therefore, we can not provide the performance comparison. However, we can provide a comparison in terms of model complexity:

Many methods, such as UMBRAE[1], use different tokenizers for different subjects. As shown in the table, in UMBRAE’s case, each subject requires a tokenizer with 11.2M parameters. As the number of subjects increases, the parameters grow linearly, leading to a substantial increase in model size. In contrast, the parameters of our method remain constant regardless of the number of testing subjects.

We hope these responses have addressed the reviewer’s concerns.

References:

[1] UMBRAE: Unified Multimodal Brain Decoding. (ECCV)

We thank the reviewer for the thoughtful review and constructive feedback on our paper. We appreciate your recognition of our work, including the positive comment on the motivation and the completeness of the experiment and dataset. Please find our responses to your points.

1.Q: The Method is nothing new. Using downsampling.

R: We focused on the model’s generalization ability, proposing a novel unified framework with new learning paradigms to address a new problem, laying a foundation for future brain decoding models. These innovations pertain to both the methodology and the framework which are all related to the machine learning. Moreover, we employ an upsampling method instead of downsampling.

2.Q: There is no quantitative comparison to relevant studies.

R: The reviewer mentioned that alignment algorithms can not be directly compared to ours. These alignment algorithms require information from the testing subjects for fine-tuning (aligning) — such as anatomical data or a small set of new data from the testing subjects. Thus, these alignment methods are more akin to few-shot approaches rather than zero-shot methods. Our approach focuses on handling unseen subjects without any prior information about them. Our method only needs size alignment through upsampling. In this case, traditional alignment methods cannot be used. “fsaverage” is a useful feature alignment method and may enhance our approach. However, our method does not rely on alignment, even simple upsampling could achieve the necessary size adjustment effectively.

3.Q: A neuroscience research task requires a dataset of its own design. This paper deserves to be published but is more suited to brain science related journals.

R: Thanks again for your recognition. We believe ICLR is a very suitable conference. First of all, the scope of ICLR includes “applications to neuroscience & cognitive science”. Second, our paper is framed from the perspective of AI research, presenting a unified framework and learning paradigms. As the reviewer pointed out, we are not addressing a specific neuroscience problem; instead, we focus on identifying commonalities in brain activity patterns. Our ultimate goal is to develop a universal brain decoding model that can be applied to all subjects. Although we are still far from this goal, we are exploring its feasibility. In summary, we believe this paper fits ICLR well.

4.Q: Downsampling fails miserably in the reconstruction task.

R: Yes, we acknowledge that downsampling can be problematic. However, we use upsampling which retains all information without data loss. Our approach differs from Clip-Mused [1], which uses downsampling to reduce dimensionality (for classification). Our pipeline maps images and fMRI voxels to the CLIP feature space for retrieval, which is more similar to the Mindeye1 [2] framework. Therefore, we expect that our method should also perform well in reconstruction tasks, and this is a part of our ongoing work.

5.Q: Does poor generalization in the NSD dataset come from having too few subjects or because subjects see different images?

R: It is an insightful question. This experiment could help for clarifying:

We train the models on the 1,000 public images from the NSD test set and test them on those training images. In this case, every subject has seen the same images. As shown in the table, the model trained on Subj1,2,5 achieves nearly 100% accuracy on seen Subj5 but performs poorly on unseen Subj7 (1% accuracy). This suggests that the primary factor influencing generalization capabilities is the number of subjects, rather than whether subjects see the same images. Even when all subjects see the same images and testing occurs on the training set, a lack of subject diversity results in poor generalization capabilities to unseen subjects.

We hope these responses address the reviewer’s concerns and that the reviewer is convinced to change the rating to a positive one.

References:

[1] CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding. (ICLR)

[2] Reconstructing the mind’s eye: fmri-to-image with contrastive learning and diffusion priors. (NIPS)

5. Innovations and contributions:

Finally, we would like to explain again the innovations and contributions of the article, which are not reflected in a certain module (such as upsampling). Our innovations lie in the whole pipeline we propose for studying generalization and advancing general brain decoding models. This includes dataset construction, learning paradigms, network architectures, and evaluation methods—all designed to expand applicability to more subjects and focus on generalization to new subjects.

Using this pipeline, we demonstrate that generalization emerges from a sufficient number of subjects. This provides a new pathway: generalization does not require alignment (only needs simple size-unification). This exploration moves us closer to a general brain decoding model that can generalize directly to any subject (Not just can do inference , but have generalization abilities to all subjects).

A contribution example: under our learning paradigm, exploring multi-subject generalization tasks with NSD is highly limited, as the dataset includes only 8 subjects. This severely limits generalization studies on new subjects. (Indeed, most of our early experiments with NSD yielded no generalization.) We do not advocate using NSD to study multi-subjects generalization ability, because even if there are some results, it will be difficult to verify on a larger number of subjects.

We hope this response adequately addresses your concerns. Please feel free to reach out with further questions or feedback. Thank you again for your valuable comments, we will provide a discussion paragraph in the paper to make it clearer.

Reference and Summary

[1] Aligning brain functions boosts the decoding of visual semantics in novel subjects

This paper proposes an alignment method. In the experiments, they explicitly mention that “with just 30 minutes of data, left-out subjects can reach performance which would have needed roughly 100 minutes of data in a within-subject setting.” This clearly indicates that they use new subject data for alignment, which is essential to their approach. Therefore, we state that a direct comparison with their alignment approach is not possible due to the fundamentally different setups. 

[2] Wills Aligner: A Robust Multi-Subject Brain Representation Learner

This paper introduces a method for learning effective shared representations across multiple subjects. Their experiments involve training on multiple subjects and testing within the training set subjects. When testing on unseen subjects, they rely on a few-shot mapping process, as noted in Tables 4 and 5, which explicitly list the proportion of few-shot data used. If there were a row in their results with a few-shot proportion of zero, it would correspond to our experimental setup. However, since their approach depends on using some data from new subjects, it fundamentally differs from ours, where no new subject data are used for alignment or mapping.

[3] Neuro-Vision to Language: Image Reconstruction and Interaction via Non-invasive Brain Recordings

This paper also does training on multiple subjects and testing within the training subjects. It does not address generalization to unseen subjects.

[4] fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding

We find the results of this paper puzzling. It utilizes fMRI reconstruction as a proxy task for self-supervised pretraining. According to Figure 2, the pretraining requires fine-tuning when applied to downstream tasks, similar to MAE. However, their description of the fine-tuning process is just a single sentence (see Page 8, Section 4.2.2). Furthermore, neither the model, code, nor supplementary materials are publicly available. During their pretraining on the UKB dataset, the model did not see any image data. After an unspecified fine-tuning process without any description, the model is claimed to reconstruct images on unseen NSD subjects. Is that reasonable? Without more details about their fine-tuning process, we can not include this paper's results as a valid reference.

After addressing this key misunderstanding, we now respond to the specific concerns (concerns 1-3 may stem from the above misunderstanding):

1. More about upsampling:

The purpose of upsampling is solely to enable the model to process data from unseen subjects, not to generate generalization abilities. As explained, this module is entirely replaceable and does not impact our conclusions about generalization. We do not consider this to be a novel module, nor do we believe it requires demonstrating its superiority (There's no superiority, just the simplest size alignment).

2. Method Comparison:

We now better understand the reviewer's concern about a lack of comparisons. When we stated "traditional alignment methods cannot be used," we referred specifically to functional alignment methods [1,2]. These methods cannot be generalized without alignment or fine-tuning. Of course, because size alignment is performed, they all can make inference on unseen subjects [1,2,3]. We are not claiming that their methods cannot perform inference on unseen subjects; rather, we are stating that without functional alignment, they lack generalization ability on unseen subjects (with only size-aligment and small number of training subjects, without functional alignment).

In fact, our experiments (e.g., the last three rows in Table 6) reproduce and compare such scenarios. In these experiments, “ours” is just MindEye1 with size alignment, and it shows no generalization ability on unseen subjects (without fine-tuning on test subject data). Thus, our results demonstrate that these methods [1][2][3], while capable of performing inference on unseen subjects, do not achieve generalization without fine-tuning on unseen subjects.

(Note: We carefully considered the references you provided, some of which we had previously reviewed. However, our discussion of “can be used” refers to different aspects (can perform inference VS have generalization ability). We summarize in the references section for clarity.)

3. Identifying commonalities in brain activity patterns:

It may be more precise to use the term "verify" here. The foundation of our work lies in commonalities in brain activity patterns. As mentioned earlier, generalization ability arises from these shared patterns, not from upsampling. These commonalities enable the training set to "include" unseen subjects, thereby producing generalization. While we have not localized these similarities to specific brain regions, we have at least verified their existence and demonstrated their role in driving generalization—an unexplored area in prior studies. For determining specific commonalities, as you mentioned, is extremely challenging and may reside in high-dimensional feature spaces rather than specific brain regions that are easily interpretable.

4. Poor Generalization:

It is expected that a model trained on 1,000 images cannot achieve good retrieval performance on the full NSD training set. This experiment was designed to show that even when the same images are seen, too few subjects fail to produce generalization. This supports our conclusion that generalization is primarily driven by the number of subjects, not shared images.

Further, we believe generalization occurs along two dimensions: subject generalization and semantic space generalization. NSD offers a broad semantic space that supports semantic generalization, while our dataset provides sufficient subject diversity to support subject generalization. In other words, we can also say that NSD-based methods all have poor subject generalization due to their limited participant pool. Therefore, even if the semantic space is overfitted, our conclusion about cross-subject generalization remains valid. That is we mentioned, if there is a large dataset including enough subjects and image contents, we believe that it is possible to implement a general visual brain decoding model.

Many thanks to the authors for their efforts on rebuttal, and for two rounds of detailed and nuanced response attitudes. This reinforced my understanding of this paper and corrected some misconceptions. I started this paper on a positive note, believing that only a few concerns needed to be clarified. It was not well resolved due to some misunderstandings that arose from the first round of discussions. Now I think the authors have done a good job of responding accordingly.

Although I still think that the addition of a certain number of comparison experiments could have been more enlightening for the researchers involved, as how to learn commonalities between subjects while avoiding fMRI noise is what the up-sampling module needs to focus on. However, this does not detract from the contribution of this paper, so I have increased my rating. Good luck to all of you!

Since Reviewer cDXj participated in the final discussion, I will update the summary accordingly. We addressed most of Reviewer cDXj’s concerns. The remaining concern revolves around: "I am still not fully convinced that the HCP dataset is entirely appropriate for this approach, nor do I believe that the fMRI data accurately and reliably represent the corresponding images."

We would like to emphasize that, this usage of the HCP dataset represents one of the best available choices for exploring generalization capabilities now, because of its sufficient number of participants. Additionally, "fMRI data cannot accurately and reliably represent the corresponding images (video frames)" is a challenge faced by the entire field, not just this work. Under such conditions, a significant number of studies on fMRI-image (or video frame) retrieval, classification, and reconstruction papers are accepted by various journals and conferences each year. Therefore, despite certain limitations, we believe this approach remains academically acceptable (The other three reviewers also accept this approach).

Finally, we sincerely thank all reviewers and area chairs for their time, effort, and valuable feedback.

Sincerely, Authors of #1426