Aligning brain functions boosts the decoding of videos in novel subjects

We thank the reviewer for their time and feedback.

Weaknesses

1. We agree with the reviewer that we should clarify the novel contributions of our work. We propose to amend the introduction with the following paragraph:
   “Overall, decoding and functional alignment studies have focused on largely disjoint objectives: decoding semantics is typically done at the individual level (e.g. Tang et al. 2023, Scotti et al. 2023, Ozcelik et al. 2023), whereas functional alignment across subjects is evaluated on simple features, such as retinotopy (Feilong et al. 2022), object categorization (Bazeille et al. 2021), and binary contrasts (Thual et al. 2022). With the rise of naturalistic datasets (NSD, Wen2017, Lebel2023, Things), it is pressing to know the conditions necessary for future studies to directly benefit from the high-level representations identified in a small set of high-quality recordings. Here we systematically investigate the data regime in which a set of standard semantic decoders would benefit from functional alignment.”
2. Reviewer tcU3 is right that we only benchmark the functional alignment method (FUGW) against the most commonly used anatomical alignment method, which is our baseline. FUGW was introduced and benchmarked in a different paper. The goal of the present study is to evaluate the condition in which functional alignment could improve standard semantic decoders. Moreover, the baseline used in our paper is the alignment method most commonly used in neuroscience.
3. We agree with tcU3 that the present method is limited to a single reference subject. To mitigate this issue, we propose to generalise our result section, by reproducing the analysis with each possible reference subject. We kindly invite you to read the description of two additional experiments in the response we make to reviewers uqYK and HiZ5. Overall, however, building a multi-subject reference is still an open research question. Thual et al. 2022 introduces a method to build functional barycenters of individuals, which we did not try here because we feel that this way of computing reference subjects is fairly new and would require extensive testing of its own.

We thank tcU3 for their helpful remarks, and hope these comments help alleviate their concerns. We would be happy to provide additional information and experiments which could help strengthen our conclusions.

We thank the reviewer for their time and feedback.

We want to stress that we respectfully disagree with the reviewer’s summary of our work. Our manuscript does not propose an alignment method. It instead shows that at least one functional alignment method (introduced in a different paper) is suited to transfer semantic decoders trained in some individuals to other, left-out individuals. It is possible that other functional alignment methods could achieve better performance in this regard, although we believe it is out of the scope of this study to compare them. Also, we stress that anatomical alignment (i.e. projecting all data to a common anatomical template like MNI or fsaverage) is the go-to strategy employed in a vast majority of studies. Therefore, since this anatomical alignment is our baseline, our work already includes an insightful comparison to the most commonly used alignment method in the literature. Finally, as stated in Bazeille et al. 2021, it is not clear that any existing functional alignment method outperforms others in the general case (in particular, performance highly depends on the downstream task and input data).

Weaknesses

1. Reviewer jWwB is right that the chosen alignment method (FUGW) has 3 hyper-parameters, namely alpha, rho and epsilon. We use FUGW because, unlike alternatives, it comes with a simple python API comparable to that of sklearn and thus facilitates reproducibility. Previous work has already explored the impact of these three hyper-parameters, and showed that alignment is robust to changing values for alpha and rho, although epsilon needed to be carefully chosen. Our goal is a proof-of-existence, i.e. that it is possible to yield significant decoding gains with a standard functional alignment preprocessing.
2. Since we did not introduce this alignment method, we did not feel compelled to compare it with a lot of other methods. However, note that our baseline (i.e. fsaverage) is the most commonly used alignment method. Moreover, we agree with you that our work could constitute an interesting downstream task, which a future benchmark could use to compare between functional alignment methods.

Questions

1. Indeed, fMRI datasets featuring a lot of data per participant are still very rare. To our knowledge, the Wen 2017 featured in our study is among the most extensive datasets in this regard. In order to strengthen our work, we have decided to implement two other experiments, which reproduce our analysis with each possible reference subject. We kindly invite you to read their description in the response we make to reviewers uqYK and HiZ5.
2. We hope our second point of the previous section addresses your concern.

We thank jWwB for their helpful remarks, and hope these comments help alleviate their concerns. We would be happy to provide additional information and experiments which could help strengthen our conclusions.

We thank the reviewer for their time and feedback.

Weaknesses

In our setups, the test set is completely separated from the training sets of both the decoder and alignment: the training and test sets contain different movies acquired during different MRI sessions.

However, reviewer HiZ5 is right that our alignment (although not the decoder) requires the participants to watch the same stimuli. This limitation applies to the vast majority of alignment methods (Bazeille et al. 2021).

To clarify this issue we propose to indicate the following in the method section: “For example, in the out-of-subject setup, a decoder is trained on movies 1, 2, 3 seen by subject A. Subjects A and B have both seen movies 4 and 5, which are used to compute an alignment. We refer to these movies as “anchors”, because they allow left-out subjects to “anchor” themselves onto a decoder that has been trained on another individual. Finally, the decoder is tested on data from subject B, acquired when they were watching movie 6. All aforementioned movies were acquired in different MRI sessions.”

In our paper, movies used to train the decoder and alignment are the same, which does not need to be true in the general case. However, there is no reason not to include “anchor” movies in the training dataset of the decoder, as it makes for more accurate decoders and more powerful “anchors” to the decoder.

Questions

1. In this case, we used only the first trial (i.e. run). We could try to use only the second run, but we think it is more crucial to measure results based on data which represents the neural response of individuals when they are first shown the stimuli. Indeed, stimuli repetition can lead to unexpected effects (drop of attention, less controlled experiment in general).
2. We hope that our response to the main weakness answers your question. Please let us know if further clarification would be helpful.
3. We further expand on our claim that alignments map vertices in accordance with brain anatomy: in Figure 4, the first column from the left shows an atlas displayed on subject 1, without functional alignment (in this case, using only anatomical alignment). Columns 2, 3, and 4 show how vertices of subject 1 are reorganised on the cortical sheet to functionally match that of subject 2, using an increasing amount of movie-watching data to fit the functional alignment. What we observe is that, with more data, computed reorganisations look more anatomically coherent. For instance, the parietal and temporal lobe look very scrambled in column 2 (i.e. some of their vertices get matched with vertices coming from very different cortical areas), but much more consistent in column 4 (based on colour, matched vertices are much more coherent). One can observe a similar effect even in premotor areas, and the occipital lobe.
   We are working on a refined version of this Figure to make these points clearer.
4. We hope that our response to the main weakness answers your question. Please let us know if further clarification would be helpful.

Again, we are thankful to HiZ5 for these valuable comments.

We thank the reviewer for their time and valuable feedback.

Weaknesses

We agree that it is unclear whether our multi-subject alignment will scale to larger numbers of individuals. Our decision to focus on a dataset with few individuals with a lot of natural stimuli stems from a simple question: under what data regime can functional alignment be beneficial to semantic decoding. In this regard, our contribution is a proof-of-concept: if decoders trained on a single participant can generalize to other unseen individuals for whom little data has been acquired, then it constitutes (1) a strong incentive to acquire very high amounts of data in a limited number of participants and (2) an additional legitimation of recent initiatives (e.g. Natural Scene Dataset, Kay, Naselaris et al. 2023; deep-fMRI-dataset, Lebel, Huth et al. 2023).

To further address this issue, we propose to add two more experiments:

• Experiment 1: evaluate decoding gains on each possible reference subject of the Wen2017 dataset. This will allow us to triple the number of experiments, hopefully strengthening our original conclusions.
• Experiment 2: evaluate decoding gains for out-of-subject setups in which the decoder has been trained on multiple aligned subjects.

We will test all possible combinations to measure the effect of multi-subject training on this cohort. We will post results for both these experiments as soon as we get them.

Questions

1. We chose to decode CLIP Vision 257 x 768 (high-level, semantic features) and VD-VAE (low-level features) latents to compare our results to that of Ozcelik et al. 2023 (in which the authors trained single-subject linear decoders on fMRI data from the Natural Scenes Dataset with much higher signal-to-noise ratio). We also included CLIP Vision CLS and AutoKL latents because they are comparable to CLIP Vision 257 x 768 and VD-VAE latents respectively, while being much smaller. We propose to clarify these choices in the “Video output” section.
   Supplementary figures S5-S7 illustrate the differences in decoding performance between these latents. In particular, we see that CLIP-based features (i.e. high level) are much more easily decoded than other, low-level features. Do you wish for us to make these results more clearly accessible in the main paper?
   The decision to use linear decoder follows a classic approach in fMRI (e.g. Naseralis et al 2011, Huth et al Nature 2016, Ozcelik et al. 2023) which present multiple advantages: (1) simplicity, (2) interpretability and (3) robustness. Furthermore, our internal testing did not show significant improvement of deep-learning pipelines over linear approaches. We propose to clarify this motivation in the method section.
2. We chose to report retrieval metrics to be comparable with other recent papers (e.g. Scotti et al. 2023) and because image reconstruction metrics (like used in Ozcelik et al. 2023) depend on the generative models used, but are also difficult to assess. Also, top-k accuracy is used in similar work (Scotti et al. 2023; Defossez et al. 2023; Chen et al 2022). Finally, we think that relative median ranks are easier to understand intuitively and to compare between settings.
3. We agree that this part of the paper is confusing. We propose to rearrange the method sections to avoid this back-and-forth. We agree and will make a higher-resolution version of Figure 2 in the appendix and add more comprehensive captions and paragraphs to introduce supplementary results.

Again, we are thankful to uqYK for these valuable comments.