Multi-modal brain encoding models for multi-modal stimuli

We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

1. How do the authors ensure the robustness of Pearson correlation metric in the context of brain alignment evaluation?

Thank you for this question.

• Our methodology for building encoding models ensures the robustness of the Pearson Correlation metric in the context of brain alignment evaluation by addressing potential outliers and non-linearities through the following steps:
  • (i) We employ z-score thresholding separately for both Input stimulus representations and brain recordings for training and test datasets. This helps identify and remove extreme outliers that could disproportionately affect the Pearson Correlation results.
  • (ii) To quantify the model predictions using Pearson correlations, we estimate cross-subject prediction accuracy as studied in previous research. Using this estimated cross-subject prediction accuracy, we measure the normalized brain alignment as Model Predictions / Cross-Subject prediction accuracy [Schrimpf et al. 2021;Oota et al. 2024;Alkhamissi et al. 2024]. Using this estimate, we see that these numbers correspond to about 50-70% of the explainable variance by a model representation. Note that we are only averaging across voxels which have a statistically significant brain alignment. We perform the Wilcoxon signed-rank test to test whether the differences between multimodal and unimodal models are statistically significant. This is explained in lines 249-253 of the paper.
• As can be seen, Pearson correlation is the most widely used metric to measure brain encoding accuracy. So, we follow the existing literature for this metric choice.

[1] Schrimpf et al. 2021, The neural architecture of language: Integrative modeling converges on predictive processing. PNAS, 2021 [2] Oota et al. 2024, Speech language models lack important brain relevant semantics, ACL 2024 [3] Alkhamissi et al. 2024, Brain-like language processing via a shallow untrained multihead attention network, Arxiv 2024.

2. Permutation Test Configuration.

Thank you for this question.

• Inspired by prior studies [2] [4] [5] [6], we employ the standard implementation of a block permutation test for fMRI data, which involves permuting blocks of 10 contiguous TRs while leaving the order within each block untouched. The choice of these specific configurations is based on established methodologies in previous research.
• Block Permutation Approach:
  • Rationale: By shuffling blocks rather than individual TRs, we preserve the temporal structure of the fMRI data (to account for the slowness of the underlying hemodynamic response), which is crucial for maintaining the integrity of the hemodynamic response.
  • Configuration: We use blocks of 10 TRs, ensuring that the order within each block remains unchanged. This method balances the need to preserve temporal correlations with the need to generate a sufficient number of permutations for robust statistical analysis.
• Thus, our approach involves shuffling the blocks without averaging over blocks of 10TRs. Specifically, the predictions are permuted 5000 times, and the resulting normalized predictivity scores are used as an empirical distribution of chance performance, from which the p-value of the unpermuted performance is estimated.
• We will clarify this in the text.

[2] Oota et al. 2024, Speech language models lack important brain relevant semantics, ACL 2024

[4] Deniz et al. 2019, The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 2019

[5] Reddy et al. 2021, Can fMRI reveal the representation of syntactic structure in the brain? NeurIPS 2021

3. Wilcoxon Signed-Rank Test Application.

• We performed a Wilcoxon signed-rank hypothesis test for all pairs of models and applied the Benjamini-Hochberg False Discovery Rate (FDR) correction for multiple comparisons.
• The FDR correction is applied by grouping together all voxel-level p-values across all subjects and choosing one threshold for all the results.
• This approach helps control the expected proportion of false discoveries among the rejected hypotheses, ensuring the robustness and reliability of our statistical findings. We will update this text and correct this oversight by including it in the final version.

4. Fig 1 caption should be more demonstrative and detailed.

Thank you for this suggestion.

• Detailed caption: we show how residual analysis can be applied to remove unimodal video model features from cross-modal representations for an input X. In the first step, a ridge regression model (r) is trained to map video model (VM) unimodal representations to cross-modal (CM) representations. In step 2, we learn another ridge regression model (g’) to map the residual representation |CM(X)-r(VM(X))| to brain activations.

We will provide more detailed caption in the final version.

5. Fig5 caption: It is proportion, not percentage.

• We will correct this typo.

We thank the reviewer for their positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

1. Clarification on feature removal

• For the removal of information from the model representations, we use the previously published approach i.e. direct approach or residual approach.
• This residual approach directly estimates the impact of a specific modality feature on the alignment between the model and the brain recordings by observing the difference in alignment before and after the specific modality feature is computationally removed from the model representations. This is why we refer to this approach as direct.
• Additionally, our work is most closely related to that of Toneva et al. 2022 [1], who employ a similar residual approach to study the supra-word meaning of language by removing the contribution of individual words to brain alignment. Further, this residual approach has been peer reviewed in good venues (Nature Computational Science, NeurIPS, and ACL).
• To remove features of a particular modality from multimodal model representations, we rely on the direct method as discussed above. In our setting, similar to prior studies, we remove the linear contribution of a unimodal feature by training a ridge regression, in which the unimodal feature vector is considered as input and the multimodal representations are the target.
• We compute the residuals by subtracting the predicted feature representations from the actual features resulting in the (linear) removal of unimodal feature vectors from pretrained multimodal features. Because the brain prediction method is also a linear function, this linear removal limits the contribution of unimodal features to the eventual brain alignment.
• Specifically, in Fig. 1B, we show how residual analysis can be applied to remove unimodal video model features from cross-modal representations for an input X.
• This is done in 2 steps.
  • In the first step, a ridge regression model (r) is trained to map video model (VM) unimodal representations to cross-modal (CM) representations. In some ways, r(VM(X)) captures that part of the information in CM(X) that can be explained or predicted using VM. Now the job of residual analysis is to check that if we remove this explainable part r(VM(X)) from CM(X) how well can it predict brain activity.
  • Hence, in step 2, we learn another ridge regression model (g’) to map the residual representation |CM(X)-r(VM(X))| to brain activations. Similarly, residual analysis can also be used to remove unimodal speech features from cross-modal representations for an input X.

[1] Combining computational controls with natural text reveals aspects of meaning composition, Nature Computational Science 2022

[2] Joint processing of linguistic properties in brains and language models, NeurIPS 2023

[3] Speech language models lack important brain relevant semantics, ACL 2024

2. How well the feature removal works?

Thank you for raising this question.

• We provided a more clear description of the feature removal method using residual approach in previous question Q1.
• We investigate this feature removal method by analyzing how the alignment between brain recordings and multimodal model representations is affected by the elimination of information related to these unimodal features. We refer to this approach as direct, because it estimates the direct effect of a specific feature on brain alignment. Fig 4 (in the main paper) shows that removal of unimodal video features from cross-modal embeddings results in significant drop in brain alignment for the language region AG.
• To perform probing analysis, unfortunately, there are no available probing tasks for the Movie10 dataset. This is an interesting future direction to annotate tasks for Movie10 dataset and verify the model interpretation via probing before and after removal of unimodal features and also perform several perturbations by doing mechanistic interpretability of models.

3. Projection method is unreliable without additional metrics like MSE scores or Pearson correlation?

Thank you for this valuable suggestion.

• To check the quality of information removal using our residual analysis method, we computed the Pearson correlation scores where unimodal video features are projected onto the multimodal IB Concat feature space using our residual approach.
• We observe a small Pearson correlation score of 0.555. This low value implies that unimodal video features are successfully removed from multimodal representations.
• Further, to our knowledge, there is no better alternative to selectively remove information from multimodal models to probe their impact on brain alignment.

4. Opposite direction of removal: project the video features out of the multimodal representation.

Thank you for this question.

• To clarify, in the paper, we have done exactly as you mentioned above. As shown in Fig. 1B (main paper), the residual is calculated as |CM(X)-r(VM(X))| which clearly shows that we have removed video features from the cross-modal/ jointly pretrained multimodal representation.
• Are you expecting us to perform the experiment in the other direction, i.e., remove multimodal features from unimodal representations? Although that does not sound very reasonable, we are happy to do this if expected.

5. Do TVLT model has better visual processing than ViT-B, considering feature removal?

• Kindly check the rebuttal PDF (Fig 2) & CQ2 response at “Common responses”.

6. Baseline performance with randomly initialized models.

• Based on the reviewers’ suggestion, we now perform experiments with randomly initialized models.
• Kindly check the rebuttal PDF (Fig 3) & CQ3 response at “Common responses”.

7. Rewriting of sentence and Fig 5 should be made bigger.

Thank you for your suggestion. We will update the framing of sentence and important editorial suggestions in the final draft.

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

1. Affect of performance by the choice of pre-trained models.

Thank you for this question.

• Impact of Pre-Trained Models:
  • To understand the relationship between brain activity and various stimuli, a large body of brain encoding literature over the past decade has utilized a variety of pre-trained language models.
  • These works have demonstrated that pre-trained models such as BERT, GPT-2, T5, BART, LLaMA, and OPT can predict both text- and speech-evoked brain activity to an impressive degree. Similarly, pre-trained speech models like Wav2vec2.0, HuBERT, AST, and WavLM have shown that speech-based language models better predict auditory cortex activity during speech-evoked brain responses.
  • Hence, brain encoding studies focus more on interpreting the representations of these models and obtaining insights into brain function rather than the choice of specific pre-trained models.
  • Prior studies have shown that irrespective of whether the models are encoders, decoders, or encoder-decoder based, they result in similar brain-language alignment.
• Our Study:
  • In our study, we tested three unimodal vision models and two unimodal speech models and observed similar alignment in brain activity predictions. This consistent performance across different models reinforces the robustness and generalizability of our approach.

2. What does observation in lines 311-312 & 313-314 indicate?

• In Fig 2, we do not observe any significant difference in brain alignment between cross-modal and jointly pretrained models at the whole-brain level or when averaged across language and visual regions.
• However, in individual language regions, we find that multimodal representations obtained from cross-modal models better predict brain activity in semantic regions such as the Angular Gyrus (AG), Posterior Cingulate Cortex (PCC), and Middle Temporal Gyrus (MTG).
• This indicates that while both cross-modal and jointly pretrained models perform similarly at a macro level, there are individual differences at micro level. This observation motivated us to do further detailed analysis in Section 6.2 and Section 6.3

3. Why is AC an exception for observation in lines 316-317?

• Since AC is an early auditory cortex which processes sound related information unlike higher cognition regions (i.e. language regions), we observe that audio embeddings result in higher degree of brain predictivity than video embeddings. We will include this insight in the final draft.

4. Why the choice of ridge regression instead of more complex machine learning models?

Thank you for your question.

• Since fMRI brain recordings have a low signal-to-noise ratio, and pretrained language models are trained in a non-linear fashion, the model representations are rich and complex.
• To understand the relationship between brain activity and various stimuli, a large body of brain encoding literature over the past two decades (some papers mentioned below [1-12]) has preferred ridge regression due to its simplicity.
• Ridge regression is a linear model, making it easier to interpret and understand compared to more complex models. Further, the regularization in ridge regression helps manage the noise effectively, leading to more robust and reliable models.

5. The results consistent with some existing neuroscientific findings.

Thank you for raising this point. Many of our findings are in line with previously established neuroscience theories. We describe some of them below.

• Speech embeddings show better alignment across all language ROIs, but most importantly in the AC region which is related to processing of sound information. This result aligns with previous work which has found that the speech-based language models better predict activations in the early auditory cortex [9] [10] [11].
• Multimodal embeddings display higher brain alignment in high-level visual processing regions than unimodal models. This observation is consistent with earlier studies which have indicated that the multimodal embeddings from the CLIP model display higher brain alignment than CNN models in high-level visual regions [12].

[1] Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS One 2014

[2] Natural speech reveals the semantic maps that tile human cerebral cortex. Nature 2016

[3] Incorporating context into language encoding models for fmri. NIPS 2018

[4] Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). NeurIPS 2019

[5] The neural architecture of language: Integrative modeling converges on predictive processing. PNAS 2021

[6] Brains and algorithms partially converge in natural language processing. Communication Biology 2022

[7] scaling laws for encoding models fMRI. NeurIPS 2023

[8] Self-supervised models of audio effectively explain human cortical responses to speech. ICML 2022

[9] Toward a realistic model of speech processing in the brain with self-supervised learning. NeurIPS 2022

[10] Joint processing of linguistic properties in brains and language models. NeurIPS 2023

[11] Speech language models lack important brain relevant semantics, ACL 2024

[12] Incorporating natural language into vision models improves prediction and understanding of higher visual cortex, Nature Machine Intelligence 2023.

6. Why do IB-concat and TVLT act differently?.

Kindly check response CQ4 at “Common responses”.

7. Questions

• r refers to the ridge regression model.
• Given that the underlying encoder models are the same and only the projection results in 1024 dimensions for the ImageBind, the difference in feature dimensionality should not significantly affect the fairness of comparisons.

8. Minor

• We will address these minors.

Thank you for your question regarding CQ4 and the distinct behaviors observed between TVLT and IB-concat in lines 350-353. We appreciate your attention to this important aspect of our work.

IB-Concat

• For cross-modality models, the alignment in regions AG and MT is extremely high, and this alignment is only partially explained by video features. This implies that significant unexplained alignment remains after the removal of video features. Conversely, the removal of speech features does not lead to a drop in brain alignment, indicating that there is additional information beyond speech features that is processed in these regions.
• This means that in cross-modality models, when transferring knowledge from one modality to another, the model relies more heavily on visual information. As a result, the model becomes more focused on video inputs rather than audio inputs. This likely reflects the model’s preference for using the detailed visual features that align closely with brain activity in regions AG and MT, leading to the observed high alignment.

TVLT

• For jointly pretrained multimodal models, the alignment in regions AG and MT is extremely high, and this alignment is partially explained by both video and audio features. Unlike cross-modal representations, the TVLT model learns a more balanced representation of both video and audio features. This leads to integrated information from both modalities, making the model less sensitive to the loss of features from a specific modality.
• As a result, we observe only a small drop in brain alignment when either modality is removed. This suggests that the model is capturing more high-level abstract and semantic information that goes beyond the specific features of just one modality.

Additional discussion:

• The model training protocol of TVLT appears more in line with how humans learn during development when they experience multiple modalities simultaneously and the learning is mediated by the experience of joint inter-modal associations.
• It is unlikely that the human system experiences these modalities in isolation, except in cases of congenital conditions where the inputs from a specific modality are not accessible.
• Given that the brain alignment observed in TVLT model in a language regions like AG is less sensitive to loss of information from specific modalities, we believe that AG serves as a multi-modal convergent buffer integrating spatio-temporal information from multiple sensory modalities to process narratives [Humphries & Tibon, 2023].
• The results of high alignment found in AG even in IB-Concat but more brittle with respect to loss of information from a specific modality are also interesting. It would be interesting to study patterns of activation in AG in patients who acquired visual or auditory function later in their life [Hölig et al., 2023] to see if one observes such brittleness in the representations acquired.

[Humphries & Tibon, 2023] Dual-axes of functional organization across lateral parietal cortex: the angular gyrus forms part of a multi-modal buffering system. Brain Structure Function 228, 341–352 (2023).

[Hölig et al., 2023] Sight restoration in congenitally blind humans does not restore visual brain structure. Cerebral Cortex. 2023;33(5):2152-2161.

Should you have any further questions or suggestions, we are ready to provide additional information or clarification as needed. We kindly request you to verify our response and consider updating your evaluation based on the revisions made.

Thanks for your help

We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

1. There exist `clock’ relationships in train-test settings lead to information leakage during inference.

• We made sure to follow proper practice for our training and testing settings, taking care that data leakage doesn’t happen.
• Please note that for the main results reported in the paper (as mentioned on lines 157-160) data from two movies “The Bourne supremacy” and “The wolf of wall street” movies was used for training, while testing was done on a third movie: “Life”.
• Thus, the train and test sets are totally disjoint and the model can’t use any clock relationships from the training data during inference. To be completely clear: independent encoding models are trained for each subject using data concatenated from two movies (The Bourne supremacy: 4024 TRs and The wolf of wall street: 6898 TRs). The test set consisted only data from the "Life" movie (2028 TRs). Thus there is no possibility of any information leakage during inference on the test set.
• The training data followed contiguous TRs, in line with prior studies where multiple stories are combined in a contiguous fashion for training, and a separate story is used for testing. This method of combining training data follows established protocols in the field. Since an entirely different movie is used for testing, our results are robust and free from temporal information leakage.

2. The data collection process should be blocked to avoid inter-data correlation. The three settings do not really account for the speciality of brain signals.

• Blocking/ Inter-data correlation: If we understand this issue correctly, we have addressed this in our answer to the previous question. If not, perhaps the reviewer can explain what he means by this question.
• To be clear, the training and testing sets are completely disjoint in the voxelwise encoding model. The data collection process is explained in detail in Section 3. This process is blocked to avoid all kinds of inter-data correlation for both unimodal as well as joint-model training. The original Courtois NeuroMod dataset creators have already ensured that data from different subjects and modalities are collected independently.
• In all three settings, testing is always on the third movie, “The Life”. This allows us to evaluate (1) the generalization of models across different training datasets, and (2) test the impact of increasing the number of TRs in the training dataset on brain prediction results. Therefore, the three settings account for generalization of models rather speciality of brain signals.

3. Cross-modal setting using ImageBind: with and without alignment of video and audio

Thanks for asking this interesting question.

• During inference, ImageBind does not require all modalities to be present simultaneously. We can provide data from just one modality (e.g., audio) and the model will still function correctly by finding related embeddings in other modalities (e.g., images, text) within the same embedding space.
• This flexibility allows for a variety of applications, such as cross-modal retrieval and classification, without needing to input all modalities at once.
• However, it is not true that this setting does not differ from the uni-modality setting, because as mentioned previously, even if we pass a single modality to ImageBind, it retrieves the relevant embeddings from other modalities from its aligned embedding space, so we don’t need to explicitly align the video and audio.
• If we provide two modalities to the ImageBind, then the retrieval from ImageBind would be conditioned on the explicit signal for the other modality that we are providing, so we are limiting the exploration scope of ImageBind to search for aligned embeddings of other modalities.
• To investigate the cross-modal setting with and without alignment of video and audio, we created a plot (Fig 1 in the rebuttal PDF) comparing the normalized brain alignment for three regions AG (angular gyrus), SV (scene visual) and MT (Middle Temporal). This plot shows that with alignment improves brain predictivity for the video modality across all three regions, while the audio modality shows improved alignment in the MT region, with the other regions maintaining similar performance.
• Kindly also check the rebuttal PDF at “Common responses”.

4. The conclusion “Both cross-modal and jointly pretrained models demonstrate significantly improved brain alignment with language regions” is somehow questionable. More detailed analysis is needed.

Thank you for raising this concern. We agree with the reviewer and in fact we never claimed that language is a sensory modality.

• Our intuition in the abstract and introduction pertains to the hierarchical nature of processing of information via early sensory regions (early visual and early auditory), and then on to higher cognitive processing regions, including language areas.
• While the extant literature focused on brain alignment with unimodal data (including studies with incongruent unimodal inputs), the current study investigates brain alignment when information from multiple modalities is utilized – trained either in a cross-modal fashion or in a joint-modality setting.
• Further in order to compare with the existing results, we undertook experiments with additional unimodal settings.
• Consequently, all our brain region results include visual, auditory, as well as language regions.
• Our results seem to point out that multiple-modal models seem to capture additional variance than unimodal models (see Fig. 3 in main paper) both in the sensory regions as well as in the language regions.
• In this context, we feel that our conclusion, "Both cross-modal and jointly pretrained models demonstrate significantly improved brain alignment with language regions," is reasonable and appropriate.

Dear Reviewer 28FZ,

We appreciate your feedback and are confident that it has enhanced the paper's quality.

Should you have any further questions or suggestions, we are ready to provide additional information or clarification as needed. We kindly request you to consider updating your evaluation (score) based on the revisions made.

Regards,

Authors

The rebuttal addressed most of my concerns. I keep my original rating.