Multi-modal brain encoding models for multi-modal stimuli

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. Challenges in comparing relatively small number of models. Impact of diverse model architectures on performance comparison

Thank you for this interesting question.

• We have provided a detailed discussion in our responses to CQ1 and CQ2. Kindly refer to those responses for more comprehensive information.

Q2. To my mind, there are two ways to mitigate this concern: 1) controlled model training and evaluation so that only one factor varies at a time, or 2) testing many different models of a given class such that even across significant model variations there is a robust effect of modality regardless of particular model factors.

Thank you for this valuable suggestion. We also acknowledge that testing a larger number of models within a given class can help determine if the effect of modality is robust across various model configurations.

• In our study, we follow the second alternative focusing on models that belong to the same class, particularly those based on the Vision Transformer (ViT) architecture and the Audio Spectrogram Transformer (AST).

Unimodal Video Models:

• VideoMAE, ViViT, and ViT-H are all built upon the ViT architecture. These models share the same foundational structure, with variations primarily in training strategies or specific implementation details.
  • VideoMAE: Employs masked autoencoders for video representation learning based on ViT.
  • ViViT: Extends ViT to video by processing spatio-temporal tokens.
  • ViT-H: A larger version of ViT with more parameters but the same architectural backbone.

Multimodal Models:

• ImageBind and TVLT are also based on the ViT architecture for the vision component and utilize AST for the audio component.

• ImageBind: Binds multiple modalities by projecting them into a shared embedding space using ViT and AST encoders.

• TVLT: Integrates visual and auditory information through ViT and AST, focusing on temporal alignment.

• Thus, we believe that our current approach addresses your concern by utilizing multiple models within the same class, thereby reducing the impact of confounding factors related to architecture and training differences. By focusing on models based on ViT and AST, we provide a more controlled comparison that highlights the effect of input modality on brain alignment.

Q3. I think that this is a serious concern because not all prior work has found that multi-modal models are more brain aligned.

Thank you for pointing this out and for giving us the opportunity to clarify our findings in the context of existing research.

Multimodal Models and Brain Alignment:

• Recent brain encoding research has shown that multimodal models align more closely with brain activity than unimodal models, even when subjects are exposed to single-modality stimuli.

• [Tang et al. 2023] investigated encoding models trained using representations from a multimodal transformer (BridgeTower) and a unimodal transformer (RoBERTa). They found that the multimodal transformer learned more aligned representations of concepts in both language and vision than unimodal ones.

• [Wang et al. 2023] reported that CLIP, a multimodal model trained on image-text pairs, explains greater unique variance in higher-level visual areas compared to models trained only with image/label pairs (e.g., ResNet) or text-only models.

• [Oota et al. 2022] examined four multimodal models—LXMERT, VisualBERT, CLIP, and ViLBERT—and found that these multimodal models showed improvements over the unimodal models in explaining brain activity.

• [Nakagi et al. 2024] examined multimodal vision-semantic LLMs (GIT, BridgeTower and LLaVA-v1.5] that predict brain activity and find that these models uniquely capture representations in the association cortex better than unimodal models [BERT, DEiT, ResNet] combined.

• Furthermore, reviewer gD8j suggested the following works: Subramaniam et al. (2024) and Dong et al. (2023). These studies also demonstrate that multimodal models outperform both unimodal models and language-vision models with linearly integrated features, such as the concatenation of vision and language features.

Our study extends this line of inquiry by examining brain encoding during multimodal stimuli (simultaneous audio and visual input), which more closely reflects real-world sensory experiences.

[Tang et al. 2023], Brain encoding models based on multimodal transformers can transfer across language and vision, NeurIPS-2023

[Nakagi et al. 2024], Unveiling Multi-level and Multi-modal Semantic Representations in the Human Brain using Large Language Models, EMNLP-2024

[Wang et al. 2023], Incorporating natural language into vision models improves prediction and understanding of higher visual cortex, Nature Machine Intelligence 2023

[Oota et al. 2022], Visio-Linguistic Brain Encoding, COLING-2022

Q2: Model variability

Overall, I find this explanation by the authors to somewhat mitigate my concern. However, I want to emphasize that belonging the same general class of architecture (e.g., ViT) is not the same as having the same architecture. The authors need to address this as a limitation in the main text of the paper.

Q4: Conwell et al. findings

I am not satisfied by the authors explanation of the Conwell findings. In controlled comparisons of models with and without language alignment, Wang et al. (2023) also did not find an increase in performance using an encoding approach as a result of language alignment throughout most of high-level visual cortex (Figure 5c). I will also emphasize here that my primary concern related to the Conwell and Wang findings is that when models are well controlled for architecture and training data, multimodality training may not have a meaningful effect. The authors can mitigate my concern here by just acknowledging that not all studies have found multimodal effects in the Related Work section and acknowledge that tightly controlled comparisons of models are needed in future work in the Limitations section (see Q2).

Q6: Multimodal effects

I appreciate the clarifications that the authors provided here. I would additionally like to see these clarifications reflected in the text.

We thank the reviewer for their valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. Challenges in comparing relatively small number of models. Impact of diverse model architectures on performance comparison

Thank you for this interesting question.

• We have provided a detailed discussion in our responses to CQ1 and CQ2. Kindly refer to those responses for more comprehensive information.

Q2. The fMRI dataset uses a small-n, data-rich design, which is good, but given this, it would help to see the results at the individual subjects’ level (in the appendix). On bar plots, it would be nice to plot each of the six subjects as a point overlaid on the average bar

We thank the reviewer for this suggestion.

• For brain encoding studies using naturalistic fMRI datasets, we follow the common practices in this area of research in that the number of samples per participant is more important than the number of subjects (six) because the predictive models are trained independently for each participant. Thus, having more samples per participant helps us learn a better predictive model. The Movie10 fMRI dataset is one of the largest datasets in terms of samples per participant (~12,950 samples), and that is the main reason for its frequent use. Our results also clearly show that this dataset is sufficient to learn a good predictive model, as we can predict up to 50% of the explainable variance for held-out brain recordings that were not used for training (e.g., Fig 2). Hence, our fMRI dataset is not small.

• Based on the reviewer’s suggestion, we have updated the bar plots in Appendix Figs. 7, 8, 9 and 10 to display each subject’s predictivity score as individual points instead of using error bars. This visualization provides a clearer representation of variability across subjects, allowing readers to assess individual differences more effectively. We believe these changes address the reviewer's concerns and enhance the clarity and interpretability of our results.

Q3. The residual analyses and results are somewhat confusing. Alternatively, the authors could do the residual analysis on the brain instead of the models. Relatedly they could calculate the product measure or unique variance explained by multimodal models above unimodal models from this joint model.

Thank you for this interesting question.

Residual Analysis on the Brain:

• The features can be removed from the model representations (as we do), or from the brain recordings (as suggested by the reviewer).
• Conceptually, the results of these approaches should be the same because when the feature is removed completely from either the input or/and the target, it would not be able to further impact the observed alignment.
• However, practically, brain recordings are noisier than model representations and so estimating the removal regression model will be more difficult especially with fMRI data of low SNR.
• Thus residual analysis on the brain is less effective. Therefore, we opt to remove features from the model representations where we can exercise more control, similar to [Toneva et al, 2022; Oota et al., 2023; Oota et al., 2024]. We will clarify this reasoning in the main paper.

Unique Variance Calculation:

• Based on reviewer’s suggestion, we now build two voxelwise joint encoding models:
  • (i) that includes both unimodal and cross-modal features,
  • (ii) that includes both unimodal and jointly-pretrained features.
  • Using these joint encoding models and prior encoding models, we compute the unique variance explained by each model i.e. unimodal and cross-modal, unimodal and jointly-pretrained models.
  • Also, we compute the shared variance between these models.
  • The results, presented in Appendix N, Figures 12, 13 and 14, for the whole brain and the language network, reveal that the shared variance between the jointly pretrained (TVLT) model and the unimodal models is significantly lower than that observed between the cross-modal (IBConcat) model and the unimodal models.
  • This finding aligns with earlier residual analysis results, where IBConcat showed a larger performance drop when unimodal information was removed compared to the TVLT model (see Figure 4). Specifically, we observe a more pronounced performance drop when video features are removed compared to when speech features are removed.
  • For the visual network, as shown in Figure 14, we observe that the unique explained variance between TVLT and unimodal VMs is comparable, while the cross-modal model IB Concat exhibits a higher unique explained variance compared to unimodal VMs. In contrast to unimodal VMs, both cross-modal and jointly pre-trained models demonstrate higher unique explained variance compared to unimodal SMs.

[Toneva et al. 2022], Combining computational controls with natural text reveals aspects of meaning composition, Nature Computational Science 2022

We thank the reviewer for their strong positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. Relationship with previous works that uses multimodal models to probe for multimodal processing in the brain should be discussed.

Thank you for this question. We would like to clarify that our study is different from previous studies in the following aspects:

• Subramaniam et al. (2024) utilized vision-language models based on image frame-text pairs, despite the stimuli being continuous movies. While effective for certain tasks, this approach may overlook the temporal dynamics inherent in videos. Additionally, their use of stereoencephalography (SEEG), although offers high temporal resolution, is limited in spatial resolution and typically restricted to specific brain regions where the electrodes are implanted. Finally, their study focused on multimodal integration in a cross-modal model setting and did not explore jointly pretrained settings. Additionally, each participant watched a different movie while their SEEG activity was recorded, therefore the input stimuli varied widely across participants.
  • In contrast, our study leverages video-audio models that capture the temporal events in videos, providing richer and more dynamic representations of the stimuli. By incorporating audio data, we preserve acoustic information that may be lost in text-based transcriptions. Moreover, we utilize fMRI data, which offers whole-brain coverage and higher spatial resolution, enabling a more comprehensive analysis of the brain activity.
  • Our approach also considers both cross-modal and jointly pretrained multimodal models, offering a more nuanced understanding of how different modalities interact and integrate information in the brain.
• Dong et al. (2023) utilized the Friends web series fMRI dataset to investigate the effectiveness of pretrained versus fine-tuned multimodal video transformers using video+text stimuli-based brain activity. Their study specifically examined the impact of fine-tuning a multimodal model on brain alignment, comparing performance before and after fine-tuning. However, they did not explore cross-modal vs. jointly pretrained model analysis or the comparison between multimodal and unimodal models, leaving it unclear which type of multimodal models perform best for predicting brain activity. In contrast, our study focuses on video+audio stimuli and includes a comprehensive residual analysis, providing deeper insights into the contributions of different modalities.
• Kewenig et al. (2024) conducted a study involving 200 human participants who provided ratings while watching short audio-visual clips (6 seconds each) to estimate the predictability of upcoming words. This study specifically focused on behavioral evidence, demonstrating that multimodal attention models can leverage contextual information to predict upcoming words in a manner more aligned with human behavior. Notably, this study did not involve brain recordings; instead, the authors collected behavioral ratings and focused on human attention as indexed through eye-tracking data.
  • These differences suggest that our study is better placed to provide deeper insights into how the brain processes and integrates multimodal information, leveraging the strengths of video-audio models leveraging the comprehensive spatial coverage of fMRI.

We have included this extended related work in Appendix J of the revised paper.

Q2. Baseline Analysis: Scrambling Inputs to Multimodal Models

Thank you for suggesting this new experiment.

• Based on the reviewer’s suggestion, we conducted an additional baseline experiment where we kept the trained weights unchanged and shuffled the movie stimuli into a scrambled order as input to the two multimodal models: cross-modal and jointly-pretrained models. The results of these experiments have been added to Appendix K, Fig. 12 of the revised paper.
• The results demonstrate that embeddings from multimodal models exhibit significantly better brain alignment compared to both randomly initialized models and when passing scrambled clips as input. Furthermore, when comparing scrambled input to pretrained models with randomly initialized models, the scrambled input shows improved alignment over random initialization.
• Overall, this sanity check confirms that representations from multimodal models maintain meaningful alignment with brain activity, even when the stimulus order is scrambled, highlighting their robustness and effectiveness.

Q3. Line 276 typo: wmploy -> employ

Thank you for pointing this out. We have corrected the identified typos in the revised draft.

I thank the authors for their thorough response. I am more convinced that the work covers novel ground, based on their explanations of prior work. I have upgraded my score accordingly.