Instruction-Tuned Video-Audio Models Elucidate Functional Specialization in Brain

We thank the reviewer for their positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. selected instruction-tuned models that could be compared against their pretrained, non-instruction-tuned counterparts

Thank you for this insightful suggestion.

• Based on the reviewer's suggestion, we conducted two additional experiments: (i) VILA-7B: a pretrained vision-language model with few-shot prompting capabilities, (ii) Qwen-2.5-Omni-7B, a pretrained-only counterpart to the instruction-tuned Qwen-2.5-VL-7B-Instruct model.
  • For both VILA and Qwen-2.5-Omni, we report alignment scores across whole brain, language, visual, and auditory regions, alongside all previously evaluated models.

We make the following observations based on the updated results in the Table:

• Both VILA and Qwen-2.5-Omni, which are pretrained multimodal models without instruction tuning, perform significantly better than TVLT and other unimodal baselines across all brain regions, including whole brain, language, visual, and auditory.
• However, instruction-tuned video MLLMs continue to outperform these pretrained models, demonstrating higher alignment across all regions.
• These findings support the conclusion that instruction tuning provides additional gains beyond few-shot prompting in pretrained multimodal models.

In response to the reviewer’s suggestion, we will include the updated results from pretrained models with in-context learning capabilities, along with their corresponding brain activation maps, in the final version.

We thank the reviewer for their positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1. The evaluation relies solely on Pearson Correlation (PC). Could you include additional metrics (e.g., Spearman’s, MSE) to better assess brain-model alignment?

Thank you for this question.

• We would like to clarify that Pearson Correlation (PC) is the standard evaluation metric widely used in brain encoding studies (e.g., Schrimpf et al., 2021; Wehbe et al., 2014; Huth et al., 2016; Toneva et al., 2019; Antonello et al., 2021; Tuckute et al., 2024; Deniz et al., 2019; Oota et al., 2022, Caucheteux et al. 2022, Goldstein et al. 2022).

• We further quantify these model predictions by measuring the normalized brain alignment where the resulting model prediction correlations are divided by the estimated cross-subject prediction accuracy, as established in prior work [Schrimpf et al., 2021; Oota et al., 2024; Alkhamissi et al., 2024; Oota et al., 2025].

• In response to the reviewer’s suggestion, we additionally computed Spearman correlation for one representative model and subject.

  • Across the whole brain, we observed a mean correlation of 0.144 using Pearson and 0.146 using Spearman, indicating that both metrics yield similar voxelwise alignment performance.
  • These results suggest that model-brain alignment is robust across both linear (Pearson) and rank-based (Spearman) correlation measures, supporting the use of Pearson as a reliable evaluation standard in this context.

[Schrimpf et al. 2021] The neural architecture of language: Integrative modeling converges on predictive processing. PNAS, 2021

[Alkhamissi et al., 2024] Brain-Like Language Processing via a Shallow Untrained Multihead Attention Network, NAACL-2025

[Oota et al. 2024] Speech language models lack important brain relevant semantics, ACL-2024

[Oota et al. 2025] Multi-modal brain encoding for multi-modal stimuli, ICLR-2025

[Wehbe et al 2014]. Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLoS One 2014

[Huth et al. 2016] Natural speech reveals the semantic maps that tile human cerebral cortex, Nature 2016

[Toneva & Wehbe 2019] Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain), In NeurIPS-2019.

[Goldstein et al. 2022] Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 2022

[Antonello et al. 2021] Low-Dimensional Structure in the Space of Language Representations is Reflected in Brain Responses, NeurIPS 2021

[Caucheteux et al. 2022] Brains and algorithms partially converge in natural language processing, Communication Biology

[Tuckute et al. 2024] Driving and suppressing the human language network using large language models, Nature Human Behaviour

[Deniz et al. 2019] The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality

Q2. The paper would benefit from a more detailed comparison with existing MLLM-based brain encoding and decoding methods to better clarify its novelty.

Thank you for this question.

• As indicated in Table 4 (Appendix), researchers have recently investigated brain alignment using multimodal models (Dong & Toneva et al. 2023, Oota et al. 2025, Subramanian et al. 2024, Nakagi et al. 2024) and MLLMs with unimodal stimuli (Oota et al. 2025). We provide a comprehensive overview of these works in multimodal evaluation settings in Table 4.

• Building on this literature, our study introduces and systematically evaluates instruction-tuned video and audio MLLMs, which had not been previously examined in brain alignment tasks.

• To our knowledge, this is the first study to:

  • Apply instruction-tuned video and audio MLLMs to model multimodal naturalistic stimuli,
  • Perform task-specific decomposition across brain regions to understand representational specialization,
  • Compare instruction-tuned, unimodal, and non-instruction-tuned multimodal baselines within a unified encoding framework.

• While we acknowledge the growing interest in brain decoding using MLLMs (e.g., reconstructing images or captions from brain data), our focus is strictly on brain encoding-predicting brain activity from external stimuli representations. As such, comparisons with decoding studies fall outside the scope of our contributions.

Dong et al. (2023) Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain, ICLRW-2023

Q3. Figure 3 visualization is difficult to interpret, and the task-specific brain maps do not appear to reveal meaningful spatial patterns.

Thank you for this question.

• We would like to clarify that the goal of Figure 3 is to support our central claim that no single task representation best predicts the entire brain. Instead, we observe spatial dissociation across tasks, which suggests that instruction-tuned video and audio MLLMs learn task-specific representations that align with functionally distinct brain regions.

For example, we find that:

• Sound event detection representations strongly align with the auditory cortex and temporal lobe, and
• Audio captioning representations are more predictive of language-associated regions and frontal cortex.

This task-level separation indicates that instruction-tuned MLLMs disentangle modality- and function-specific signals, producing semantically distinct embeddings aligned with neural specialization. In addition, a detailed explanation of Figure 3 is provided in Section 4.2, and we will revise the figure to improve clarity.

Q4. The checklist says LLMs were only used for grammar checking, but the method relies heavily on MLLMs/LLMs. Why?

Thank you for this question. We thank the reviewer for pointing this out and agree that our initial checklist response was unclear.

• In this study we use instruction-tuned MLLMs, multimodal video-audio models, unimodal video and audio models as frozen feature extractors to obtain stimulus representations for brain encoding experiments.
• These models were not developed or fine-tuned as part of our contribution but were used off-the-shelf. We only used LLMs for grammar checking during manuscript preparation in addition to this methodological usage.
• In the final version, we will revise the checklist to accurately state that LLMs were used for feature extraction in the core method and for checking the grammar assistance.

We thank the reviewer for their feedback.

• We acknowledge that in brain encoding studies, commonly used evaluation metrics include Pearson correlation, 2-versus-2 (2V2) accuracy, and R² score. For brain decoding models, typical metrics include pairwise accuracy, rank accuracy, R² score, and mean squared error.

• Unlike brain encoding or decoding models that rely on linear regression, CKA and RSA are nonparametric similarity metrics that do not require additional training, making them complementary tools for model-brain alignment evaluation.

• In this study, we primarily use Pearson correlation and also quantify these model predictions by measuring the normalized brain alignment where the resulting model prediction correlations are divided by the estimated cross-subject prediction accuracy, as established in prior work.

• Regarding related work, we have already provided a detailed discussion in Appendix Sections A and B.

• We sincerely appreciate the reviewer’s time and effort in evaluating our work, especially given their noted background in brain encoding studies. We are confident that the feedback provided will help further improve the clarity and impact of the paper.

We thank the reviewer for their positive, insightful and valuable comments and suggestions which are crucial for further strengthening our manuscript.

Q1.1 Can the better performance of instruction-tuned video models be explained by differences in depth or architecture?

Thank you for this insightful question.

• We agree with the reviewer that instruction-tuned video and audio MLLMs generally have more layers compared to the non-instruction-tuned multimodal (TVLT) and unimodal baselines. However, our results suggest that depth alone does not account for the observed performance differences. Notably, only the instruction-tuned video MLLMs show significantly higher brain alignment across all regions, while the audio MLLMs—despite having similar depth—only show improvements in the auditory cortex. This disparity indicates that instruction-tuning, rather than depth alone, plays a key role in enhancing brain alignment.
• Furthermore, our analysis (see Fig. 4) shows that different layers of the MLLMs align with distinct brain regions in a hierarchical manner, consistent with prior findings in neuroscience. If model depth were the primary driver of performance, we would expect a more uniform improvement across brain regions and models, which is not supported by our results.

Q1.2 What datasets are the models trained on? Are the datasets similar in their overall size and diversity?

Thank you for this important question.

• We now provide additional discussion of model training data to clarify both the dataset composition and diversity for each model.
• In the below Table, we provide a detailed summary of the datasets they were trained on, along with information about dataset size and content diversity.

Summary:

The instruction-tuned video MLLMs differ substantially in training sources:

• Some (like LLaVA-NeXT-Video and Video-LLaVA) include explicit video instruction datasets, while others (like InstructBLIPVideo and LLaVA-OneVision) primarily adapt image-based instructions to video via frame sampling.
• Model training sizes close to several million across all instruction-tuned video MLLMs, with varying modality mixes and task diversity.
• The unimodal and non-instruction-tuned baselines are trained on narrower or more specialized domains, lacking the instruction-driven, multimodal task generalization seen in instruction-tuned MLLMs.

Overall, we hypothesize that the use of task-specific instructions during training is a key factor driving the similarly high levels of brain alignment observed across instruction-tuned video MLLMs, regardless of differences in architecture or dataset scale.

Q1.3 One additional baseline: Would a supervised video model (TimeSFormer) be closer to the instruction-tuned video models?

Thank you for this insightful suggestion.

• Based on the reviewer's suggestion, we now perform one additional baseline experiment: (i) TimeSformer model. For the TimeSformer model, we use the “facebook/timesformer-base-finetuned-k400” model for extracting feature representations.
• We report the normalized brain alignment across whole brain, language, visual, and auditory regions for TimeSformer alongside all previously evaluated models in the table below. This replicates the results shown in Fig. 2, now including TimeSformer for direct comparison.

• While TimeSformer demonstrates stronger performance than unimodal baselines (e.g., VideoMAE, AST) and is comparable to the non-instruction-tuned multimodal model TVLT, it still significantly underperforms relative to instruction-tuned video MLLMs across the whole brain, language, visual, and auditory regions.
• This further reinforces our conclusion that instruction tuning-rather than supervision alone-is critical for learning representations that align with fMRI brain activity.

Q2.1 There seems to be a discrepancy between the text (spatial understanding) and Figure 5b flatmap (IPS regions not highlighted). Can you clarify?

Thank you for pointing this out and providing a valuable suggestion.

• We acknowledge that the intraparietal sulcus (IPS) was not included in the current ROI set because our analyses were constrained to Fedorenko’s language network, which excludes dorsal parietal regions. This omission likely caused the discrepancy noted in the flatmaps shown in Figure 5.
• Following your suggestion, we will include key IPS subregions-specifically IP1, IP2, and LIPd from the Glasser atlas-in the final version of the figure to more accurately reflect spatial task-related variance in the dorsal parietal cortex, especially for the spatial understanding task (line 301), and ensuring that descriptions in the text accurately match the data.

Q2.2 It would help to see these results quantified for certain ROIs or label the ROIs on the flatmaps to make the patterns clearer.

Thank you for this question.

• Based on the reviewer’s suggestion, we now provide normalized brain alignment values for both spatial and temporal understanding tasks in two key ROIs: the intraparietal sulcus (IPS) and the posterior cingulate cortex (PCC) which is part of Fedorenko’s language network.
• Consistent with prior work, IPS shows stronger involvement in spatial processing (Sack, 2009; Papadopoulos et al., 2018), while PCC is active in both the domains (Coull & Nobre, 2008):

IPS:

• Spatial: 0.723±0.020
• Temporal: 0.672±0.013

PCC:

• Spatial: 0.675±0.020
• Temporal: 0.651±0.012

These findings support the functional specialization of IPS for spatial cognition, and a more domain-general role of PCC across both spatial and temporal understanding.

Sack, 2009. Parietal cortex and spatial cognition. Behavioural brain research

Coull & Nobre, 2008. Dissociating explicit timing from temporal expectation with fMRI. Current opinion in neurobiology

Papadopoulos et al. 2018. Functional subdivisions within the human intraparietal sulcus are involved in visuospatial transformation in a non‐context‐dependent manner. HBM

• We have labeled several language and visual ROIs on the flatmap in Appendix Figure 6. In line with the reviewer’s suggestion, we will provide quantitative results for multiple language ROIs in the final version of the manuscript.

Q3. In the section on banded ridge regression (line 171), it’s a bit confusingly worded. How did you use this to decompose the variance? Did you also use variance partitioning?

Thank you for this question.

• We would like to clarify that different task-specific representations require different levels of regularization, which is why we use banded ridge regression.
• This approach enables separate regularization strengths (as pointed out by the reviewer) for each feature group (e.g., different tasks) and incorporates an implicit feature-space selection mechanism that ignores the contribution of non-predictive or redundant feature spaces.
• For single-modal baselines (i.e., those with only one feature space), we use banded ridge regression with a single group, but still allows us to maintain consistency across evaluations.
• To measure per-task predictive performance in the joint model, we use the split-correlation measure, which computes the correlation between the predicted and actual brain responses from each task-specific feature space independently.
• This allows us to interpret how much each task-specific representation contributes within the joint predictive framework. Implementation ref: predict_and_score_weighted_kernel_ridge(Ks_test, dual_weights, deltas, Y_test, split=split, n_targets_batch=n_targets_batch, score_func=score_func).
• In addition, we also perform individual task modeling and conduct variance partitioning between task pairs to evaluate unique and shared variance explained.
• Appendix Table 13 reports the results of variance partitioning across all 13 video tasks, averaged across subjects, for the whole brain, visual, and language regions using the Qwen-2.5-VL model.

We will clarify these distinctions more explicitly in the final revised version of the manuscript.

Q4. I would suggest a different color for Figure 5b.

Thank you for this valuable suggestion. For the middle panel in Figure 5, we will update the visualization using a more distinguishable and accessible color palette in the final revised version.

We thank the reviewer for their positive, insightful and valuable comments and suggestions, which are crucial for further strengthening our manuscript.

Q1. Strong pre-trained models (e.g., OpenFlamingo, VILA) can perform well in few-shot settings, sometimes surpassing instruction-tuned counterparts on unseen tasks (e.g., MM130B). Could the reported benefits stem from few-shot prompting capabilities of pre-trained models rather than instruction tuning?

• We would like to clarify that our definition of a non-instruction-tuned multimodal model refers specifically to multimodal models (like TVLT), which are jointly pretrained on video and audio without any instruction tuning or in-context learning capabilities. We do not consider in-context learning models in this category, as those still rely on prompting capabilities, albeit implicitly. We would clarify this in the revised draft.

• However, we agree with the reviewer that a few-shot prompting comparison will also add good value to the current paper.

• Based on the reviewer's suggestion, we conducted two additional experiments: (i) VILA-7B: a pretrained vision-language model with few-shot prompting capabilities, (ii) Qwen-2.5-Omni-7B, a pretrained-only counterpart to the instruction-tuned Qwen-2.5-VL-7B-Instruct model.

  • Since the OpenFlamingo model has version compatibility issues with PyTorch and the MM1 model is not publicly available, we chose the VILA model and extracted video features from image frames.
  • For both VILA and Qwen-2.5-Omni, we report alignment scores across whole brain, language, visual, and auditory regions, alongside all previously evaluated models.

We make the following observations based on the updated results in the Table:

• Both VILA and Qwen-2.5-Omni, which are pretrained multimodal models without instruction tuning, perform significantly better than TVLT and other unimodal baselines across all brain regions, including whole brain, language, visual, and auditory.
• However, instruction-tuned video MLLMs continue to outperform these pretrained models, demonstrating higher alignment across all regions.
• These findings support the conclusion that instruction tuning provides additional gains beyond few-shot prompting in pretrained multimodal models.

In response to the reviewer’s suggestion, we will include the updated results from pretrained models with in-context learning capabilities, along with their corresponding brain activation maps, in the final version.