Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models

We thank the reviewer for their review of our work and literature suggestions. Below, we clarify the distinctions of our contributions compared to the mentioned previous work (further supporting it with new results), address interpretability concerns, and discuss the suggested relevant literature.

Q1: Novelty Compared to Recent Work

A. Distinction from Brain decoding work by Tang & Huth (2025)

We appreciate the pointer to the recent decoding work by Tang & Huth (2025). However, we want to highlight that this study investigates generalization in brain decoding rather than brain encoding (and is not the first to show decoding generalization). The two domains differ a lot, and transferring methods between them can be challenging.

Tang & Huth (2025) demonstrate strong generalization across stimulus modalities, extending previous decoding successes in speech [1], language [2, 3, 4], and vision [5, 6, 7] in unifying participant responses and generalizing to new participants. Some of these methods show impressive generalization [8, 9], including zero-shot decoding [10]. Despite these advances, achieving similar generalization in brain encoding remains challenging and unresolved. Previous attempts typically resulted in substantial performance drops compared to single-subject models (see below). Adapting decoding approaches directly may not be straightforward or even feasible for encoding tasks, as discussed in [11]. This emphasizes the non-triviality of generalization in brain encoding, an important gap that our work addresses.

B. Challenges in Brain Encoding Generalization

Unlike decoding, brain encoding models typically fail to generalize effectively to new participants. Common participant-alignment methods, such as SRM[22] and FreeSurfer, substantially degrade encoding performance (~40% drop, as shown below; cross-participant encoding models trained on 3 participants and tested on a 4th). Even recent fine-tuning approaches like BrainWavLM [15] show around a 30% drop in cross-participant performance (where the speech model is fine-tuned on one participant, but the encoding model is trained and evaluated on others).

Our approach is, to our knowledge, the first to demonstrate significant improvements and generalization in speech and language encoding by fine-tuning with neural data. Crucially, our method simultaneously improves performance on both trained and held-out participants, addressing an important gap unresolved by prior work.

C. Comparison with BrainWavLM (Vattikonda et al., 2025)

Both Brain-tuning [14] and BrainWavLM [15] introduce a framework for fine-tuning speech models with fMRI data, with Brain-tuning [14] being the first such work. However, neither paper offers a way to adapt to multi-subject or multi-dataset settings; they also show limited transferability across subjects. We emphasize that successful generalization across participants requires extensive, careful design choices and is not a trivial extension of those works. Specifically, our results demonstrate:

1. The method of aggregating and presenting multiple participant responses critically influences generalization (Fig. 6, Supp A.3).
2. Loss function and model capacity substantially impact performance and generalization ability (Fig. 4).
3. new results, Investigating different methods for Multiple participant handling: In order to further demonstrate the effectiveness of our design choices, we explore the performance of different multiple participants handling methods. Specifically, we compare our Multi-brain-tuning against three alternatives: (i) separate participant-specific projection heads (as in prior work [23]), (ii) a non-linear projection network (MLP instead of linear), and (iii) SRM-based participant alignment (instead of FreeSurfer). The table below summarizes alignment improvements for training and held-out participants.

Our Multi-brain-tuned model outperforms all alternatives while being:

• more parameter-efficient compared to separate heads or non-linear alternatives, which scale poorly with participant number (e.g., 16 separate projection heads for Narratives).
• more controllable in selecting targeted brain areas unlike SRM.
• easier to apply across multiple datasets due to FreeSurfer’s widespread adoption.

Q2: Interpretability of Multi-Brain-Tuned Models

We argue that our multi-brain-tuned models are at least as interpretable as their pretrained counterparts. Our method fine-tunes the full architecture, unlike BrainWavLM [15], which removes some of the model layers. Thus, existing interpretability analyses for speech models (layer-probing, attribution, representation analysis) remain fully applicable.

Moreover, understanding and interpreting why models align with the brain in the first place is an active research area ([16, 18, 19, 20, 21]. Methods developed in this context apply equally to our Multi-brain-tuned models, potentially yielding further insights into the effect of increased neural alignment. For instance, recent work [16] demonstrates that speech brain-tuned models’ layers reflect hierarchical brain processing stages better than pretrained models. Similar analyses can be easily employed on our Multi-brain-tuned models, and we hope that our work inspires such future work.

In essence, since our models can be used, analyzed, and deployed in the same manner as their popular pretrained counterparts, our approach wouldn’t complicate interpretability analyses of these models.

Q3: Revising Literature and Terminology

We appreciate the reviewer’s feedback regarding scholarly literature balance. We note, nevertheless, that our related work discussion covers relevant brain encoding work, and that our repeated comparison to [14, 15] is because they are the only recent works related to brain-tuning of speech models. However, we agree that expanding our literature review to integrate more brain decoding and neuroscience-focused work (such as Tang & Huth, 2025, and others mentioned earlier) will better contextualize our contributions.

Regarding terminology, our use of "model organisms" aimed at highlighting parallels with biological research models, distinguishing our usage from general AI "models." This term has been similarly employed in recent survey literature [17]. Nevertheless, we understand the potential confusion and will clarify this usage in the manuscript.

We greatly appreciate the reviewer’s feedback and believe these clarifications significantly strengthen the paper. We hope these revisions clearly highlight the contributions, novelty, and interpretability of the work, and hope these improvements are reflected positively in your evaluation.

References

[1] Défossez et al. (2023). Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence.

[2] Lévy et al. (2025). Brain-to-Text Decoding: A Non-invasive Approach via Typing.

[3] d’Ascoli et al. (2024). Decoding individual words from non-invasive brain recordings across 723 participants.

[4] Liu et al. (2025). Learning interpretable representations leads to semantically faithful EEG-to-text generation.

[5] Benchetrit et al. (2024). Brain decoding: toward real-time reconstruction of visual perception. ICLR.

[6] Liu et al. (2023). BrainCLIP: Bridging brain and visual-linguistic representation via CLIP for generic natural visual stimulus decoding.

[7] Chen & Wei (2024). Mind’s Eye: Image recognition by EEG via multimodal similarity-keeping contrastive learning.

[8] Thual et al. (2023). Aligning brain functions boosts the decoding of visual semantics in novel subjects.

[9] Dahan et al. (2025). SIM: Surface-based fMRI analysis for inter-subject multimodal decoding from movie-watching experiments. ICLR.

[10] Aristimunha et al. (2025). EEG Foundation Challenge: From cross-task to cross-subject EEG decoding.

[11] Beliy et al. (2024). The wisdom of a crowd of brains: A universal brain encoder.

[12] Schwartz et al. (2019). Inducing brain-relevant bias in natural language processing models. NeurIPS.

[13] Lu et al. (2024). Achieving more human brain-like vision via human EEG representational alignment.

[14] Moussa et al. (2025). Improving semantic understanding in speech language models via brain-tuning. ICLR.

[15] Vattikonda et al. (2025). BrainWavLM: Fine-tuning speech representations with brain responses to language.

[16] Moussa & Toneva (2025). Brain-tuned speech models better reflect speech processing stages in the brain. Interspeech.

[17] Oota et al. (2023). Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey). TMLR.

[18] Caucheteux & King (2022). Brains and algorithms partially converge in natural language processing. Communications Biology.

[19] Merlin & Toneva (2022). Language models and brains align due to more than next-word prediction and word-level information.

[20] Oota et al. (2023). Joint processing of linguistic properties in brains and language models.

[21] Oota et al. (2024). Speech language models lack important brain-relevant semantics. ACL.

[22] Chen et al. (2015). A Reduced-Dimension fMRI Shared Response Model. NeurIPS.

[23] Jayalath et al. (2025). The Brain’s Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning. ICML.

We thank the reviewer for the thorough evaluation of our work and the valuable literature suggestions. We are glad the reviewer found the problem important and the experiments well-designed. Below, we address the raised concerns in detail and provide additional results and clarifications. All changes described will be included in the final paper.

Showing Generalization to Novel Subjects and Stimuli from a Different Dataset

Experiment on novel stimuli + novel participants: To evaluate our model’s generalization to both unseen participants and unseen stimuli, we report below the improvement in brain alignment of multi-brain-tuned and single-brain-tuned models on an entirely new fMRI dataset that was not used at all in our brain-tuning approach: the Narratives dataset [K]. Results were calculated over 16 participants using only a 56 mins stimulus. Furthermore, as a strong baseline, we compare to a Multi-brain-tuned model trained directly on the Narratives participants (i.e., multi-brain-tuning with all of the 16 participants). The table below shows the means and standard error for the percent improvement in brain alignment relative to the pretrained model. The reported P-values are to indicate statistical significance from pretrained (obtained using the paired t-test).

Our brain-tuned models show general improvement on the new fMRI stimuli and participants from the Narratives dataset. Notably, our original Multi-brain-tuned model performs close to the Narratives-multi-brain-tuned model. This further demonstrates the generalization and efficiency of our method on novel subjects and stimuli with very limited data.

Design Choices for Multi-brain-tuning Compared to Alternatives like [C]

Our work indeed aims to create more generalizable and efficient brain-language models. While the method of unifying different participants in the same space is one of the most important design choices, it’s not the only one. In the paper, we explored several factors (data presentation/binding, loss function, number of trainable parameters) that significantly impact performance (see Fig. 6, Appendix A.3 for the effect of shared vs. separate stimuli presentation; Fig. 4 for loss and LoRA rank ablations). Here, we provide new additional analysis comparing our approach to alternative multi-participant handling strategies:

1. Separate projection heads per participant: We trained a variant of our model where each participant has their own projection layer (similar to the conditional projections in [C]). This drastically increases the number of parameters as participants increase.
2. Shared Response Model (SRM): We applied the classic SRM [H] to the fMRI data (i.e., instead of FreeSurfer), projecting all participants’ data into a shared space as a pre-processing step, and then trained a model on this aligned data.
3. Non-linear projection head: We replaced our simple linear projection with a multi-layer perceptron to test if a non-linear mapping of brain features improves alignment.

We measured the brain alignment improvement (relative to the pretrained model) on training and held-out participants for each variant:

Analysis: Our Multi-brain-tuned model outperforms these alternatives on train and held-out participants. This highlights that our proposed approach is not only better, but it’s also:

1. much more parameter-efficient than alternatives, which blow up with the number of participants; for instance, performing Multi-brain-tuning with separate heads on the Narratives data above would require 16 projection heads.
2. more controllable than SRM since we can control which brain areas to tune with, unlike SRM.
3. easier to perform across multiple datasets and recording devices since FreeSurfer is widely and easily adopted and doesn’t require re-fitting with new data.

Finally, we reiterate that our method aims at finding the best framework to achieve generalizability and efficiency for brain encoding, which we show does not depend on a single design choice (e.g., using LoRA) but rather many design choices, for which we test each one against strong alternatives.

More Clarifications on Novelty and Relevant Literature.

Thank you for pointing out the relevant recent literature on multi-participant generalization in brain decoding. While our work is primarily concerned with brain encoding, we agree that discussing the recent approaches in brain decoding can help contextualize the gap in the literature and highlight our contributions better. We promise to add that in the updated version of the paper, along with the new results mentioned earlier.

However, we want to emphasize that the generalization in brain encoding by adapting the same methods in brain decoding literature can be challenging and might not even make sense at times, as discussed in [I]. Unlike decoding, brain encoding models often fail to generalize effectively to new participants. Popular multi-participant unification approaches, such as SRM and FreeSurfer, substantially degrade encoding performance when applied across subjects (~40% drop as shown below; cross-participant encoding models trained on 3 participants and tested on a 4th). Even recent fine-tuning-based approaches like BrainWavLM [J] struggle with cross-subject generalization.

To the extent of our knowledge, our approach is the first to demonstrate substantial improvements in speech and language encoding performance and generalization by fine-tuning with neural data, while simultaneously beating single-participant models on training participants' performance.

We appreciate the reviewer’s positive feedback, insightful comments and for helping us improve the paper. We hope that our revisions and new results have addressed your concerns, and we would be grateful if the reviewer could reflect this in their score.

References:

[H] Chen et al. (2015). *A Reduced-Dimension fMRI Shared Response Model.NeurIPS 2015.

[I] Beliy et al. (2025). The wisdom of a crowd of brains: A universal brain encoder. arXiv preprint arXiv: 2406.12179

[J] Vattikonda et al. (2025). BrainWavLM: Fine-tuning speech representations with brain responses to language. arXiv preprint arXiv:2502.08866

[K] Nastase et al. (2021). The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Sci Data 8, 250.

.

We sincerely thank the reviewer for the very positive evaluation and thorough understanding of our work. We’re very pleased that the reviewer recognized the clarity, strength, rigor, and effectiveness of our approach - and "found no substantive weaknesses that would motivate rejection." We promise to release the full code publicly alongside the final version of the paper.

Thank you to the authors for their response. As other reviewers have noted, the paper applies the well-established idea of cross-subject generalization from decoding to the encoding context. Still, employing multi-brain tuning for encoding is novel, and the experiments are technically solid. I therefore maintain my Accept rating.

We thank the reviewer for the positive evaluation and insightful feedback. We’re pleased that the reviewer recognized the strengths of our approach, including its flexibility and robustness across individuals and stimuli. Below, we address the concern regarding generalization to auditory regions in detail and provide additional supporting analyses.

Clarification on Performance in Auditory Regions

The observed slight alignment drop in auditory regions alongside improvements in frontal and parietal regions reflects the specific tuning approach we adopted:

1. Reported Layers: The reported results in our main analysis were averaged over the upper-middle to late layers of Wav2Vec 2.0, known to encode more semantic rather than acoustic information. Thus, these layers are expected to increase in semantic rather than acoustic performance. This is also consistent with single-brain-tuning layer analyses in [1, 2].
2. Included Brain Regions: Our multi-brain-tuned model predominantly includes higher-level (late language) brain areas (see Supp A.1 for details on the regions). Consequently, improvements naturally concentrate in regions capturing semantic processing, which are much larger than early auditory regions.

Maintaining Acoustic Performance

To further clarify the impact of Multi-brain-tuning on auditory and acoustic performance, we report below the performance of our Multi-Brain-tuned model on 1) alignment with only the auditory regions, and 2) a low-level acoustic task (namely MFCC prediction detailed in [1]). As a baseline, we compare it to a multi-brain-tuned model that was tuned using only the brain auditory regions (rather than both auditory and late language areas).

The tables below show percent improvement relative to the pretrained model for our Multi-brain-tuned vs Auditory Multi-brain-tuned for (i) the brain alignment of early auditory regions and (ii) performance on MFCC prediction; reported results are the mean over early to mid layers of the model (which perform best in acoustic tasks [1]).

• Table i: | | Auditory Multi-Brain-tuned | Our Multi-Brain-tuned | | --- | --- | --- | | Train participants (mean ± STE) | 4.95 ± 0.13 | 3.465 ± 0.43 | | Held-out participants (mean ± STE) | 2.7 ± 0.53 | -1.25 ± 1.4 |

• Table ii:

Our Multi-brain-tuned shows small positive to zero improvement, indicating no strong negative effects on the acoustic performance of the model despite being tuned on semantics-dominated data. Auditory Multi-Brain-tuning shows improved performance on both alignment and acoustic performance, consistent with the findings of the single-participant case in [2]. The reason for not having as huge an improvement as in late language areas is both the small size of the auditory cortex and the already high alignment of pretrained models with auditory regions (pretrained speech models are able to predict ~80% of the estimated explainable variance in auditory regions [3]).

To sum up, substantial performance gains on high-level rather than low-level tasks is expected, but the above analysis demonstrates that the low-level utility of our Multi-brain-tuned model is either unaffected or slightly improved, alleviating the worry of a compromised low-level performance.

We appreciate the reviewer’s constructive comments and evaluation, which we believe will strengthen the manuscript. We will include these additional analyses, alongside detailed ablations exploring alignment improvements across different brain regions, in the final version of the paper.

References

[1] Moussa & Toneva (2025). Brain-tuned speech models better reflect speech processing stages in the brain. Interspeech 2025.

[2] Vattikonda et al. (2025). BrainWavLM: Fine-tuning speech representations with brain responses to language. arXiv preprint arXiv:2502.08866.

[3] Oota, et al (2024). Speech language models lack important brain-relevant semantics. ACL, 2024.