Improving Semantic Understanding in Speech Language Models via Brain-tuning

Thank you Reviewer 61Sy for your detailed review of the work. We are happy you found the paper to be clearly written and the results to be solid and impressive. Below, we address the questions and concerns you have raised, which we hope increase the clarity and credibility of the work.

1. fMRI Dataset Size: The brain-tuning in our work trains one model for each participant. Therefore, the main criterion for choosing the dataset was the unique amount of data available for each participant, which is the largest for the LeBel's dataset (6.4 hours or more vs 4.6 for the Narratives dataset). Additionally, we repeat this per-participant brain-tuning over enough participants to show statistically significant results in brain alignment. The point of the work is that here we show that even data from one participant, which is equivalent to 0.7% of pretraining data of Wav2Vec2.0 is sufficient to improve semantic understanding in the models. We plan to extend the method to be applied across different participants in future work, which will be an ideal setting for the Narratives dataset that has many more participants.

2. Influence of model parameters: We are not certain we fully understand what the reviewer means by “influence of model parameters“, but we assume it to be the influence of model size (# parameters). We note that we leave brain-tuning larger models for future work, when the methods can incorporate brain data from multiple experiments in order to have more data. Still, brain-tuning smaller models is beneficial, and we further show that brain-tuning can bridge the gap in performance between smaller and larger models as shown in Appendix A.2 and E.2 (i.e., the performance of brain-tuned smaller models gets much closer to the performance of the pretrained larger models).

3. Details on Low-level Speech Features: We added more details about the definition and the extraction method of the low-level speech features in Appendix A.3.

4. ًWhisper Performance: Our results show that brain-tuning improves Whisper’s semantic understanding because 1) its downstream performance on more semantic tasks is improved and 2) the impact of low-level features on its brain alignment is decreased. As for the brain alignment results, please note that the results reported are not only for the voxels used during fine-tuning but for a wider set of voxels, which might explain the final brain alignment not necessarily increasing. That’s why, to more comprehensively assess the changes in a model thoroughly, we rely on multiple types of evaluation.

5. Cortical Parcellation Method: To extract language and speech regions, the cortical parcellation is indeed conducted by projecting the participant data to the same space. We use the FAverage standard surface version 7. [1, 2] Thank you very much for your help in evaluating our work! We hope we have addressed your questions and look forward to continuing the discussion if there are unresolved issues.

[1] Bruce Fischl, Martin I. Sereno, Roger B.H. Tootell, and Anders M. Dale. High-resolution intersubject averaging and a coordinate system for the cortical surface. Human Brain Mapping, 8(4):272–284, 1999.

[2] https://surfer.nmr.mgh.harvard.edu/fswiki/FsAverage

Thank you Reviewer SHAj for your attentive review of the work. We are happy you find the work novel, thorough, and significant. Below, we address the concerns you have raised, which we hope increase the clarity and quality of the work:

1. New baseline and clarification of existing baselines:

   1. LM fine-tuned baseline: we now implement a new baseline to alleviate your concerns. We fine-tune each pretrained model using representations from a text-based language model (GPT2), and show that this baseline also performs worse than the brain-tuned models (Appendix D.2). Please see point 4 in the common response for more details.

   2. BigSLM fine-tuned baseline: we believe a BigSLM Baseline is the best baseline to understand the added benefit of fMRI data over speech models, which is the aim of our approach. We also note that this baseline outperforms the pretrained model or performs on par with it for most tasks (Fig 4 in the updated paper).

   3. Random-fMRI baseline: we believe this baseline is important as it’s not always true that random or noisy targets are trivially insignificant to the model’s performance. As we detail in the common reply (Point 4.3), sometimes they act as regularization and can help the model. Hence, it’s important to add a grounding baseline to compare the fMRI against. ‌ ‌

   Based on the points above, we think that we covered important comparison models that provide both semantic and speech improvement comparisons against the brain-tuning models. We have also shown that their behavior is consistent across 3 model architectures. We hope this clarifies the concerns about the effectiveness of the comparison models we used, and please let us know if you have more thoughts on other baselines.

2. Brain Data Limitation: The concern about data size and collection is of course valid. The benefit of the current work is that here we show that even data from one participant, which is equivalent to 0.7% of pretraining data size of Wav2Vec2.0 is sufficient to improve semantic understanding. In the future, our method can be extended to incorporate data from multiple experiments ( multiple participants and datasets). This may also enable the application of brain-tuning to larger models.

Thank you very much for your help in evaluating our work! We hope we have addressed your questions and look forward to continuing the discussion if there are unresolved issues.

Thanks, I believe this two new baselines will make the work substantially convincing and I've updatied the score.

Thank you Reviewer BDuH for your in-depth review of the work. We are pleased you enjoyed the paper and found our approach and results impactful and beneficial. Also, thank you for referring us to the work of Dapello et. al. 2023 in the vision domain, which we will refer to in the next revision of the manuscript. Below, we address the raised concerns:

1. LM-finetuned baseline: Please see point 4.1 of the common response above. Briefly, we have added an LM (GPT2)-finetuned baseline, and we report its results on downstream tasks against brain-tuned models in Appendix D.2. Our brain-tuned model consistently outperforms this LM baseline.
2. BigSLM Baseline: Please see the common response point 4.2 for more details. Briefly, we believe a BigSLM Baseline is the best baseline to understand the added benefit of fMRI data over speech models, which is the aim of our approach.
3. Dimensionality of Targets: We clarify that the BigSLM targets and the fMRI data are of similar dimensions: ~30-40K dimensional vector for the fMRI targets and ~26K for the BigSLM targets). Both fMRI targets and BigSLM targets are also treated similarly in the MSE loss as it is applied to the flattened high-dimensional vector of both in the same way, not per voxel for the fMRI case (all used voxels are flattened into a vector and used to obtain one MSE value). Hence, the way we apply the loss and the complexity of the reconstruction is comparable in both cases.
4. Brain areas covered in brain-tuning: Based on qualitative analyses, our brain-tuning approach covers a large number of semantic regions, as well as the auditory cortex (Appendix F.1). Your suggestion to brain-tune with specific regions is very interesting; however, only a couple of regions (namely Broca and Auditory Cortex (AC)) are anatomically annotated across subjects by the authors of the datasets, limiting our ability to do a thorough ablation study on the different regions/ networks of the brain. Still, we did initial experiments with brain-tuning Wav2vec2.0 only using these two regions (Broca and AC). We include the results in Appendix E.1 of the paper, which show that brain-tuning using these two regions boosts the downstream performance compared to the pretrained model, but it still underperforms compared to using the noise-filtered whole brain. Trying to efficiently post-process the original dataset to be able to functionally localize all relevant semantic and speech regions is one of the future directions.
5. Concern about potential forgetting of speech capabilities: Please take a look at point 3 in the common response. Briefly, we believe our brain-tuning setup avoids catastrophic forgetting of the speech capabilities because the brain regions that are included in the training do not only include semantic regions, but also low-level speech regions (the auditory cortex). Furthermore, we now show results for 2 additional low-level tasks (MFCC and FBANK prediction), further supporting that brain-tuned models maintain speech capabilities (Appendix D.1).
6. 3-lobed Lanczos filter: This filter acts as a low-pass antialiasing filter (with a cut-off at the Nyquist frequency of the fMRI) and allows for better and more efficient downsampling of the convolved input signal (the input signal after applying the filter). Moreover, it’s been shown to be consistently better than applying an average filter to downsample the input [2], which is why it's a standard choice in the literature [1,2].

Again, thank you very much for your detailed and clear review! We hope we have addressed your questions and look forward to continuing the discussion if there are unresolved issues.

[1​​] Oota, S.R., cCelik, E., Deniz, F., & Toneva, M. (2023). Speech language models lack important brain-relevant semantics. Annual Meeting of the Association for Computational Linguistics.

[2] LeBel, A., Wagner, L., Jain, S. et al. A natural language fMRI dataset for voxelwise encoding models. Sci Data 10, 555 (2023). https://doi.org/10.1038/s41597-023-02437-z

I thank the authors for additional clarifications and baselines. However, I am left a bit confused about the reported result from using an LM representation as a target as requested. The authors claim that they can remove low level features and improve semantic understanding in speech encoder models. Yet they do so with fMRI, an incredibly noisy target. When they replace this target with a GPT-2 embedding -- a more feature rich, clean representation of semantics -- the result is worse? In fact, if I am reading the figure correctly, it performs even worse than the base model at times. I had initially explained the result from the BigSLM as follows: the audio model as a target was reinforcing particular features, causing less robust improvement. This would have been complemented by using an LLM target. But now the explanation seems inadequate.

I would really appreciate if the authors could provide some discussion on this result and their intuitions for what is happening. My sense is that the language model should be an upper bound for a clean representation of semantics. That intuition could be wrong, but I find myself a bit surprised by the finding.

It's possible though. LM representation may not contain much low-level speech feature.

Also note in Figure 10, authors only post partial comparison results, not as comprehensive as Figure 4.

I also doubt that is the evaluation dataset more similar to the fMRI audio data than to the pretrained audio data.

Thank you Reviewer x8dp for your detailed reading and constructive criticism. We were happy you found the paper well-motivated and novel. Below, we address the questions and concerns you have raised, which we believe improve the clarity and strengthen the confidence in the results:

1. Architecture: The only addition to the speech model for brain-tuning is a linear projection layer from the output of the speech model to the fMRI signal (similar to any fine-tuning pipeline). There are no additional architectural changes to the original version of each model.

2. Evaluation audio datasets: All datasets used for downstream evaluation were not seen before by any of the models. The datasets are completely new audio, different from the fMRI audio data used for brain-tuning as well. We believe this ensures fairness in the evaluation. Please refer to the common reply point 2 for further details.

3. ASR performance: Thank you for the careful reading of the methods and results. We want to clarify two points concerning ASR results.

   • The ASR performance we report in the revised Fig 4a is on par with previous results on the TIMIT dataset. There was a plotting mistake in the original subplot, which made the ASR performance appear low. The correct results were reported in the text, and we have adjusted the plot to correctly reflect these results. Please refer to point 1 of the common reply above for more details.

   • The reason why our reported ASR results are on TIMIT and not on the Librispeech dataset, as in the original Wav2Vec2.0 paper, is that we wanted to use a dataset for evaluation that did not unfairly bias the evaluation towards either the pretrained or the brain-tuned model (Wav2Vec2.0 is pretrained on Librispeech).

4. Concern about potential catastrophic forgetting of speech capabilities: Please take a look at point 3 in the common response. Briefly, we believe our brain-tuning setup avoids catastrophic forgetting of the speech capabilities because the brain regions that are included in the training do not only include semantic regions, but also low-level speech regions (the auditory cortex). Furthermore, we now show results for 2 additional low-level tasks (MFCC and FBANK prediction), further supporting that brain-tuned models maintain speech capabilities (Appendix D.1).

5. Low-level Impact Reasoning: The work discussed in lines 237-238 shows that the late language regions process information that is 1) partially correlated with low-level features, but also 2) partially independent of low-level features. This previous work shows that pretrained speech models align with late language regions because of 1). The inspiration behind our brain-tuning approach is to allow speech models to learn 2) directly from the brain signal. If successful, this approach would align with both 1) and 2) types of information in late language regions, thereby reducing the fraction of the alignment that is due only to 1) , i.e. the low-level impact on the brain alignment with late language regions. We will clarify this line of reasoning in the paper.

6. Methodology figure X-axis labels: We have added X-axis labels to Figure 1c to disambiguate the targeted downstream tasks by mentioning examples of tasks we expect these models to be measured on.

We thank you very much for your thorough review! We hope we have addressed your questions and look forward to continuing the discussion if there are unresolved issues.

[1] J. S. Garofolo, L. F. Lamel, W. M. Fisher, J. G. Fiscus, D. S. Pallett, and N. L. Dahlgren. The DARPA TIMIT Acoustic-Phonetic Continuous Speech Corpus CDROM. Linguistic Data Consortium, 1993.

I thank the authors for taking the time respond. I would like to clarify a few things. Is this the procedure for brain-tuning?

1. An off-the-shelf speech model is taken
2. This model is fine-tuned to predict brain voxels and nothing else
3. When evaluated on speech datasets, the performance of the original speech model on audio tasks is not only retained, it actually improves.

I feel I must be missing something. How can a model be trained on a completely different task, albeit one with correlation to the original task, and still retrain such good performance? I think it would be really good to test on the original data that wav2vec2 and HuBERT tested on, even if it is "biased" to the pretrained models. This will help determine what braintuning does to the original model performance.

[1] A. Baevski, W. Hsu, A. Conneau, and M. Auli. "Unsupervised speech recognition." NeurIPS 2021.

[2] https://github.com/elgeish/transformers/tree/cfc0bd01f2ac2ea3a5acc578ef2e204bf4304de7

[3] A. Vaidya, S. Jain, and A. Huth. Self-supervised models of audio effectively explain human cortical responses to speech. ICML 2022.

[4] S. Zada, I. Benou, and M. Irani. "Pure noise to the rescue of insufficient data: Improving imbalanced classification by training on random noise images." ICML 2022.

[5] C. Bishop, "Training with Noise is Equivalent to Tikhonov Regularization," in Neural Computation 1995.