The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models

We are deeply appreciative of the time and attention you have dedicated to thoroughly reviewing our paper, and for your insightful remarks. We have responded to the comments on a point-by-point basis. We sincerely apologize for our inability to upload our response at an earlier time, which would have allowed for a more timely discussion. This delay was largely due to the substantial number of reviewers (5) for our work, and the time it took us to consolidate our responses to all their comments. We highly value your understanding regarding this matter and remain committed to considering and addressing all pertinent feedback to improve our work.

We would like to bring to your attention that we have made reference to several new figures that were generated as part of our response. These additional visual aids can be found in the supplementary file that we have uploaded, and you have access to this supplementary material.

Q1: While the authors state they used 10 patients, details are lacking on the specific demographic and clinical characteristics of the patient sample. This could impact the generalizability of the findings.

A1: We thank the reviewer for this comment and agree that not including this result obscures the generalizability of our findings. We have added a table to the supplementary material (Supp Table T1), which includes patient age, ethnicity, gender and language. We included a table in the supplementary materials with clinical characteristics of the patients and will make sure we add a reference to it in the main body of the paper (Supp Table T2).

Q2: Preprocessing steps for the ECoG data should be described in more detail (e.g. filtering, re-referencing, artifact rejection).

A2: We thank the reviewer for this comment. We do have a section on the preprocessing of the ECoG data in the Supplementary, which we have copied below. Given the opportunity we will add a reference to this section in the main body of the paper.

A.2 PREPROCESSING 66 electrodes from all patients were removed due to faulty recordings. Large spikes in the electrode signals, exceeding four quartiles above and below the median, were removed, and replacement samples were imputed using cubic interpolation. We then re-referenced the data to account for shared signals across all electrodes using the Common Average Referencing (CAR) method or an ICA based method (based on the participant’s noise profile). High-frequency broadband (HFBB) power provided evidence for a high positive correlation between local neural firing rates and high gamma activity. Broadband power was estimated using 6-cycle wavelets to compute the power of the 70- 200 Hz band (high-gamma band), excluding 60, 120, and 180 Hz line noise. Power was further smoothed with a Hamming window with a kernel size of 50 ms.

Q3: The encoding model parameters, such as context length, smoothing windows, regularization, could be optimized more thoroughly. Ablations could be performed to test the effect of these modeling choices.

A3: We thank the reviewer for their comment. We followed the encoding procedure of Goldstein et. al 2022, where they predict the brain from GPT2 last layer embeddings. We did this so that our results would build upon what they had observed for the last layer. We also wanted to avoid overfitting our model as we felt this might make our results less generalizable. However, we did verify robustness against different context lengths in the input to GPT2-XL (see Supplementary Figure S2) and window sizes of neural signals centered around word onset (see Supplementary Figure S12).

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

We continue our response in the next comment.

We are deeply appreciative of the time and attention you have dedicated to thoroughly reviewing our paper, and for your insightful remarks. We have responded to the comments on a point-by-point basis. We sincerely apologize for our inability to upload our response at an earlier time, which would have allowed for a more timely discussion. This delay was largely due to the substantial number of reviewers (5) for our work, and the time it took us to consolidate our responses to all their comments. We highly value your understanding regarding this matter and remain committed to considering and addressing all pertinent feedback to improve our work.

We would like to bring to your attention that we have made reference to several new figures that were generated as part of our response. These additional visual aids can be found in the supplementary file that we have uploaded, and you have access to this supplementary material.

Reviewer Comments and Responses:

Q1: There are really important methodological details that are difficult to find or not described. For instance, the neural signal that is being predicted is high-frequency broad band power (Appendix A2). This should be mentioned in the main text.

A1: We thank the reviewer for this comment. Given the opportunity we will add information about the neural signal that is being predicted to the main body of the text and will add references to the supplementary for more details on neural signal preprocessing.

Q2: There is a preselection for electrodes that is not explained beyond this statement: ``We selected electrodes that had significant encoding performance for static embeddings (GloVe) (corrected for multiple comparisons).'' This needs to be explained in much more detail. What exactly does that mean? How many electrodes were excluded?

A2: We thank the offer for this comment. We include a more in depth description of electrode selection in the appendix but did not refer to it in the main text. Given the opportunity we will fix this so that this information is more easily accessible. We have copied the section below, with modifications in bold to add clarity. We will incorporate these modifications into the text.

A.8 ELECTRODE SELECTION As stated in A.1, we began with 1106 electrodes on the left hemisphere, of which 66 were removed due to faulty recordings leaving 1040 (A.2). To identify significant electrodes from this set, we used a nonparametric statistical procedure with correction for multiple comparisons (Nichols & Holmes, 2001). At each iteration, we randomized each electrode’s signal phase by sampling from a uniform distribution. This disconnected the relationship between the words and the brain signal while preserving the autocorrelation in the signal. We then performed the encoding procedure for each electrode (for all lags). We repeated this process 5000 times. After each iteration, the encoding model’s maximal value across all lags was retained for each electrode. We then took the maximum value for each permutation across electrodes. This resulted in a distribution of 5000 values, which was used to determine the significance for all electrodes. For each electrode, a p-value was computed as the percentile of the non-permuted encoding model’s maximum value across all lags from the null distribution of 5000 maximum values. Performing a significance test using this randomization procedure evaluates the null hypothesis that there is no systematic relationship between the brain signal and the corresponding word embedding. This procedure yielded a p-value per electrode, corrected for the number of models tested across all lags within an electrode. To further correct for multiple comparisons across all electrodes, we used a false-discovery rate (FDR). Electrodes with q-values less than .01 are considered significant. This procedure identified 160 electrodes in early auditory areas, motor cortex, and language areas in the left hemisphere (hence 880 of the 1040 were excluded for being insignificant). We used subsets of this list corresponding to the IFG (n=46), TP (n=6), aSTG (n=13) and mSTG (n=28).

We continue our response in the following comment.

Q5: Are there differences between spoken words and written words? Do their brain-paths converge at some point? Where? Do we believe that representations are different depending on whether a word was perceived auditorily and visually? If semantic representation is what we are after, should we only compare representations "after" this point of convergence once the word took its abstract semantic representation form that is divorced the "mechanics" of delivery?

A5: A thorough literature review on the relationship between comprehension of written and spoken language is beyond the scope of our paper, which specifically focuses on speech comprehension. However, Regev, Honey, Simony, & Hasson (2013) did conduct a study comparing the brain dynamics of written and spoken language comprehension. Their findings revealed that certain areas, such as the IFG, were consistently activated regardless of the medium of language. In contrast, other areas, such as the STG, were specifically activated for auditory language, and the visual cortex exhibited unique activation for written words. In alignment with these findings, we commence our paper by emphasizing the significance of the IFG, as it remains invariant across different language modalities and later look at areas that are specific for auditory comprehension (such as STG). Regev, M., Honey, C. J., Simony, E., & Hasson, U. (2013). Selective and invariant neural responses to spoken and written narratives. Journal of Neuroscience, 33(40), 15978-15988.

Q6: How did you handle multiple subjects: were their data put into one large pot, or each subject was treated separately (by training subject-specific models)?

A6: We thank you for this question and the opportunity to clarify this point in our work. We trained a separate encoding model for each electrode. Thus we do not need to explicitly address the different patients.

Q7: Figure 2: How would the whole lag-layer-correlation picture look like if you would shuffle the words that are being decoded? I am trying to understand if this picture we are seeing is indeed due to language structure and representational similarities between brains and DLMs, and whether this nice picture would disappear if we carefully and deliberately remove the sought signal from the data?

A7: We thank the reviewer for their insight. Following the comment we shuffled the words, and by doing so mismatching the words from their neural signal and repeated the analyses. None of the ROI yielded significant correlation between the lag that yields the maximal correlation and the layer index. The plots could be seen at Supp S6. Given the opportunity we will include this new analysis in the paper.

Q8: Figure 2: Also, if we, once again, remove the signal from the data via some sort of permutation test, how all these plots will look like? Some other shape - which one? Just flat - why? The answer to this question will, in turn, lead to a question about why those shapes look like they do in the absence of the true signal. Figure 2: And one more here: how high the correlation would rise for the data we know should have no correlation? Would it be actually 0 all the way from -4000 to 4000, or because of some general dynamics and the way how to the linear readout models are trained some positive correlation will still be observed? How high?

A8: We thank the reviewer for these questions. We performed an analysis where we phase shuffled the neural signal and re-ran our encoding procedure. The lag layer correlation effect disappears. We also see maximum correlations in the coding plots < 0.05 (much less than for true signal) and none of them turned significant (p>0.1). The plots can be seen at Supp S7. Given the opportunity we will include this new analysis in the paper.

Q9: Could the correlation be explained by, say, non-sparsity of the representation in DLM in the mid layers, or some other technical reason, and not by the actual match between representations?

A9: We thank the reviewer for this comment. It is always the case that there are confounders we could not control (because we did not think about them). However, the control analysis we introduce in the original paper, as well as the controls suggested by the current reviewers give us confidence in the results.

We are deeply appreciative of the time and attention you have dedicated to thoroughly reviewing our paper, and for your insightful remarks. We have responded to the comments on a point-by-point basis. We sincerely apologize for our inability to upload our response at an earlier time, which would have allowed for a more timely discussion. This delay was largely due to the substantial number of reviewers (5) for our work, and the time it took us to consolidate our responses to all their comments. We highly value your understanding regarding this matter and remain committed to considering and addressing all pertinent feedback to improve our work.

We would like to bring to your attention that we have made reference to several new figures that were generated as part of our response. These additional visual aids can be found in the supplementary file that we have uploaded, and you have access to this supplementary material.

Reviewer Comments and Responses:

Q1: Maybe I've missed it, but did you measure the decoding accuracy (Decoder : ECoG -> Words). Before we we ask if we can reconstruct those representations it would be nice to know that they actually carry information about the words and are not just unrelated neural processes. Without this, from Reconstruction : Embeddings -> Signal we know that we can reconstruct and capture the temporal dynamics in the brain signal, but this does not necessarily mean that this dynamics is relevant to language processing (I understand that the area where the signal is from is a language area, but who knows what else it might be doing that is contributing to the signal we are so diligently are trying to reconstruct).

A1: We appreciate the reviewer's inquiry. Indeed, Goldstein et al. (2022) undertook decoding on this data set, demonstrating the feasibility of classifying the identity of words from the neural signal. In fact, they revealed that the contextual embedding induced by the final layer could be predicted from the neural signal and by comparing the predicted embedding it is possible to classify the identity of the words. Therefore, the reconstruction of the embedding indeed holds word-specific information. Given the opportunity we will cite these results and explain their importance in the interpretation of our results.

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

Q2: What happens with not-so-much-predictable words? Why?

A2: We thank the reviewer for this question. We realized that we did not refer to this result in the manuscript. These analyses can be found in Supp S4&S5).If given the opportunity we will refer to it in the revision. Interestingly, the temporal ordering of the layers is robust to the choice of predictable or unpredictable words, however it occurs later for unpredictable words (between 0 and 250ms for predictable and between 250ms and 500ms for unpredictable). Though we have not explored this effect in depth, it may indicate a neural response that occurs in response to prediction error.

Q3: How well would you be able to classify (using as deep and powerful model as needed) individual words from ECoG signal alone? In my mind (in)ability to do this would be informative in terms of whether the recorded activity actually contains strong enough signal to claim that it carries any bits of language representation in the brain.

A3: We thank the reviewer for the question. We believe our answer to A1 applies here as well but please let us know if there is something we missed and we will happily address it.

Q4: What is the canonic path of a word through the brain? Which areas are being activated in which order according to the modern day knowledge on this? The reader would benefit of a figure explaining this, same as we've seen in all the vision papers.

A4: To the best of our knowledge, a definitive neural architecture for word processing does not currently exist. Furthermore, contextual embeddings are not limited solely to representing the identity of a word, but also consider its meaning in relation to its context. However, in a study conducted by Hasson, Chen, and Honey (2015), they propose a model of language comprehension and provide a diagram in Figure 2B (right) that illustrates the flow of information. Interestingly, this diagram demonstrates a similar pathway to the one we describe in our paper (mSTG->aSTG->TP, IFG). Additionally, further supporting our claims is the observation that the order of regions of interest (ROIs) in this pathway correlates with the temporal receptive field, which we measure in our manuscript through the spread of the peaks. Hasson, U., Chen, J., & Honey, C. J. (2015). Hierarchical process memory: memory as an integral component of information processing. Trends in cognitive sciences, 19(6), 304-313.

We continue in the next comment.

We are deeply appreciative of the time and attention you have dedicated to thoroughly reviewing our paper, and for your insightful remarks. We have responded to the comments on a point-by-point basis. We sincerely apologize for our inability to upload our response at an earlier time, which would have allowed for a more timely discussion. This delay was largely due to the substantial number of reviewers (5) for our work, and the time it took us to consolidate our responses to all their comments. We highly value your understanding regarding this matter and remain committed to considering and addressing all pertinent feedback to improve our work.

We would like to bring to your attention that we have made reference to several new figures that were generated as part of our response. These additional visual aids can be found in the supplementary material section of openreview

Reviewer Comments and Responses:

Q1: Current work focuses more on layer-wise transformations learned by GPT2-XL map onto the temporal sequence of transformations of natural language. However, the current paper lacks in providing fine-grained details as follows: (i) Why are intermediate layers well aligned with the Brain?(ii) It could be interesting if authors could report more fine-grained details like the following study: Subba Reddy Oota, Mariya Toneva. Joint processing of linguistic properties in brains and language models, NeurIPS-2023, https://arxiv.org/pdf/2212.08094.pdf (iii) Supraword meaning analysis and layer-wise hierarchy details.

A1: We thank the reviewer for this comment. Previous work noted the higher alignment between intermediate layers and the brain during language processing (Toneva and Wehbe 2019 and Caucheteux and King 2022, for example). We also replicated this result as can be shown in Figures 2 and 5, however, the focus of our paper is different - our primary finding revolves around the correspondence between temporal dynamics in the brain and the layered structure (or sequence of embeddings) of GPT2. We demonstrate that the deeper the layer is, the later the embedding it induces correlates with the brain. This suggests a shared dynamic between the brain and GPT-2 computations or representaions. For greater clarity, we have scaled the encodings to 1 in plots 2 and 3, thereby emphasizing the importance of temporal dynamics rather than focusing on determining which layer induces the optimal correlation with the brain.

Caucheteux, C., King, JR. Brains and algorithms partially converge in natural language processing. Commun Biol 5, 134 (2022). https://doi.org/10.1038/s42003-022-03036-1 Toneva, Mariya, Leila, Wehbe. "Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)." Advances in Neural Information Processing Systems. 2019.

Q2: Caucheteux et al. 2022 "Brains and algorithms partially converge in natural language processing, shown that both MLM and CLM models yield best alignment with fMRI & MEG in middle layers and reported better performance at predicting next word -> better prediction of fMRI & MEG. How does the current work differ in terms of new findings except for the use of ECoG recordings?

A2: We thank the reviewer for the question. Indeed, Caucheteux et al. (2022), also demonstrates correlation with embedding induced by deep language models and the brain. However, here we focus on the shared dynamic between the brain and the flow of information through the GPT-2 layers (see A1 for more details).

Q3: It could be interesting if authors could report syntax and low-level surface feature analysis across language regions.

A3: We fully acknowledge the intriguing nature of assessing the predictive ability of LLM-induced embeddings in comparison to symbolic language representations such as syntax and part-of-speech (PoS). In fact, we are actively engaged in the writing of a manuscript focused specifically on this topic. However, in the context of the present manuscript, we believe that diverting the discussion towards a comparison with symbolic representations would deviate from its primary scope and purpose.

Q4: Did authors try the encoding with different context lengths?

A4: We thank the reviewer for this question. In response to this request we generated embeddings by running GPT2-XL on the podcast data with input sizes of 500 tokens and 100 tokens. We then re-ran our analysis for correct words in the inferior frontal gyrus (IFG). The global trends are robust to different context lengths during embedding generation (see in Supp S2). If given the opportunity we will report this analysis in the manuscript.

We continue our response in the following comment.

We are deeply appreciative of the time and attention you have dedicated to thoroughly reviewing our paper, and for your insightful remarks. We sincerely apologize for our inability to upload our response at an earlier time, which would have allowed for a more timely discussion. This delay was largely due to the substantial number of reviewers (5) of our work, and the time it took us to run new analyses and to consolidate our responses to all their comments. We highly value your understanding regarding this matter and remain committed to considering and addressing all pertinent feedback to improve our work.

We would like to bring to your attention that we have made reference to several new figures that were generated as part of our response. These additional visual aids can be found in the supplementary material section of openreview.

Reviewer Comments and Responses:

Q1: Lack of comparative baselines: The paper only uses GPT2-XL, and other language models such as BERT, as well as recently released LLMs (e.g., LLaMA, Vicuna) should be considered for comparison.

A1: We appreciate the comment offered by the reviewer. Three foundational studies establish a connection between language comprehension in the brain and GPT-2 (Goldstein et. al 2022, Caucheteux et al. 2022, Schrimpf et al 2021). In particular, two of these papers (Goldstein et. al 2022, Schrimpf et al 2021) explicitly discuss the selection of autoregressive language models (such as GPT-2) as particularly suitable for modeling brain responses to natural language stimuli as the brain is known to make predictions about subsequent words before they are spoken On the contrary, the BERT model is bidirectional and less suitable for modeling the cognitive system, which does not have access to the future words at prediction time. However, LLAMA is autoregressive, fitting the required model criteria. We are currently generating word embeddings from LLAMA and will re-run our analysis. This process takes some time as we need to get familiar with a new tokenizer and other technical details. If given the opportunity we will add the LLAMA based results. In addition, we will include the rationale we have outlined here in the manuscript for choosing GPT-2 for modeling the process of language comprehension in the brain. Following the reviewer's request, we also performed an analysis for BERT embeddings. However, it is important to note that there are several considerations that arise when generating an embedding from BERT, considerations that are not present with GPT2. Firstly, BERT is designed to train on complete sentences. Given that spoken language doesn't always present in full sentences, we were required to manually intervene in punctuation (aided by the nltk package). Nevertheless, this implies that the word embeddings carry information from future words. From a cognitive viewpoint, this constitutes an entirely different model. We extracted embeddings from BERT-Large’s 24 layers for predictable words. The mSTG, IFG and aSTG did not have significant results (p>0.5, p> 0.1, p>0.7). Neither the IFG nor the aSTG show the effect of increasing peak encoding correlation time with the layer index that we see with GPT2-XL embeddings. In the TP we get a significant effect (p<0.03) of a negative correlation between layer index and lag, which is unexpected. As pointed out before, there are multiple aspects that need to be considered when using BERT to model neural signal. Thus a further investigation is needed to understand these results. The BERT based plots can be seen in Supp table S1.

Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., ... & Hasson, U. (2022). Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3), 369-380.

We thank the reviewer for their suggestion to repeat our analyses using embeddings from different language models. We generated embeddings from the 33 layers of LLaMA 7b using the podcast words. The results can be seen in Supp Fig. S13. We get significant correlations for the lag layer result for the aSTG, IFG and TP (r=0.587, r=0.825, r=0.639; p< 10e-3, p<10e-8, p<10e-4). We get a similar range of times of peak encoding correlation for these three ROIs as well (0-300ms for IFG and TP, -50ms to 100ms for aSTG). The layers all peak around 0ms in the mSTG which also reproduces our results. Thus, our results were replicated for LLaMA 7b, but not for BERT, further strengthening our claim that autoregressive deep language models are powerful cognitive models of language comprehension in the human brain.