Hierarchical Frequency Tagging Probe (HFTP): A Unified Approach to Investigate Syntactic Structure Representations in Large Language Models and the Human Brain

Thank you for your detailed evaluation and thoughtful recommendations. Your feedback has significantly contributed to the refinement of our study. We respond to all of your points in the following part.

Q1: Reconcile strong sentence–phrase layer correlations with prior layer-specialisation results of Tenney et al. (2019).

A1: Our correlation shows that when a layer devotes many units to phrase rhythms it also hosts sentence rhythms, revealing a shared hierarchical scaffold. Tenney et al. locate the depths where supervised edge‑probes become linearly decodable, yielding an early‑vs‑late split. Co‑location of phrase‑ and sentence‑responsive neurons does not contradict that split; it means both levels ride the same representational backbone before different linguistic functions are teased apart by linear probes.

Q2: Current interpretation of model-specific layer distributions is “unsatisfying” and needs deeper analysis (e.g., probing).

A2: To identify the linguistic operation performed by the phrase‑ and sentence‑selective neurons that appear in widely separated layers, we implemented a structure‑probe analysis closely following Hewitt & Manning (2019). For every model layer we first isolated the HFTP‑flagged phrase and sentence neurons, then fed the original stimuli forward and recorded their activations for every token.  For each layer ℓ we recorded activations, built a cosine‑distance matrix $D_ ℓ$, and compared it (Spearman ρ) with two reference RDMs: one marking phrase boundaries, the other sentence boundaries. Averaging across trials yields a layer‑wise syntactic profile.

Results are similar to Figure 4.GPT‑2 peaks for sentence and phrase structure in early-mid layers and declines in last few layers. Llama‑2‑7B shows modest phrase alignment but strong sentence alignment rising in the final third of the network, consistent with late compositional integration. Llama‑3.1‑8B maintains consistently high alignment for both phrase and sentence structures in mid-to-late layers. Thus the “distant” selective layers reflect genuine hierarchical syntactic representations, not generic linguistic sensitivity, and our probe robustly maps each model’s preferred depth for phrase‑ and sentence‑level processing.

Q3: Why use cosine-distance SRDMs instead of simply comparing 1 & 2 Hz peak amplitudes?

A3: Cosine-distance SRDMs encode the full spectral geometry of each condition, making the metric immune to baseline shifts or local noise spikes that often distort single-peak measures. In pilot tests, peak-only scores sometimes failed to detect true model–brain matches when slight frequency drifts occurred, whereas SRDM-based RSA consistently retrieved the same top layers and channels. SRDMs therefore deliver a more robust and sensitive similarity estimate without adding methodological complexity.

Q4: Explain exactly how SRDMs are computed (which neurons, averaging procedure, etc.).

A4: For each layer ℓ we isolate phrase-only, sentence-only, and dual-responsive neurons, obtain their activations for every stimulus, and build three SRDMs based on phrase, sentence, dual neurons by taking cosine distances between FFT spectra of all stimulus pairs. Each sEEG channel receives an analogous SRDM. We correlate each model SRDM with every channel SRDM (Spearman ρ), average the top 100 ρ values to yield the layer-brain similarity for that neuron group, and then average across layers to obtain three model–brain scores. The mean of these three scores is reported as $S(m,b)$.

Q5: A1 emerges as best-predicted region in Table 1—contradicts prior ECoG studies of Goldstein et al. (2022)

A5: In fact, our study and Goldstein et al. (2022) address fundamentally different questions with distinct methods, so it is unsurprising that their peak regions diverge from ours. In reality, Goldstein et al. (2022) did not investigate syntactic processing; they focused on semantic next‑word prediction, showing that both the brain and GPT‑2 “match their pre‑onset predictions to the incoming word”. They model high‑γ (70–200 Hz) ECoG power recorded from surface grids and strips that do not sample the core of Heschl’s gyrus (A1) because subdural electrodes cannot reach that medial sulcal cortex. When contextual embeddings are added, they highlight additional fits in frontal/temporal‑pole areas, but early auditory contacts remain in the significant set.

Our work, by contrast, targets hierarchical syntactic rhythms (0.5–2 Hz) and uses depth‑SEEG shafts that traverse A1, giving much denser coverage and allowing us to measure phase‑locked low‑frequency responses that originate in early auditory cortex. Because syntax‑driven rhythmic entrainment is strongest at the auditory core, A1 naturally emerges as the best‑predicted ROI. Hence our findings complement Goldstein’s semantic‑frontal effects rather than contradict them.

Q6: Provide error bars and control for electrode-placement confounds (suggest mixed-effects analysis).

A6:We implemented two complementary analyses exactly as suggested:

1. Bootstrapped 95% confidence intervals
   • For every model‑layer we averaged the top‑100 channel correlations, then bootstrapped (10 000 resamples of channels × layers) to obtain mean ± 95 % CI.
   • For each brain region the same procedure was applied across layers, and the three syntactic neuron types were then averaged to give a single region‑level value.
2. Mixed‑effects regression to account for the confound that different subjects contribute different sets of electrodes.
   • Model: Alignment ~ ROI + (1 | Subject)  (random intercept per subject).
   • Significance therefore reflects within‑subject ROI effects, while the group‑level intercept absorbs between‑subject signal‑quality differences.
   • We additionally report per‑subject OLS tables (four examples shown below) to demonstrate that effects are not driven by a single individual.

Table S1: Llama 2 | Sentence condition — Bootstrap 95 % CI

Tabls S2: Mixed‑effects highlights († p < 0.01 vs baseline A1)

σ_subj is the random‑intercept SD; its small value confirms residual inter‑subject variance is minimal after modelling ROI.

Table S3: Per‑subject confirmation (two richest grids per side)

In conclusion, bootstrap CIs verify stable estimates. Mixed‑effects results reveal reliable within‑subject ROI shifts—left IFG/STG up, right Amygdala/Insula/ Temporal Pole up—with negligible residual subject variance. Agreement in the two best‑covered subjects per hemisphere confirms these patterns are genuine and not due to electrode placement.

Q7: Why not analyze at the syllable level (4 Hz) as done in Ding et al. (2016).

A7: Our analysis does not directly operate at the syllable level for two main reasons. First, when we artificially add a temporal axis to LLM activations and set the sampling rate to 4 Hz as sEEG recording did, the Nyquist–Shannon sampling theorem dictates that the highest resolvable frequency is 2 Hz. This means that, to match the sEEG experiment’s playback rate, the model cannot resolve syllable‑level peaks. Second, the syllable level represents the most basic linguistic unit, lacking the hierarchical syntactic integration present at the phrase or sentence levels. Therefore, we focus on these higher-order levels to better assess the syntactic representations in LLMs, which aligns with our central research questions about structural hierarchy rather than basic phonological rhythms.

Q8: Clarify meaning of bold numbers in Table 1 and why no second constraint on electrode responses.

A8: We apologize for not providing a clear explanation in the main text. For the model–brain similarity row $S(m,b)$, bold numbers indicate the model with the highest syntactic alignment with human brain data (GPT‑2 in Table 1). For the model–region similarity rows (each ROI listed in the first column), bold numbers highlight the top three ROIs where a given model shows the strongest alignment. This ranking combines both left and right hemispheres to reveal potential lateralization patterns—that is, whether a model’s syntactic processing mirrors the human brain’s known left-dominant language regions.

In LLMs, z‑scores normalize over thousands of neurons; sEEG channels number only in the low hundreds and exhibit high baseline variability. A second z‑score step would disproportionately penalize true syntactic channel responses. Our strict permutation pipeline therefore matches specificity across modalities.

Thank you for your encouraging follow-up. We’re delighted that the additional analyses and the naturalistic English experiment addressed your concerns. Your feedback has been invaluable in strengthening our paper, and we sincerely appreciate your recognition of our work : )

We appreciate your detailed and constructive suggestions. Below, we address each of your concerns one by one.

Q1: 4 Hz sampling constraint seems unmotivated.

A1: The 4 Hz grid is not meant to “inject” time into the model; it is simply the clock that matches the sEEG experiment, where syllables are played at 4 Hz and the target phrase‑ and sentence‑beats therefore occur at 2 Hz and 1 Hz. If the stimuli were paced at 6 Hz or 8 Hz we would re‑sample activations at the same rate, and those beats would shift proportionally (e.g., 8 Hz → 2 Hz and 4 Hz peaks). In other words, the sampling rate is an external synchronisation parameter that lets model and brain be analysed with a shared FFT, without altering the linguistic content inside the network. We also applied HFTP to naturalistic corpora in Q4, showing that our method generalizes beyond rhythmic input to both Chinese and English naturalistic stimuli. Consequently, the sampling rate serves only as a temporal granularity parameter—regardless of its value, HFTP consistently recovers the underlying sentence- and phrase-level rhythms.

Q2: Very high sentence–phrase correlations suggest probe may detect general linguistic sensitivity, not distinct syntax levels.

A2: The strong count correlation simply reflects the hierarchical nature of language: every sentence contains phrases, so layers that allocate more capacity to phrase rhythms inevitably allocate more to sentence rhythms as well. Importantly, the neurons themselves remain selective—phrase‑tagged units peak only at 2 Hz, sentence‑tagged units only at 1 Hz—showing two non‑overlapping spectral signatures rather than a single, undifferentiated “language” response.
To identify the linguistic operation performed by the phrase‑ and sentence‑selective neurons, we implemented a structure‑probe analysis closely following Hewitt & Manning (2019). For every model layer we first isolated the HFTP‑flagged phrase and sentence neurons, then fed the original stimuli forward and recorded their activations for every token.  For each layer ℓ we recorded activations, built a cosine‑distance matrix $D_ ℓ$, and compared it (Spearman ρ) with two reference RDMs: one marking phrase boundaries, the other sentence boundaries. Averaging across trials yields a layer‑wise syntactic profile. Results are similar to Figure 4.GPT‑2 peaks for sentence and phrase structure in early-mid layers and declines in last few layers. Llama‑2‑7B shows modest phrase alignment but strong sentence alignment rising in the final third of the network, consistent with late compositional integration. Llama‑3.1‑8B maintains consistently high alignment for both phrase and sentence structures in mid-to-late layers. Thus the detected syntactic neurons reflect genuine hierarchical syntactic representations, not generic linguistic sensitivity, and our probe robustly maps each model’s preferred depth for phrase‑ and sentence‑level processing.

Q3: Generalizability limited by Chinese-only sEEG; need cross-lingual validation.

A3: We have already collected a bilingual sEEG session from one native‑Chinese participant who also understands everyday English. Using the four‑word English (e.g., Old ox plows field, A friend invites guests) and four‑character Chinese (e.g., 老牛耕地，朋友请客) stimuli, we observed identical spectral signatures—robust peaks at 1 Hz (sentence) and 2 Hz (phrase)—for both Chinese and English inputs.
For English speakers who also understand Chinese, we likewise anticipate dual peaks at 1 Hz and 2 Hz. Hierarchical chunking is driven by universal constraints—working-memory limits and predictive parsing mechanisms—rather than by language-specific lexical cues. Hence cross-linguistic factors such as function-word density may shift power, but they do not eliminate the peaks themselves. This envision also aligns with Ding et al. (2016), which showed that the 1–2 Hz hierarchy is a language-general signature of syntactic grouping, independent of word-order typology or writing system.
Our bilingual model results reinforce this view: Appendix C, Fig. 7 show both language‑specific and language‑general syntactic neurons within a single LLM. This finding is especially informative because, although Tang et al. (2024) also identified language-specific neurons, they did not investigate their functional roles—syntactic, semantic, or otherwise. Consequently, these results lead us to hypothesize that the human language network contains both language‑specific channels, tuned to idiosyncratic phonological or orthographic statistics, and language‑general channels that track abstract hierarchical structure independently of the surface code. Such a division is neuro‑computationally efficient: language‑general pathways support rapid transfer of syntactic skills across languages, while language‑specific pathways fine‑tune predictions to each lexicon, minimizing processing cost and error.

Q4: Request evaluation on complex syntactic phenomena.

A4: We have included the Chinese natural‑data HFTP experiment in Appendix E of the main text, with frequency‑domain results shown in Figures 8–9 and corpus details in Tables 6–7. To confirm generalizability, we also ran an English natural‑text experiment using balanced sets of eight‑ and nine‑word sentences (50 per set) drawn from everyday dialogue, news reports, literary prose and so on—mirroring the genre diversity of the Chinese stimuli.

The frequency‑domain patterns are consistent across both Chinese and English naturalistic datasets:

• Eight‑word set: Robust peaks appear at 0.50 Hz, 1.00 Hz, 1.50 Hz, 2.00 Hz, corresponding to the sentence envelope, canonical 4‑word phrases, intermediate 2‑to‑3‑word groupings, and the dominant 2‑word lexical rhythm.
• Nine‑word set: Peaks shift to 0.44 Hz, 0.89 Hz, 1.33 Hz, 1.78 Hz: the lowest peak reflects the whole clause, 1.33 Hz aligns with frequent 3‑word phrases, 1.78 Hz captures rapid two‑to‑three‑word alternations, and 0.89 Hz indexes a four‑to‑five‑word prosodic “breath group”.

The bilingual natural‑text study shows that HFTP scales from controlled four‑syllable strings to unrestricted prose: for both Chinese and English, we recover a clean hierarchy of clause, phrase, and lexical rhythms even when sentence length and genre vary. This demonstrates that the probe is not limited to simple structures but generalises to the richer combinatorics of everyday language.
For long‑distance dependencies (e.g., wh‑movement over multiple clauses) we predict an additional, lower‑frequency peak (< 0.5 Hz) reflecting the longer integration window; all higher‑level peaks should remain, merely compressed along the frequency axis. For garden‑path sentences the initial mis‑parse and subsequent reanalysis should appear as two successive bursts in the same frequency band (first at the garden‑path interpretation, then at the corrected structure), a pattern we can time‑lock in sliding‑window FFTs. For recursive embeddings we expect a nested harmonic series: each new level of recursion introduces a further sub‑harmonic (½, ⅓, ¼ × the phrase rate) while preserving the base phrase and lexical peaks. Because HFTP reads frequency rather than word position, these phenomena require only appropriate presentation rates (longer stimuli → lower sampling) to fall within the analyzable band.

Q5: Test on non-syntactic controls for specificity.

A5: Yes. In the main text, we ran random‑label control in which all words were permuted so that sentence and phrase structure—and most semantics—were destroyed while lexical statistics were preserved. Across models this control produced flat spectra with no 1 Hz/2 Hz peaks (see Figure 3, 5, 8, 9, “random” labels). We also include a phrase‑only data (phrase corpus in the main text) that preserves two‑word phrase boundaries but removes sentence structure. FFT results show a strong 2 Hz peak while the 1 Hz sentence peak disappears. Conversely, the random‑label control flattens both peaks. Together these manipulations demonstrate that HFTP responds precisely to the syntactic boundary present—phrase or sentence—rather than to general lexical, phonological, or semantic patterns.

Q6: Explain why should FFT expose syntax in LLMs, and how heavy is the method.

A6: Hierarchical syntax produces nested periodicities whenever tokens are serialized—clauses recur slower than phrases, phrases slower than lexical pairs. The transformer’s feed‑forward clock preserves this rhythm in its hidden activations; applying an FFT simply makes the nested rates explicit. For all models, we performed activation extraction in float16 precision on two NVIDIA A100 GPUs (40 GB each), with end‑to‑end runtimes spanning from tens of seconds to a few minutes depending on model size.

Thank you very much for your follow-up and helpful suggestions. We will incorporate all additional results (e.g., the English natural-text HFTP analysis and our preliminary cross-lingual sEEG findings) into the final manuscript.

We greatly appreciate your recognition of our work : )

We sincerely appreciate the constructive feedback and the effort you invested in evaluating our manuscript. We respond to every your concern in the following paragraphs.

Q1: Why analyze only hidden MLP neurons? Consider layer outputs as well.

A1: Thank you for pointing this out. We have tested the HFTP pipeline on layer outputs (i.e. hidden states). Below are the hidden‑state results for three representative models; values for MLP neurons (reported in the main paper) are systematically higher, confirming their stronger brain alignment.

Feed-forward blocks function as key–value stores (Geva et al. (2021)): the first linear projection expands the signal so individual units can store disentangled lexical, syntactic, or factual features; the second projection then recombines these features back to d_model (i.e. hidden states). Dai et al. (2022) shows that most “knowledge neurons” live in this expanded space. Probing with HFTP at this inner-MLP stage keeps phrase- and sentence-selective units distinct, producing sharper 1 Hz/2 Hz peaks and higher model–brain alignment. Running the same analysis on the compressed hidden states blurs these rhythms and lowers scores. We therefore prioritise inner-MLP neurons, listing hidden-state results only for completeness.

Geva, et al. "Transformer Feed-Forward Layers Are Key-Value Memories." EMNLP. 2021.
Dai, et al. "Knowledge Neurons in Pretrained Transformers." ACL. 2022.

Q2: Evidence for “seamless” generalization to naturalistic text is weak and limited to a small Chinese appendix.

A2: Thank you for pointing this out. We are sorry for the misleading claims that our method generalizes "seamlessly to naturalistic text". We have now added English natural‑text experiments as follows, which reveal the same hierarchical peaks as in Chinese. Therefore, the more precise statement is that “HFTP generalizes across Chinese and English naturalistic datasets and is highly likely to extend to other character-centric scripts (e.g., Japanese, Korean) as well as further alphabetic languages (e.g., French, German).”

To complement the Chinese corpus, we conducted an English natural‑text experiment that uses balanced sets of eight‑ and nine‑word sentences (50 sentences per set). The materials span everyday dialogue, news reports, literary prose, and so on—mirroring the genre diversity of the Chinese stimuli.

Frequency-domain results

• Eight‑word set: Robust peaks appear at 0.50 Hz, 1.00 Hz, 1.50 Hz, 2.00 Hz, corresponding to the sentence envelope, canonical 4‑word phrases, intermediate 2‑to‑3‑word groupings, and the dominant 2‑word lexical rhythm.
• Nine‑word set: Peaks shift to 0.44 Hz, 0.89 Hz, 1.33 Hz, 1.78 Hz: the lowest peak reflects the whole clause, 1.33 Hz aligns with frequent 3‑word phrases, 1.78 Hz captures rapid two‑to‑three‑word alternations, and 0.89 Hz indexes a four‑to‑five‑word prosodic “breath group”.

This pattern mirrors the hierarchical frequency profile found in the Chinese naturalistic experiment (see Figure 8&9 in the main text), demonstrating that HFTP reliably tracks sentence-, phrase‑, and clause‑level rhythms in typologically-distinct languages. The English naturalistic results demonstrate that HFTP scales smoothly from Chinese to an alphabetic writing system. Because the method operates at the rhythm unit that is natural for each script—characters in non-segmented CJK texts, words or sub-words in space-delimited alphabets—we are confident it will likewise generalize to other character-centric (e.g., Japanese, Korean) and alphabetic(e.g., French, German) naturalistic data.

Q3: GPT-2 is actually a Chinese-trained variant; this must be stated prominently in the main text.

A3: We apologise for not stating this at the outset and for any confusion it caused, especially given GPT‑2’s high alignment score. In the revised manuscript we will explicitly note, on first mention and again in the Results, that our GPT‑2 variant is pre‑trained on a Chinese corpus because the original English GPT‑2 cannot process Chinese text.

Q4: Handling of multi-token words

A4: We explicitly normalised tokenisation across models. Each LLM employs a different sub‑word scheme; most English‑trained models split a single word into multiple tokens (e.g., Llama‑2‑7B tokenises “A friend invites guests” as ▁A, ▁friend, ▁inv, ites, ▁guests). Because HFTP treats one word as the minimal semantic unit driving the 4‑word/8‑word rhythmic structure, we aggregate all sub‑tokens that belong to the same word: their hidden activations are averaged to yield a single word‑level vector before the FFT. This ensures that amplitude at 1 Hz (sentence) and 2 Hz (phrase) reflects linguistic—not tokeniser—boundaries.
We apply the same normalization to Chinese. Although all six models support Chinese, their tokenizers differ: some (e.g., Gemma, Llama) split a single character into multiple tokens, whereas others (e.g., GLM‑4) merge several characters. To enforce a one‑character‑per‑token mapping, we insert explicit delimiters—transforming “朋友请客”into “朋*友*请*客*”—and then average the hidden activations of any sub‑tokens belonging to the same character. This ensures that each Chinese character (and each English word) produces exactly one activation trace, making the subsequent frequency‑tagging fully comparable across languages and models.

Q5: Prior work is said to have “methodological inconsistencies” when exploring syntactic representations between LLMs and the human brain without specifying what they are.

A5: We apologise for the ambiguity. In our text, “methodological inconsistencies” refers to the heterogeneous ways prior studies probe syntax in brains versus LLMs, which makes cross‑study synthesis difficult:

• Signal source varies. Sun et al. (2020) map whole‑sentence embeddings to fMRI; Caucheteux & König (2021, 2022) first project hidden states into syntactic/semantic subspaces; Wang et al. (2020) use variational disentanglement; Oota et al. (2023) ablates tree‑depth features.
• Neural coverage is uneven. Data range from single‑region ECoG to ROI‑level MEG to whole‑brain fMRI, meaning the same syntactic signal may be sampled in one study but absent in another.

Model‑side and brain‑side probes have remained system‑specific—LLM syntax is probed via structural methods (e.g., Hewitt & Manning 2019), while human syntax relies on experimental modalities (fMRI, MEG, ECoG) that sample limited cortical regions—making direct comparisons impossible. Thus HFTP resolves this by offering a single frequency‑domain probe that treats MLP neurons and cortical channels identically, removing probe asymmetry and unifying the evaluation metric for model‑agnostic, systematic study of hierarchical syntax encoding across both systems.

Q6: Incorrect Gemma-2 feed-forward dimension (should be 28 672) need correction.

A6: Thank your for pointing this out. The apparent mismatch arises from two counting conventions. In the released HuggingFace checkpoint, Gemma 2 9B has a model width of 3 584 and an intermediate size of 14 336. Because Gemma 2 uses a GeGLU feed‑forward block, the intermediate vector is produced by two parallel projections of size 14 336 each (“gate” and “up”) that are combined element‑wise before being projected back to 3 584. The Gemma 2 paper reports the effective expanded width—the concatenation of both streams—which is 2 × 14 336 = 28 672. For our neuron counts we follow the transformer‑config convention and list the per‑projection dimension (14 336), as this represents the actual number of activations entering the non‑linearity. Hence the value in Table 3 is correct, while the paper’s larger figure simply reflects the doubled size created by GeGLU’s two paths.

Q7: Release plans of sEEG corpus, and bold-font rationale in Table 1.

A7: Yes, we plan to release the sEEG corpus upon obtaining formal permission from the authors of Sheng et al. (2019), as our sEEG data originated from that study. We apologize for not providing a clear explanation in the main text. For the model–brain similarity row $S(m,b)$, bold numbers indicate the model with the highest syntactic alignment with human brain data (GPT‑2 in Table 1). For the model–region similarity rows (each ROI listed in the first column), bold numbers highlight the top three ROIs where a given model shows the strongest alignment. This ranking combines both left and right hemispheres to reveal potential lateralization patterns—that is, whether a model’s syntactic processing mirrors the human brain’s known left-dominant language regions.

Thank you for your thoughtful follow-up and additional suggestions. We address each concern below and outline the exact revisions we will make in the final manuscript.

Model-to-model alignment variability

We share your view that these differences deserve deeper treatment. Recent work shows that brain–model alignment is not strictly monotonic with parameter count, architectural novelty, or leaderboard score—gains rise early but plateau or even fall as models outgrow language-relevant representations (Hong et al., 2024; AlKhamissi et al., 2025). Alignment is boosted when training objectives force models to internalise discourse-level structure humans actually use (Aw & Toneva, 2023), and the seeming “bigger is better” pattern often vanishes once embedding dimensionality is controlled (AlKhamissi et al., 2025) or trivial confounds such as sentence length and position are removed (Feghhi et al., 2024), confirming that raw scale alone is a poor indicator of neural fit.

On this basis, we hypothesise that Gemma 2 surpasses Gemma because knowledge-distillation and widened FFN (MLP) blocks strengthen mid-layer syntactic cues, whereas Llama 3.1—optimised for extreme context length, multilingual breadth, and tool use—reallocates capacity toward global reasoning chains, leaving fewer neurons to encode discrete 1 Hz/2 Hz rhythms, hence its lower alignment relative to Llama 2. Therefore, we believe that the key drivers of brain–model syntactic similarity are (i) the extent to which a model’s training tasks make it keep track of hierarchical syntax, and (ii) whether the FFN layer allocates its neurons to clean, sparsely coded syntactic features rather than to mixed, multi-purpose signals. We will weave this clearer discussion of mixed trends into the revised manuscript.

Reference:
Hong et al. Scale matters: Large language models with billions (rather than millions) of parameters better match neural representations of natural language[J]. BioRxiv, 2024.

AlKhamissi et al. From language to cognition: How llms outgrow the human language network[J]. arXiv:2503.01830, 2025.

Aw & Toneva Aw K L, Toneva M. Training language models to summarize narratives improves brain alignment[J]. arXiv:2212.10898, 2022.

Feghhi et al. What are large language models mapping to in the brain? a case against over-reliance on brain scores[J]. arXiv:2406.01538, 2024.

Dimensionality-controlled inner-MLP validation & Motivation for MLP usage

To rule out a dimension-only explanation, we ran two controls: (i) in the MLP we randomly subsampled the same number of syntactic neurons found in the hidden state (seed = 42), and (ii) in the hidden state we sampled an equal-sized random set. Even with identical neuron counts, the reduced-MLP still outperformed hidden states by 5–7 %, confirming that alignment stems from the disentangled nature of MLP features, not mere width : )

Besides Geva et al. (2021) and Dai et al. (2022), who showed feed-forward layers house lexical–syntactic concept neurons and “knowledge neurons,” later work shows per-token MLP activations remain clean, whereas projecting them into the residual stream mixes features through superposition (Elhage et al., 2022). Follow-up studies link FFN updates to interpretable lexical–syntactic features (Geva et al., 2022), locate confidence-regulation units mainly in MLPs (Stolfo et al., 2024), and show that widening or editing these layers yields outsized performance shifts (Gerber, 2025). Our HFTP analyses also echo this: phrase- and sentence-level frequency tags localise most strongly to MLP neurons and align with distinct cortical sites, while hidden-state spectra are noisier.

Reference:
Elhage et al. Toy models of superposition[J]. arXiv:2209.10652, 2022

Geva et al. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space[J]. arXiv:2203.14680, 2022.

Stolfo et al. Confidence regulation neurons in language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 125019-125049.

Gerber I. Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models[J]. arXiv:2505.06633, 2025.

Tokenisation protocol and table correction

As you requested, the final manuscript will explicitly detail our word-level (English) and character-level (Chinese) tokenization procedure. We also apologise for our oversight: in Table 1 the bold numbers for Llama 3.1 should highlight left-hemisphere ROIs Hippocampus (0.534), MTG (0.521), Insula (0.518), and TPJ (0.518); this will be corrected.

Once again, thank you for your time and constructive feedback. In the final manuscript we will integrate the naturalistic-English HFTP experiment, thoroughly discuss the mixed-trend findings, explain the motivation of inner-MLP probes, add tokenisation details, and correct the Llama 3.1 table entry. We are grateful for your recognition of our work : )

Thank you for your follow-up and for highlighting the cross-model differences in Figure 4. We fully agree these layer-wise shifts are intriguing and merit deeper study. We will endeavour to include, as much as possible, a concise note outlining our preliminary hypotheses (for example, wider MLP blocks concentrate syntactic coding in early layers, while long-context and tool-use objectives draw it to later layers; tokenization scheme and corpus genre can further shift peak locations), while stressing that a full analysis remains future work.

We appreciate your ongoing insights and engagement : )

Thank you for dedicating your time and expertise to review our work. Your thoughtful comments provide valuable guidance for strengthening the paper. We address each of your concerns in detail below.

Q1: Artificial 4 Hz time scale may impose non-intrinsic rhythm on LLMs.

A1: The 4 Hz sampling is a neutral reference grid that lets us read out latent hierarchical structure with the same FFT used on brain data. When the sEEG experiment presents syllables at 4 Hz, the model must be sampled at 4 Hz so that sentence and phrase rhythms align at 1 Hz and 2 Hz. If the same syllables were played at 8 Hz, we would resample model activations at 8 Hz and the corresponding peaks would appear at 2 Hz and 4 Hz. In other words, the temporal rate is an external clock that synchronises model and brain for comparability; it does not impose new linguistic content.
We also applied HFTP to naturalistic corpora in Q4, showing that our resampling procedure generalizes to both Chinese and English natural-text. Consequently, the sampling rate serves only as a temporal granularity parameter—regardless of its value, HFTP consistently recovers the underlying sentence- and phrase-level rhythms.

Q2: Mixed / contradictory model-brain alignment trends hinder conclusions about architectural progress.

A2: Our data show that a model upgrade does not automatically move the network toward human‑like syntax processing: Gemma 2 aligns better with brain rhythms than Gemma, yet Llama 3.1 falls slightly below Llama 2, and GPT‑2 still scores highest overall. Indeed, we observed that upgraded models follow divergent patterns: an architecture’s advance does not guarantee more human‑like syntactic processing. Rather than assuming that any performance gain yields closer brain alignment, our results invite deeper inquiry into the determinants of syntactic similarity. These varying trajectories are, in fact, more illuminating—they highlight that model improvements can shift network resources away from rhythmic syntax encoding. These findings set the stage for principled engineering of future LLMs and cognitive robots that can be tuned either toward or away from brain‑like syntactic organisation, depending on application needs.

Q3: Tokenization across LLMs for English data lacks clear standardization.

A3: We explicitly normalised tokenisation across models. Each LLM employs a different sub‑word scheme; most English‑trained models split a single word into multiple tokens (e.g., Llama‑2‑7B tokenises “A friend invites guests” as ▁A, ▁friend, ▁inv, ites, ▁guests). Because HFTP treats one word as the minimal semantic unit driving the 4‑word/8‑word rhythmic structure, we aggregate all sub‑tokens that belong to the same word: their hidden activations are averaged to yield a single word‑level vector before the FFT. This ensures that amplitude at 1 Hz (sentence) and 2 Hz (phrase) reflects linguistic—not tokeniser—boundaries.
We apply the same normalization to Chinese. Although all six models support Chinese, their tokenizers differ: some (e.g., Gemma, Llama) split a single character into multiple tokens, whereas others (e.g., GLM‑4) merge several characters. To enforce a one‑character‑per‑token mapping, we insert explicit delimiters—transforming “朋友请客” into “朋友请客”—and then average the hidden activations of any sub‑tokens belonging to the same character. This ensures that each Chinese character (and each English word) produces exactly one activation trace, making the subsequent frequency‑tagging fully comparable across languages and models.

Q4: Artificial 4 Hz time needs justification for scaling to natural data.

A4: We have included the Chinese natural‑data scaling experiment in Appendix E of the main text, with frequency‑domain results shown in Figures 8–9 and corpus details in Tables 6–7. To confirm generalizability, we also ran an English natural‑text experiment using balanced sets of eight‑ and nine‑word sentences (50 per set) drawn from everyday dialogue, news reports, literary prose and so on—mirroring the genre diversity of the Chinese stimuli.

The frequency‑domain patterns are consistent across both Chinese and English naturalistic datasets:

• Eight‑word set: Robust peaks appear at 0.50 Hz, 1.00 Hz, 1.50 Hz, 2.00 Hz, corresponding to the sentence envelope, canonical 4‑word phrases, intermediate 2‑to‑3‑word groupings, and the dominant 2‑word lexical rhythm.
• Nine‑word set: Peaks shift to 0.44 Hz, 0.89 Hz, 1.33 Hz, 1.78 Hz: the lowest peak reflects the whole clause, 1.33 Hz aligns with frequent 3‑word phrases, 1.78 Hz captures rapid two‑to‑three‑word alternations, and 0.89 Hz indexes a four‑to‑five‑word prosodic “breath group”.

The bilingual results confirm that HFTP scales the artificial 4 Hz tagging to naturalistic corpora in both Chinese and English, proving its cross‑linguistic robustness. Because the underlying rhythmic structure is shared, the method is highly likely to generalize to other character‑centric languages (Japanese, Korean) and to space‑delimited alphabetic languages (French, German).

Q5: Generalizability of the Chinese‑based brain–LLM alignment to English‑speaking brains remains unsubstantiated.

A5: We have already collected a bilingual sEEG session from one native‑Chinese participant who also understands everyday English. Using the four‑word English (e.g., Old ox plows field, A friend invites guests) and four‑character Chinese (e.g., 老牛耕地，朋友请客) stimuli, we observed identical spectral signatures—robust peaks at 1 Hz (sentence) and 2 Hz (phrase)—for both Chinese and English inputs.
For English speakers who also understand Chinese, we likewise anticipate dual peaks at 1 Hz and 2 Hz. Hierarchical chunking is driven by universal constraints—working-memory limits and predictive parsing mechanisms—rather than by language-specific lexical cues. Hence cross-linguistic factors such as function-word density may shift power, but they do not eliminate the peaks themselves. This envision also aligns with Ding et al. (2016), which showed that the 1–2 Hz hierarchy is a language-general signature of syntactic grouping, independent of word-order typology or writing system.
Our bilingual model results reinforce this view: Appendix C, Fig. 7 in the main text show both language‑specific and language‑general syntactic neurons within a single LLM. This finding is especially informative because, although Tang et al. (2024) also identified language-specific neurons, they did not investigate their functional roles—syntactic, semantic, or otherwise. Consequently, these results lead us to hypothesize that the human language network contains both language‑specific channels, tuned to idiosyncratic phonological or orthographic statistics, and language‑general channels that track abstract hierarchical structure independently of the surface code. Such a division is neuro‑computationally efficient: language‑general pathways support rapid transfer of syntactic skills across languages, while language‑specific pathways fine‑tune predictions to each lexicon, minimizing processing cost and error.

Q6: Long‑term positive and negative societal impacts should be declared.

A6: Thank you for pointing the missing societal impacts out. HFTP offers a transparent, neuro‑inspired lens on LLM internals. Positive impacts include safer, more controllable models—by revealing which neurons drive hierarchical syntax we can design targeted safeguards and debiasing interventions; improved language‑disorder diagnostics—spectral markers could serve as non‑invasive probes of syntactic deficits; and richer human‑AI interfaces that align model parsing with human cognition. Negative impacts also merit consideration: the same interpretability could be weaponised to optimise persuasive text or amplify covert influence; detailed brain‑model correspondences might facilitate adversarial “neuro‑phishing” that mimics neural rhythms; and, if paired with personal neural data, fine‑grained model maps could erode privacy.

Thank you for your thoughtful follow-up and for clarifying your concern. We agree that our original sentence attributing Gemma 2’s higher alignment similarity solely to “architectural improvements and refined training” was imprecise given the mixed trends across models. We will delete that sentence and instead highlight the mixed results as motivation to probe which specific design choices truly foster human-like syntax representations.

As you requested, we will incorporate the new English natural-text HFTP results and our preliminary cross-lingual sEEG findings into the final manuscript, and we will add an explicit summary—earlier in the paper—of the cross-lingual model analyses currently in Appendix C. We will also include the previously missing societal-impact discussion in the Checklist.

We truly appreciate your constructive feedback and are grateful for your recognition of our work : )