LLM Layers Immediately Correct Each Other

We’re glad you enjoyed our work, finding our results “striking,” “novel and different from previous studies,” and “impressive!” We answer your questions below:

Did you subtract the dataset mean before computing the cosine similarity? I wonder if a non-zero mean could produce an offset in the cosine similarity.

Good question. We did not subtract the mean. However, we think:

• The distribution of cosine similarities (see: Figure 2b) is strongly bimodal, which we think is the strongest evidence that there are (at least) two modes of layer operation: TLCM and non-TLCM.
• We think it’s more interesting & surprising that the absolute cosine similarity sits strongly below zero, rather than the relative/z-scored cosine similarity is low. This is contrary to our expectation that contributions between layers should have positive cosine similarity (or at minimum be orthogonal), as we discuss in L. 99-105.

In Appendix A3 you provide cosine similarity histograms for the t1 != t2 case split up by layer. I would like to see the t1 = t2 histogram (Figure 2b) split up by layer as well, maybe in an Appendix. It would be interesting to know whether the TLCM mechanism is bimodal in individual layers.

Thanks for raising this! We included this exact plot in Figure 2b (observe the strong bimodality) [L. 127-130]. We realize this should be made more prominent, so we’ve updated Sec. 4.1 to discuss this plot more.

In line 273 you say that “The mechanisms that handle the latter two tasks must be non-trivial as the previous layer’s contribution is hidden inside the residual stream soup” – do the superposition hypothesis, and the observation that vectors in high-dimensional spaces are often nearly-orthogonal, affect the difficulty of this?

Interesting point. Yes. The superposition hypothesis suggests that the residual stream will be some linear combination of feature vectors. We think identifying and correcting the features specifically contributed by the previous layer is a difficult task, unless there is some commonality between those features. For example, the features are wrong. This directly leads us to the propose-and-reject hypothesis.

The proposed “propose-and-reject” hypothesis is only weakly supported

The propose-and-reject hypothesis states that a layer proposes a broad swath of possible “features,” while the following layer—equipped with an attention head to gather context—removes the features that are not relevant.

We offer the propose-and-reject hypothesis as a conceptual synthesis that is consistent with all our empirical results thus far. Specifically, the hypothesis explains:

• Quantitative contextual dependency experiments.
• TLCM corrects only the most recent layer, rather than correcting all prior layers
• TLCM’s prevalence in early layers, which are linked to “enrichment”
• Very low TLCM activations on beginning of sequence tokens
• Attention/MLP experiments which show that TLCM emerges when considering their combination (MLPs cannot access context without attention)
• The causal relationship we show—where the following layer linearly & dynamically attenuates the prior layer—in Sec 5.2.
• Sec 5.3’s Jacobian experiment demonstrating specific reinforcement and unit attenuation (\\lambda \= 1) of features

With this framing, we intend to generate future research directions, not to close the discussion.

In the propose & reject hypothesis you propose that the following layer uses context to correct the predictions of the previous layer. This makes sense in the first (few) layers, but by the midpoint of the model the amount of context available to different layers (number of preceding attention layers) seems quite close. Do you still think this is the reason for the TLCM?

Good point. This seems reasonable and likely explains why the prominence of TLCM decreases in the final two-thirds of model layers. Figure A5 illustrates this phenomenon across many models.

In section 5.2 you ask “is the entire layer scaled back uniformly or are certain subspaces targeted for correction?” – scaling the entire layer back seems like a simplistic alternative, of course that wouldn’t be happening. Are there other alternative hypotheses?

In Appendix A, we provide a theoretical rationale for why the model might fully “scale back” the previous layer. Essentially, we introduce the concept of LayerNorm blindness, which is the phenomenon that LayerNorm causes attention and MLP sublayers to be “blind” to the norm of the residual stream. Thus, they do not know how to size their contributions and may be biased to over-contribute so their contribution is not overshadowed. However, we find TLCM in models that do not have LayerNorm blindness, contradicting this explanation.

This fact (along with our Sec 5.2 Jacobian eigenbasis and Sec 4.3 experiments) strongly suggest that TLCM is responsible for attenuating a small subspace of the prior layer, leading us to the propose-and-reject hypothesis, in which layers are correcting incorrect features added by the prior layer.

Line 293 introduces the Jacobian J – what are the dimensions / indices of this Jacobian? Is it the embedding dimension? How are derivatives across-token positions handled?

We take the layer Jacobian $\\mathbf{J} \\in \\mathbb{R}^{d\_\\text{model} \\times d\_\\text{model}}$, which includes both the attention and MLP sublayers. We are taking the Jacobian with respect to the output of the prior layer, so the effects of other tokens are considered (linearly) in this Jacobian. In practice, this is materialized by computing the gradient of the layer's 4096 outputs, one at a time using PyTorch.

Section 5.2 is less clear than the rest of the paper, specifically how the Jacobian is computed is unclear (see below), and the proof is hard to follow.

Great point! We have moved the bulk of the proof's content to the appendix and increased its clarity. In its place, we've included details on how we compute the Jacobian and the intution for the eigenbasis approach.

In lines 334 to 355 the paper discusses TLCM explaining issues with SAEs and steering. I was not convinced by this that TLCM would explain the SAE and steering issues. (see “Questions” for suggestions)

Thanks for giving this thought. We're curious to understand your concerns with this explanation.

In lines 334 to 355 you discuss how TLCM could cause certain SAE failure modes. This part would be more convincing if you included experiments e.g. checking whether low-specificity SAE features line up with negative-eigenvalue-subspaces.

Good suggestion; we agree this would be valuable. Anthropic has predominantly completed the most substantial work with SAEs and high-quality labeled features. However, they release neither their model weights or learned features. We are excited about open-source SAEs with great natural language annotations, which would make experiments like these possible. For now, we see it as important future work.

We’re glad you thought our work was “novel and interesting” and conducts good “systematic studies!” Thanks for your questions; we address them below:

Figure 3 -- why is this done over 5 random tokens only? Can't it be done at a larger scale?

Thanks for raising this. We include full results for this experiment (total of 135 randomly sampled tokens across 18 layers) in Appendix E, Figures A7 and A8. We see that the results in Figure 3 are largely representative of the appendix figures.

Line 210-211. What does a TLCM 'activation' mean? This concept is not defined anywhere (or may be I missed it? In which case, it ought to be defined more saliently). I in general was not able to understand the y-axis of figure 2b(left).

Great question. We define a TLCM activation as any pair of adjacent layers whose contributions’ cosine similarity is less than -0.1. More formally, $\\operatorname{cossim}(\\mathbf{c}\_{i,t}, \\mathbf{c}\_{j,t}) \< \-0.1$ (L. 174-175). Note that an LLM with $n$ transformer layers can have up to $n-1$ activations. The y-axis of Figure 2b (left) is the average number of TLCM activations on the $i$th token across a corpus of 2,000 wikitext documents.

We’ve updated our wording in this section (and Sec 4.3) to explicitly define “TLCM activation,” to your suggestion.

Why does Qwen 0.5B has this phenomenon but not GPT-2 XL? Is dropout being present/absent the sole determinent of this phenomenon?

This remains an open question.

We have not seen TLCM on any model with dropout, which indicates that dropout could be involved. However, GPT-2 is also undertrained compared to modern models, being trained on at most 100B tokens. Observe in Figure A9 that the OLMo 1B models (similar size to GPT-2 XL) do not exhibit any traces of TLCM until being trained on 500B tokens. We’ve run a similar test on OLMo-2 7B and observe that TLCM starts to appear around 2T tokens. For additional context, Llama-3 8B was trained on 15T tokens!

It would be interesting to develop some sort of improvement to intervention vectors based on TLCM. Naively seems like adding intervention vectors at alternate layers might be the best strategy to overcome TLCM?

We agree this is an important area of future work. That’s an interesting approach: it seems like even if you add an intervention vector at layer $i$, our results indicate that you will get a correction at layer $i+1$. Although it’s possible if you add interventions at both layers, this correction breaks down. We’re interested in seeing how this plays out.

Line 162-163: "Appendix Figure A4 demonstrates that the mechanism’s strength increases markedly during the latter two-thirds of training, suggesting a systematic development pattern." I could not understand figure A4 from its caption. My best guess is that authors meant some other figure here (probably A9)? Or otherwise figure A4 needs to be explained better.

Thank you for catching this. You are right–we are referring to Figure A9. Because the paper and appendix were submitted separately at different times, it seems like we unintentionally compromised the correspondence of this figure in the paper and in the appendix. This is fixed for the camera-ready version.

Figure 2a is referenced much after 2b. Flip the order of the figure.

Good point; we’ve fixed this.

We’re happy to address any other questions you have.

Thank you for your thoughtful feedback! We respond to your questions below.

§4.3 only reports qualitative results, where, according to the description, quantitative evaluation was performed

Great clarification. Appendix G.2 contains the quantitative results underlying the findings in §4.3; we list the average number of TLCM activations across ~225 tokens. To highlight the statistical significance of this result, we’ve added the standard errors for each token (which are around ~0.2).

individual studies §4.2 and §4.3 are only done on a single model, making it difficult to understand the generalizability…

Thanks for this point. To address your suggestion, we’ve run both of these experiments on new models.

Due to limited checkpoint availability, we re-run our training experiment (4.2) on OLMo-2 7B, which features a new data mix and updated architecture. First signs of TLCM appear later, around 2T tokens, and TLCM solidifies around 3T tokens. OLMo-2 7B is trained on a total of 4T tokens. We’ve added these plots to our appendix.

Re-running experiment 4.3 on Gemma-2 2B Instruct yields highly consistent results. Below, we list tokens with the most and least average number of TLCM activations. The upper bound of the standard error for the following TLCM activation averages is 0.38, demonstrating strong statistical significance. Like the Llama models, the beginning of sequence token has the fewest TLCM activations by far.

10 lowest activating

challenges: 9.53
improved: 9.52
interactive: 9.42
enhance: 8.97
improve: 8.91
explore: 8.74
feedback: 8.70
mitigate: 7.88
Write: 7.00
<bos>: 1.00

13 most activating

<space>: 17.89
4: 17.31
6: 17.09
\t: 16.56
2: 16.56
': 16.48
]: 16.39
<space><space>: 16.36
7: 16.17
<tab>: 16.00
%: 15.63
3: 15.61
\n: 15.58

Figure 3 only shows layer 8 for an unknown model, and 5 “randomly sampled tokens”

Although we state in L. 127–130 that this is Llama-3 8B, we agree this should be clearer. We’ve added model identifiers throughout the relevant sections.

Additionally, we include full results for this experiment (total of 135 randomly sampled tokens across 18 layers) in Appendix E, Figures A7 and A8. The results in Figure 3 are largely representative of the appendix figures.

§4.2 raises three interesting potential scenarios about when TLCM may arise, but they only evaluate a single one*

The experiment in Sec 4.2 tests all three scenarios. We show:

1. Learning-induced: TLCM does not appear in untrained models.
2. Persistent: TLCM strengthens over training, and is not—for example—some training pathology caused by instability.
3. Pretraining-derived: TLCM emerges during pretraining, not SFT or RL.

many of the findings are purely empirical, however, there lacks… theoretical discussion about TLCM

Thank you for raising this point about theoretical grounding. We prioritized empirical characterization for several reasons:

1. Phenomenon complexity requires strong empirical foundation Our results reveal TLCM's multifaceted nature—spanning training dynamics, information flow patterns (context experiments), sublayer interactions (MLP/attention), and potentially architecture-specific effects (Appendix A). Given this complexity, we believe empirical documentation is essential before proposing theoretical explanations that might be premature or overly narrow.

2. Building foundations for mechanistic interpretability We view our contribution as part of a broader scientific process: establishing a corpus of well-characterized, reproducible phenomena provides the empirical constraints necessary for developing robust theories. Just as early observations of superconductivity (1911) provided crucial data for the eventual BCS theory (1957), we hope TLCM's documentation enables both practical applications and theoretical understanding.

We aim to contribute:

1. Discovery of TLCM—a novel, broadly-observed phenomenon that reveals unexpected layer dynamics
2. A set of characterization experiments (e.g., 12+ models, MLP/attention differentiation, token-level statistics, training dynamics, causal interventions, context experiments).
3. A Jacobian eigenbasis-based interpretability technique, which we develop and apply to TLCM to show selective subspace rejection and reinforcement.
4. A conceptual scaffold—the propose-and-reject hypothesis that organizes observations and suggests research directions
5. Actionable insights for recent results in mech interp: CLTs outperforming SAEs, misfires, and model steering difficulties.

We position this work as empirical science: establishing a robust phenomenon that challenges current understanding.

The derivation of the “propose-and-reject” hypothesis seems rather speculative

The propose-and-reject hypothesis states that a layer proposes a broad swath of possible “features,” while the following layer—equipped with an attention head to gather context—removes the features that are not relevant.

We intend this to be a conceptual synthesis that is consistent with all our empirical results thus far. Specifically, the hypothesis explains:

• Quantitative contextual dependency experiments.
• TLCM corrects only the most recent layer, rather than correcting all prior layers
• TLCM’s prevalence in early layers, which are linked to “enrichment”
• Very low TLCM activations on beginning of sequence tokens
• TLCM emerges when considering the combination of Attention/MLP sublayers (MLPs cannot access context without attention)
• The causal relationship we show—where the following layer dynamically attenuates the prior layer—in Sec 5.2.
• Sec 5.3’s Jacobian experiment demonstrating unit attenuation (\\lambda \= 1) of features

With this framing, we intend to generate future research directions, not to close the discussion.

what does it mean to “partially oppose” in a general setting, or “reverse” other layers’ contributions? Similarly, L.52… Having negative cosine similarity does not mean “erasing information”

Good clarification! Our precise claim is that adjacent layers attenuate specific components of the prior layer’s contribution.

This is supported by stronger evidence beyond cosine similarity:

• In Sec. 5.1, causal interventions show that scaling $\\mathbf{c}\_i$​ leads to proportional counter-scaling in $\\mathbf{c}\_{i+1}$​,

• In Sec. 5.2, our Jacobian eigen-analysis reveals subspaces where unit increases receive unit correction.

To reflect your suggestion, we’ve updated our wording to emphasize “correction” and “attenuation” rather than “erasure” or “reversal.”

L.99-101: how do we know that contribution vector similarity tells us something about “novel information” and “reinforcing existing representations”?

Great question! Our interpretation draws from the common assumption in mechanistic interpretability that contributions approximate a linear combination of interpretable features (L. 89–92). Under this view:

• An orthogonal contribution roughly corresponds to introducing new features,

• A positively aligned contribution roughly corresponds to reinforcing existing features.

To ground this in examples: Recent work (On Layer-wise Representation Similarity by Jiang et al) has found positive cosine similarity between layers in ViTs. Other work (Residual Connections Encourage Iterative Inference by Jastrzebski et al) views each residual block as performing a gradient step in activation space, so TLCM would correspond to a too high “learning rate,” which is unexpected because pretraining should find the optimal learning rate.

L.40 How do we quantitatively measure “high context dependency” in tokens?

Thanks for raising this.

Quantitatively, we use number of preceding tokens as a proxy for contextual dependency (L. 206–211). In Figure  2b, TLCM activation shows a linear relationship with this proxy: every additional 1k context tokens corresponds to about one additional TLCM activation, on average. While computed across 2,000 wikitext documents, we agree that this is a limited proxy.

Qualitatively, we find high TLCM activation in tokens (newlines, numbers) known from prior work to rely on context. For example, newline tokens, as discussed in Lindsey et al.  $4$ , often trigger planning behavior that integrates surrounding information.

lack of more rigorous derivation/description why choosing exactly for the threshold for TLCM…Looking at Figure 2b) this is not even at the low point of the distribution

Good question! Figure 2b is computed across 3.5 million adjacent layer contributions across ~100K tokens, making the bimodality of the cosine similarities significant. This is strong evidence that there are (at least) two distinct interaction regimes: TLCM and non-TLCM. We agree these details are important to highlight in the paper, so we’ve added them around L. 130.

Because the $\\operatorname{cossim} \< \-0.1$ threshold is further left from the low point of the distribution, we believe it captures greater specificity than using the low point. In practice, more conservative thresholds (e.g., $\< \-0.18$) yield consistent results, as most plots already involve stronger TLCM activations due to the distribution shape.

why $are we$ surprised by the empirical findings (elaborations of L. 99-105)

Great question! See L. 131–141 and Appendix B.1–B.2, where we explore both the theoretical and empirical significance of TLCM. Using two different sets of assumptions, we show that TLCM is highly statistically significant.

L.115-117: the authors compute the cosine similarity...but report on “correlation” in L. 116 and throughout

We intended these terms to refer to the same quantity. However, we recognize your point and have updated the paper to use “cosine similarity.”

Typos, etc

Thanks for pointing these out! We have fixed these and will improve clarity for the camera-ready version.

Thanks for your review. We’re glad you found the “idea and exploration” interesting, the paper to “read well,” and the tools we introduce useful/interesting “for understanding the mechanisms of transformers!”

We address your questions and some of your concerns below.

It could also be interesting to see the results on multimodal LLMs (e.g vision LLMs), where perturbations in images could be used to understand TLCM better.

Thanks for raising this point. Recent work (On Layer-wise Representation Similarity by Jiang et. al.) has found positive cosine similarity between layers in ViTs. However, it remains an open question whether this phenomenon is limited to just language models.

How does this paper relate to Layer by Layer: Uncovering Hidden Representations in Language Models (https://arxiv.org/pdf/2502.02013)

Great question. One intuition from our work and other recent research is that:

1. Early layers are responsible for building rich representations (note: transformers have an incentive to produce representations that will be useful for future tokens, even though they cannot “see” them).
2. Middle-to-later layers are where final processing occurs.
3. Late layers focus on ensuring that the output distribution is correct, often by removing detail from the residual stream.

We think this flow makes sense, and it’s also well supported by the paper you’ve linked. TLCM's high prevalence in early layers and contextual dependency gut-checks with the early layers' responsibility to “build rich representations.” Additionally, TLCM is notably less prevalent in later layers on most models, which correlates well with the less context-dependent tasks of (2) and (3).

We also cite papers which build the above intuitions, see $20, 21, 32$ .

Can different domains have different TLCM characteristics? (e.g, Code) and it would be interesting to see the results for different languages

Indeed, we find TLCM activations occur on code; we use code for some of our experiments (see Appendix F).

Additionally, our initial experiments in Sec. 4.1 are computed across 100,000 tokens of Wikitext, which—while mostly English—includes sections in other languages. In short, we see TLCM activate across varied domains.

Could not find diagram FigureA4 in appendix corresponding to Section 4.2

Good catch. We mean to refer to Figure A9. Because the paper and appendix were submitted separately at different times, it seems like we unintentionally compromised the correspondence of this figure in the paper and in the appendix. This is fixed for the camera-ready version.

TLCM can be directly leveraged to develop more effective interpretability tools or even guide architectural improvements in LLMs to optimize information flow.

Great point. In Appendix A, we begin to explore the architectural features that could induce TLCM. Essentially, we introduce the concept of LayerNorm blindness, which is the phenomenon that LayerNorm causes attention and MLP sublayers to be “blind” to the norm of the residual stream. Thus, they do not know how to size their contributions and may be biased to over-contribute, so their contribution is not overshadowed. If true, this presents a real inefficiency and an opportunity to “optimize information flow.”

However, we find TLCM in models that do not have LayerNorm blindness. Thus, this is not the full explanation. We agree that there is more work to do and are excited about directions like this for future work.