Emergence of Linear Truth Encodings in Language Models

We thank the reviewer for the thorough review and the encouragement to report empirical results on more realistic settings.
To address concerns about small models and synthetic data, we conducted two complementary experiments:

1. Natural‑language facts. We replicated our study by training fully‑parameterized Transformers on the CounterFact dataset.
2. Large‑model trajectory. We traced the entire training path of Pythia‑6.9 B to see whether the same dynamics emerge in the wild.

Training dynamics on more realistic data

Following your suggestion, we re‑ran the study in a more realistic setting—the CounterFact dataset [1], which contains simple factual statements in natural language. Please note that in contrast to our original synthetic setting, the examples from this dataset contain many different sentence structures and much variation that is characterized to natural language,

We trained standard Transformer models with 2, 5, and 9 layers (4 attention heads and one MLP block per layer) on a single relation, updating all parameters.
Each training sample is a concatenation of two sequences, just as in the paper’s main experiments. To enforce the truth‑co‑occurrence hypothesis, the factuality of the two sequences is matched; in 5 % of the samples we corrupt the attribute by randomly swapping it with another attribute.

The results reproduce the same two‑phase learning pattern seen in our synthetic setup:

1. Phase 1 — rapid memorization
   Early in training the model places almost all probability mass on the memorized attribute for both true and false contexts, so the entropies are equally low (H_true ≈ H_false).

2. Phase 2 — abrupt emergence of a linear encoding
   A few hundred steps later a strong linear signal appears in the penultimate token of the second sequence (AUC_S2 ≈ 0.97).
   At the same time H_false shoots up while H_true drops, exactly as predicted by theory.

Because NeurIPS technical constraints, we summarize the average numbers across training phases.

Here, AUC_S1 / AUC_S2 refer to the Area Under the ROC Curve of a linear probe trained on
    • S1: the last token of the first sentence, or
    • S2: the penultimate token of the second sentence.

In the 2‑layer model the memorization plateau lasts until ≈ 1.2 k steps; then the linear encoding emerges and H_false rises, reducing loss on corrupted samples.
The deeper models transition earlier—after only 500–750 steps—but follow the same trajectory.
All metrics here are taken from the final layer. In the deeper networks the penultimate layer reaches comparable accuracy, while earlier layers stay near chance. Overall, these results confirm the two‑phase dynamics we predict in a more realistic setting.

Evaluating checkpoints of a “real” LM

To test whether the two‑phase dynamics also appear in a large model trained on open‑web data, we analyzed the Pythia‑6.9 B training checkpoints released by EleutherAI. Using the CounterFact dataset we construct each input by concatenating $K = 4$ factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup.
For every checkpoint we measure three signals on the final statement:

• Memorization: the percentage of cases where greedy decoding suceeds in completing the correct token.
• Uncertainty: the entropy of the model’s full‑vocabulary distribution for predicting the last token (we record the difference in entropy between true and false context).
• Linear separability: the accuracy of a linear probe trained to classify the truth value of the surrounding context.

$\\Delta H$ denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model’s output distribution places the correct continuation token at the top‑1 position.

Findings.

• Early training (≤ 1 k steps). $\\Delta H \\approx 0$; the model memorizes indiscriminately.
• Mid training (3 k–80 k). Memorization jumps, then plateaus, while $\\Delta H$ and probe accuracy climb steadily.
• Late training (≥ 80 k). Entropy separation continues to widen even after memorization saturates, mirroring Phase 2 but over a longer horizon.

This echoes the two‑phase pattern we observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase in entropy separation. There are, however, several differences. The second phase is more gradual, and classification and entropy increase even before memorization stabilizes. These differences likely stem from the model’s continual exposure to new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest final memorization and classification scores align with prior reports that the Pythia series is under‑trained relative to its capacity.

In sum, both experiments corroborate the predicted two‑phase dynamics we observe.

Additional models in the intervention experiments

Following your concerns on the intervention experiments, we repeated the experiments on three more models with 1,024 examples (8× larger). Reported values, as in the main text, are NLL deltas between false and true contexts:

• Δ (Base F → Base T) — change when switching the base model from false to true context.
• Δ (Base F → Interv F) — change between base and intervened model on false context.

Patterns are broadly consistent, though Gemma is more sensitive to false context than the others.

Reference

[1] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT, 2022.

We thank the reviewer for their insightful suggestions and for encouraging additional large‑model experiments.
Below we address each point and indicate the text changes planned for the camera‑ready version.

Theoretical modeling and idealized synthetic setting

Our intention is to provide a rigorous characterization in a minimal setting: if the phenomenon shows up even under these minimalist conditions, it becomes a plausible building block for the far richer dynamics of large‑scale models. We emphasize that even our simplified objective (Eq. (2) after line 141) already contains a weight matrix $W$ trained inside a non‑linearity. This results in a non‑convex function that resembles a 2‑layer network, and its optimization process is known to be challenging to analyze, as accomplished in Theorem 3. Our analysis goes a step further and studies the properties of the learned weights (Theorems 1 and 2), which show the occurrence of the truth‑linear subspace.

Please note that in the appendix we show that releasing the $K,Q,V$ and embeddings parameters (still one layer, but with nothing frozen) retains the effect, both the two-phases and the structure of the attention matrix (though when the embeddings are also trained, some of the structure disappears because they can also memorize). We will move this result into the main text for visibility.

Training dynamics on more realistic data

Following your suggestion, we re‑ran the study in a more realistic setting—the CounterFact dataset [1], which contains simple factual statements in natural language. Please note that in contrast to our original synthetic setting, the examples from this dataset contain many different sentence structures and much variation that is characterized to natural language,

We trained standard Transformer models with 2, 5, and 9 layers (4 attention heads and one MLP block per layer) on a single relation, updating all parameters.
Each training sample is a concatenation of two sequences, just as in the paper’s main experiments. To enforce the truth‑co‑occurrence hypothesis, the factuality of the two sequences is matched; in 5 % of the samples we corrupt the attribute by randomly swapping it with another attribute.

The results reproduce the same two‑phase learning pattern seen in our synthetic setup:

1. Phase 1 — rapid memorization
   Early in training the model places almost all probability mass on the memorized attribute for both true and false contexts, so the entropies are equally low (H_true ≈ H_false).

2. Phase 2 — abrupt emergence of a linear encoding
   A few hundred steps later a strong linear signal appears in the penultimate token of the second sequence (AUC_S2 ≈ 0.97).
   At the same time H_false shoots up while H_true drops, exactly as predicted by theory.

Because NeurIPS technical constraints, we summarize the average numbers across training phases.

Here, AUC_S1 / AUC_S2 refer to the Area Under the ROC Curve of a linear probe trained on
    • S1: the last token of the first sentence, or
    • S2: the penultimate token of the second sentence.

In the 2‑layer model the memorization plateau lasts until ≈ 1.2 k steps; then the linear encoding emerges and H_false rises, reducing loss on corrupted samples.
The deeper models transition earlier—after only 500–750 steps—but follow the same trajectory.
All metrics here are taken from the final layer. In the deeper networks the penultimate layer reaches comparable accuracy, while earlier layers stay near chance. Overall, these results confirm the two‑phase dynamics we predict in a more realistic setting.

Evaluating checkpoints of a “real” LM

To test whether the two‑phase dynamics also appear in a large model trained on open‑web data, we analyzed the Pythia‑6.9 B training checkpoints released by EleutherAI. Using the CounterFact dataset we construct each input by concatenating $K = 4$ factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup.
For every checkpoint we measure three signals on the final statement:

• Memorization: the percentage of cases where greedy decoding suceeds in completing the correct token.
• Uncertainty: the entropy of the model’s full‑vocabulary distribution for predicting the last token (we record the difference in entropy between true and false context).
• Linear separability: the accuracy of a linear probe trained to classify the truth value of the surrounding context.

$\\Delta H$ denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model’s output distribution places the correct continuation token at the top‑1 position.

Findings.

• Early training (≤ 1 k steps). $\\Delta H \\approx 0$; the model memorizes indiscriminately.
• Mid training (3 k–80 k). Memorization jumps, then plateaus, while $\\Delta H$ and probe accuracy climb steadily.
• Late training (≥ 80 k). Entropy separation continues to widen even after memorization saturates, mirroring Phase 2 but over a longer horizon.

This echoes the two‑phase pattern we observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase in entropy separation. There are, however, several differences. The second phase is more gradual, and classification and entropy increase even before memorization stabilizes. These differences likely stem from the model’s continual exposure to new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest final memorization and classification scores align with prior reports that the Pythia series is under‑trained relative to its capacity.

In sum, both experiments corroborate the predicted two‑phase dynamics we observe.

Additional models in the intervention experiments

Following your concerns on the intervention experiments, we repeated the experiments on three more models with 1,024 examples (8× larger). Reported values, as in the main text, are NLL deltas between false and true contexts:

• Δ (Base F → Base T) — change when switching the base model from false to true context.
• Δ (Base F → Interv F) — change between base and intervened model on false context.

Patterns are broadly consistent, though Gemma is more sensitive to false context than the others.

References

Meng, K., Bau, D., Andonian, A., & Belinkov, Y. Locating and Editing Factual Associations in GPT, 2022.

Regarding your question on layer 11, this is in line with previous work that indicates that many human-interpretable concepts are "maximized" in middle layers. We agree that understanding why it is maximized in that particular layer is very interesting, but think that this is a different question than the focus of this paper, and therefore leave it for a future work.

Once again, we appreciate the reviewer’s constructive feedback and believe these additional results strengthen the paper’s empirical grounding.

Thanks for the response with rich content! I am glad that the authors tried their best to address most of my concerns.

Me: The toy example can be too different from real LLMs.
Authors: Train real small-scale transformers on the CounterFact dataset again to show the results.

Me: There are Pythia models with model checkpoints that you can try.
Authors: Try new experiments with Pythia, show some similar observations, but also show a small difference (entropy increase before memorization stabilizes).

Me: Show me experiments with more model families.
Authors: Try Gemma and Qwen and show the results.

These additional experiments/contents addressed most of my original concerns. I will be happy to see this paper get accepted if these new experiments/results/discussion can be incorporated into the main sessions of this paper, which will fundamentally makes this paper much more solid. However, there are still a lot of experiment details, not shown in the tables above, that should be added later on. I tends to trust the authors will do their best in the camera ready version to incorporate these new contents with all the details. If so, I will be happy to raise my scores to accept this paper.

We thank the reviewer for their positive assessment and for highlighting both the paper’s structure and novelty.
Below we answer the specific questions and note the text changes we will make in the camera‑ready version.

General edits

We appreciate the comments on typos and the need to improve the captions and results descriptions. We will incorporate these in the final version.

Figure 2

Yes—the patterns are reproducible. The particular values that are arg‑maxed naturally differ between examples where the gold token changes, but the distribution becomes flattened when conditioned on false contexts, as shown by the aggregated entropy numbers we report. We will convey this more clearly in the final version.

Example length

In the experiments included in this paper, the sequence length is indeed fixed; this makes the theoretical analysis easier. However, in preliminary runs we inserted extra padding tokens at random, and this did not change the empirical picture.

Mapping (subject ↔ attribute)

The relation between subjects and their attributes is defined by a random permutation that depends only on the random seed. The results are not sensitive to that seed.
Please also refer to the additional experiment described in the response to the other reviewers (technical constraints at NeurIPS prevented us from attaching a general response). There we reproduce similar dynamics in a model trained on natural‑language data satisfying the truth‑co‑occurrence hypothesis. In that setting the examples no longer have a rigid structure, yet we observe the same phenomena as in the synthetic data.

Eq. 1

Yes—thank you for pointing this out.

Notation

Thank you as well; we will standardize the notation for the subject and attribute throughout.

Once again, we are grateful for the reviewer’s careful reading and helpful suggestions, and we look forward to refining the paper accordingly.

Thank you for the follow-up and for maintaining your positive assessment of our work! Please note we aimed to answer the other reviewers’ concerns in detail in the corresponding responses, providing new results that substantiate our findings in more realistic settings.

We sincerely thank the reviewer for their thoughtful and constructive feedback, and we appreciate the opportunity to clarify our work.
We address each point in turn below.

Regarding the sentence at line 272

This sentence was unfortunately a remnant from a preliminary version and doesn't accurately describe our findings. We apologize for the confusion. Frozen attention is not necessary for reconstructing the pattern in Fig. 1. Indeed, in Appendix C (“Bridging the gap between the fully‑trainable model and the toy model”, line 852) we report the trainable‑attention results: the heat‑map in Fig. 10a retains the same pattern we see in Fig 1 in the main text, while Fig. 10b shows that if embeddings are also learned, part of the memorization moves there and the pattern weakens. The two phases dynamics and emergence also remain when updating all parameters; this is how figure 3 was generated. We will clarify this in the main text and fix line 272. Please also see “Training dynamics” below.

Uniform‑attention head and other assumptions

We recognize that idealizations such as uniform, fixed attention (together with a single‑layer architecture) are not faithful to production LMs. They are, however, standard tools in mechanistic‑transformer theory because they strip away distractions and let us isolate one concrete route by which a linear “truth” direction can form. Similar works that study different properties of self‑attention also make simplifying assumptions, such as fixing the attention weights and training only a single weight matrix—see, for example, [1] on implicit bias and [2] on benign overfitting in self‑attention.

Our intention is therefore to provide a rigorous characterization in a minimal setting: if the phenomenon shows up even under these minimalist conditions, it becomes a plausible building block for the far richer dynamics of large‑scale models. We emphasize that even our simplified objective (Eq. (2) after line 141) already contains a weight matrix $W$ trained inside a non‑linearity. This results in a non‑convex function that resembles a 2‑layer network, and its optimization process is known to be challenging to analyze, as accomplished in Theorem 3. Our analysis goes a step further and studies the properties of the learned weights (Theorems 1 and 2), which show the occurrence of the truth‑linear subspace.

We also remark that attention is typically uniform at initialization in high dimension, and that the value matrix usually receives more gradient signal than the key‑query matrix initially [4, 5], making our assumption relevant for the initial phase of training. After the value matrix has learned the structure in Eqs. (4–5)—which already gives good performance—we found empirically that when $\\rho = 1$ (only true sequences), the key‑query matrix tends to learn a positional pattern that avoids attending to the x and y tokens from x′. We now have an analysis of how this happens via a gradient step on this matrix: the model with uniform attention over‑ (resp. under-)estimates the probability of $y′ = g(x)$ (resp. $y′ = y$) on true sequences, hence the gradient places smaller attention on the x, y tokens, using positional embeddings. We will happily include this proof in the appendix.

Small vocabulary and “effectively low rank”

Please note that in the appendix (Figure 7) we re‑ran the toy experiment with vocabulary sizes 128, 512, and 4096 and different hidden‑state dimensions. In all cases, Linear‑probe AUC stays above 0.86. Furthermore, the positive‑classification margin guaranteed in Theorem 2 does not depend on vocabulary size; we will point this out directly after the theorem.

Training dynamics on more realistic data

Following your suggestion, we re‑ran the study in a more realistic setting—the CounterFact dataset [3], which contains simple factual statements in natural language, with variations in sentence structure.

We trained standard Transformer models with 2, 5, and 9 layers (4 attention heads and one MLP per layer) on a single relation, updating all parameters.
Each training sample is a concatenation of two sequences, just as in the paper’s main experiments. To enforce the truth‑co‑occurrence hypothesis, the factuality of the two sequences is matched; in 5 % of the samples we corrupt the attribute by randomly swapping it with another attribute.

The results reproduce the same two‑phase learning pattern seen in our synthetic setup:

1. Phase 1 — rapid memorization
   Early in training the model places almost all probability mass on the memorized attribute for both true and false contexts, so the entropies are equally low (H_true ≈ H_false).

2. Phase 2 — abrupt emergence of a linear encoding
   A few hundred steps later a strong linear signal appears in the penultimate token of the second sequence (AUC_S2 ≈ 0.97).
   At the same time H_false shoots up while H_true drops, exactly as predicted by theory.

Because NeurIPS technical constraints, we summarize the average numbers across training phases:

Here, AUC_S1 / AUC_S2 refer to the Area Under the ROC Curve of a linear probe trained on
    • S1: the last token of the first sentence, or
    • S2: the penultimate token of the second sentence.

In the 2‑layer model the memorization plateau lasts until ≈ 1.2 k steps; then the linear encoding emerges and H_false rises, reducing loss on corrupted samples.
The deeper models transition earlier—after only 500–750 steps—but follow the same trajectory.
All metrics are taken from the final layer. In the deeper networks the penultimate layer reaches comparable accuracy, while earlier layers stay near chance. Overall, these results confirm the two‑phase dynamics we predict in a more realistic setting.

Evaluating checkpoints of a “real” LM

To test whether the two‑phase dynamics also appear in a large model trained on open‑web data, we analyzed the Pythia‑6.9 B training checkpoints released by EleutherAI. Using the CounterFact dataset we construct each input by concatenating $K = 4$ factual statements whose preceding context is either entirely true or entirely false, mirroring our previous setup.
For every checkpoint we measure three signals on the final statement:

• Memorization: the percentage of cases where greedy decoding suceeds in completing the correct token.
• Uncertainty: the entropy of the model’s full‑vocabulary distribution for predicting the last token (we record the difference in entropy between true and false context).
• Linear separability: the accuracy of a linear probe trained to classify the truth value of the surrounding context.

$\\Delta H$ denotes the entropy gap between matched prompt pairs presented with false versus true context. The memorization rate is the share of instances in which the model’s output distribution places the correct continuation token at the top‑1 position.

Findings.

• Early training (≤ 1 k steps). $\\Delta H \\approx 0$; the model memorizes indiscriminately.
• Mid training (3 k–80 k). Memorization jumps, then plateaus, while $\\Delta H$ and probe accuracy climb steadily.
• Late training (≥ 80 k). Entropy separation continues to widen even after memorization saturates, mirroring Phase 2 but over a longer horizon.

This echoes the two‑phase pattern we observed in simpler experiments: an initial jump in memorization followed by a slower, steadier increase in entropy separation. There are, however, several differences. The second phase is more gradual, and classification and entropy increase even before memorization stabilizes. These differences likely stem from the model’s continual exposure to new facts during training, unlike our idealized setup where all facts are seen in a single gradient step. The modest final memorization and classification scores align with prior reports that the Pythia series is under‑trained relative to its capacity.

In sum, both experiments corroborate the predicted two‑phase dynamics we observe.

References

[1] Vasudeva, Bhavya, Puneesh Deora, and Christos Thrampoulidis. Implicit bias and fast convergence rates for self‑attention, 2024.
[2] Magen, Roey, et al. Benign overfitting in single‑head attention, 2025.
[3] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and Editing Factual Associations in GPT, 2022.
[4] Li, Li, and Risteski. How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding, ICML 2023.
[5] Bietti, et al. Birth of a Transformer: A Memory Viewpoint, NeurIPS 2023.