Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities

We thank the reviewer for their detailed feedback regarding clarifications on theoretically predicted generalization behavior, experimental setups, and writing improvements. We intend to address the above mentioned weaknesses and questions in separate sections below.

Response to Weaknesses

My main concern is that one of the key motivations of the paper is to examine whether theoretical limitations on length generalization actually impact the performance of large pretrained models in practice. However, I believe this question is most meaningful when tested on relatively long sequences, whereas the experiments in Section 4.1 focus on rather short ones.

1. Indeed, we acknowledge that the sequence limit of 50 tokens in Section 4.1 does not exhaust the potential real world sequence lengths. The important point in Section 4.1 is that the key failures already appear at shorter lengths, which are potentially more frequent in the training data and particularly practically relevant. As the reviewer mentioned, we also test longer sequences up to 500 tokens on the Lorem-Ipsum-style copying task in the Section 4.2.

Model performance mainly depends on whether the sequence lengths used during testing were also present during training. As a result, performance (especially on shorter sequences that are more common in training data) may reflect the influence of the training distribution more than any fundamental architectural limitations.

The observation in Section 4.2, where models fail to copy because of ambiguous tokens, aligns with the theory prediction and is performed on longer sequences. This also supports the idea that the sequences used in Section 4.1 may have been too short to properly test the predictions of the length generalization theory.

Therefore, the observed gap between theoretical predictions and empirical performance might be more attributable to the nature of the training data than to the scale of the models or the web-scale data they were trained on, as suggested in the introduction (lines 37–41). I believe this point should be clarified more.

2. While we agree with the reviewer that the influence of the training distribution does impact performance in many ways, that is precisely why we found our results exciting: our task setups allowed us to disentangle the effects of pretraining from architectural limitations. Through our experiments in Section 4.1, we were already able to observe the effects of pretraining — certain abilities were selectively amplified more than others, despite there being little to no theoretical difference between them. Since we were able to shortlist which abilities the training distribution amplifies the most, we moved on to the experiments in Section 4.2. We are glad the reviewer agrees that the failure modes observed in Section 4.2 indeed align with theoretical predictions. It was only in Section 4.3, through our fine-tuning experiments and mechanistic insights, that we were able to conclude that the directional asymmetry (e.g. left/right or forward/backward) was due to pretraining, while certain other failures (e.g. in retrieval models, or repeated copying) could be attributed to architectural limitations. In these experiments, we tested our length generalization setup by training models on shorter sequences and evaluating them on longer ones. We were able to eliminate failures caused by pretraining, but the effects of architectural limitations remained. This helped us finally disentangle the effects of pretraining from those of architectural limitations. As suggested by the reviewer, we will make this point clearer in the main text of the paper.

Response to Questions

In addition to my earlier point in the weaknesses section: if the goal of Section 4.1 is to cross-check the C-RASP length generalization predictions on pretrained language models, it’s important to note that the C-RASP framework assumes learned absolute positional embeddings. However, the models evaluated in this section use RoPE. This discrepancy is not clearly acknowledged or discussed in the paper, and it likely introduces a confounding factor into the evaluation.

1. Indeed, the C-RASP[pos] framework assumes a Transformer architecture using APE. In the footnote of the Section 3 we note that while Huang et al. results rely on this assumption, we find that empirically these results extend to the RoPE setting, which are more extensively discussed in the Appendix B, providing the from-scratch results for both APE and RoPE Transformers. However, we agree to make this note more explicit in the body of the paper.

The finding in Section 4.2 and Figure 4 closely resembles the results of Jelassi et al. [2024], who demonstrated that transformers rely on bigram lookup mechanisms for copying. It would be helpful to more explicitly acknowledge (or possibly differentiate) this.

2. Although we do cite the results of Jelassi et al. [2024], we agree that we could make this discussion more explicit, and we promise to do so.

The discussion on “token-level alignments” (lines 217–222) and the “Git Commit History” experiment were difficult to follow and could benefit from clearer explanations.

3. We acknowledge that the token-level alignment and the Git Commit task can be described more clearly. We had hoped that the additional details and example traces provided in Appendix C.3 would help clarify our method. Nonetheless, we will make a concerted effort to further improve the explanations in the main body of the paper.

Is there a particular reason the performance of the GPT-2 model prior to fine-tuning isn't reported? Including this baseline would help quantify the relative gains achieved through fine-tuning on each task.

4. We found that the baseline performance across tasks was at a chance level (i.e. the baseline was 0), so, we have not included those in-context results. We will consider making these results explicit in Figure 6.

The task names (e.g., UL vs. UB) are somewhat confusing when reading the text. As a suggestion, perhaps adding a prefix, like Copy-U-L or Retrieve-U-B could improve readability and help readers quickly identify tasks.

5. Thank you for your suggestions. We will carefully consider ways to improve readability of the paper, including potential task names clarifications.

We are thankful to the reviewer for their comments and aim to address the concerns regarding testing lengths, asymmetry discussion, and evaluation choices below.

Response to Weakness:

Real-world evaluations remain limited to specific tasks at moderate context lengths.

1. While we acknowledge that real-world applications involve processing of larger text sequences, we believe it to be important and interesting that the mentioned effects can already be observed at smaller lengths. Additionally, Section 4.2 does extend our experiments to larger context lengths, showcasing a more naturalistic test bed.

Response to Questions:

The paper can also benefit from some more discussions on the initial asymmetry.

2. We are unsure what the reviewer means by initial asymmetry. If this refers to the from-scratch behavior in comparison to the behavior of pretrained LLMs, we report those results in Appendix B. Otherwise, we would be grateful if the reviewer could clarify this in the discussion.

For the git cherry picking task, some performance drop might be due to the LLM's unfamiliarity with this command. Would be good to have a more straight forward task.

3. While we acknowledge that the model's familiarity with the git cherry pick command might affect the performance, we provided an explanation of the task which uses the same terms as the terms mentioned for the git revert task, and also provided three few-shot examples in the prompt to further explain what the task is, which taken in conjuction should alleviate the effect of token unfamiliarity, as we observed in our experiments in section 4.1 as well. Despite this, we still find a gap in performance between these tasks, matching the theoretical predictions.

Generality of this paper would benefit from evaluating on SOTA closed-source systems.

4. The focus of our paper is to understand the role of architectural limitations and biases from the pretraining data in shaping LLMs' abilities. These details are not known for closed-source systems -- thus, evaluation of the same setup in closed-source setting, even if results are similar, cannot be directly attributed to the model's architecture or pre-training regimen. We thus believe that open systems are most relevant to testing our research questions.

We thank the reviewer for their feedback and suggestions on clarifications of explanations. We would like to jointly respond to question and weaknesses below.

from Weaknesses: The paper does not fully address whether this “bias” is surprising, especially given that Transformers are trained autoregressively (left-to-right).

from Questions: Isn’t a rightward bias natural given the autoregressive training? Why is this presented as an architectural failure rather than an expected outcome?

1. We would like to clarify that we consider the rightward bias to be an artifact of pretraining, and not an architectual failure. Indeed, we agree with the reviewer this it is a natural artifact of autoregressive training on language. We consider it to be especially interesting as the theory predicts equal difficulty of both directions for the architecture for precisely autoregressive generation of output by Transformers. So, despite the bias being intuitive, we consider finding a more precise reason to explain it to be worthwhile. We believe that explaining this bias mechanistically using induction and anti-induction circuits gives an important empirical insight relevant for downstream applications.

from Weaknesses: While reading the paper, it remains ambiguous whether the glitches in non-unique copying are related to the directional asymmetry or are a separate phenomenon entirely. In this case, it is not completely clear what this behavior adds to the "unexpected bias from pretraining" picture.

from Questions: Is there a connection between the glitch phenomenon and the induction/anti-induction head asymmetry? Do the authors believe that predicting multiple tokens at the same time (e.g. with diffusion-based methods) could remove this bias?

2. The copying glitches and the left-right asymmetry are separate phenomena. The copying glitches are tightly related to the inherent architectural difficulty of non-unique copying and retrieval. In contrast, the left-right asymmetry is a bias arising from pre-training. Theory describes the existence of the copying glitches as inherent to Transformer architecture, while asymmetry is removable by fine-tuning. We believe that fine-tuning the model to remove the above mentioned asymmetry is already a good way to address the issue in more practical settings. Non-autoregressive methods lie beyond the scope of our paper, which focuses on Transformers. Confirming the existence or absence of both the glitches and the left-right asymmetry in such architectures is an interesting future work direction that might be a promising solution on par with fine-tuning.

from Weaknesses: To a non-expert reader, it is unclear if the presence of a C-RASP[pos] program implies a necessary and sufficient condition for generalization, or just an empirical trend.

from Questions: Does the existence of a C-RASP[pos] program prove that a task will generalize under all training regimes and model types?

3. Thanks for this question. We would like to point out that, in Section 3, we sketch the theoretical length generalization conditions, including required positional encodings and training procedures. To provide more detail, we will expand this discussion in Section 3 along the following lines: the theoretical length generalization guarantee for C-RASP[pos] by Huang et al. 2024 applies to APE Transformers under an idealized training procedure, where all training data up to some maximum input length N are available and the exact minimizer of a regularized loss is found. Under these specific conditions, the existence of a C-RASP[pos] program for that task guarantees length generalization to larger lengths. While theory has only been worked out for this specific setup, empirically we observe converging results across different PE types and realistic SGD-based training (as shown in Huang et al. 2024 as well as in the experiments in Appendix B). In the future, we would be excited to see the extensions of the existing theory to more PE types and more practical regimes.

from Weaknesses: The fine-tuning experiments (showing that the bias can be removed) are conducted on GPT-2 model rather than the same models (LLaMA, Qwen) used earlier, making comparisons less direct.

from Questions: Why was GPT-2 used for fine-tuning instead of LLaMA/Qwen? Do the authors expect the results to hold for modern LLMs? Can the authors provide a baseline for comparison showing the bias of GPT2 before fine-tuning?

4. Regarding the baseline of GPT-2 before finetuning: In experiments, we found that the in-context performance of GPT-2 was at chance level (i.e. 0), hence, we did not include those results in the paper. We will consider making these results explicit in Figure 6. Regarding the choice of GPT-2: Due to computational budget constraints as well as their usage of RoPE, we did not fine-tune larger, 7-8B and 70B models used in the Section 4.1 at the time of the submission.

from Weaknesses (no corresponding Question): The explanation/analysis of the induction head mechanism appears to be slightly rushed if one does not assume prior familiarity with mechanistic interpretability.

5. Thank you for your feedback. We will strive to improve this.

We thank the reviewer for their suggestions and comments about the paper. As there were no explicit questions from the reviewer, we address the potential weaknesses below:

A seq_len limit of 50 tokens is too low for any real world application which often takes at least thousands of tokens. A more practical setup is needed to understand the full picture.

• The seq_len limit of 50 tokens applies specifically to Section 4.1. We'd like to clarify that Section 4.1 is not about practical setups, but the point here is to cleanly check theoretical predictions about uniqueness and left/right, forward/backward (a)symmetries. We swept lengths from 10 to 50 tokens across multiple model sizes and families and observed the same asymmetry at every length. Because the pattern was already stable in this range, we chose 50 as a convenient cutoff. We note that a positive result (i.e., no difference across task setups) at short lengths would have motivated further scaling, whereas a negative result (a clear pattern towards asymmetry in one direction over the other) on such basic tasks is itself striking. Based on our observations in Section 4.1, we were then motivated to examine longer lengths, which led to the more practical experiments described in Section 4.2. By extending the testing lengths and making the tasks more realistic—such as copying Lorem Ipsum–style paragraphs and performing git revert and cherry-pick on a list of commits—we were able to replicate the findings from Section 4.1. These larger-scale experiments again demonstrated the same asymmetries between left/right or forward/backward directions (as in the git commits case), as well as the relative ease of unique or unambiguous tasks compared to non-unique or ambiguous ones (as in the copying of lorem ipsum paragraphs).

• To further strengthen our point, we have conducted some additional experiments with the Lorem-Ipsum setup at even longer scales (up to 2000 tokens) and report the results in the table below for 2 of our models. Once again, we do see the same effects we see at lengths of 500, i.e., a lower success rate for ambiguous bigrams than for unambiguous ones. Running these models for longer and longer lengths is computationally expensive, and as we consistently see these effects at lengths up to 2000, we have concluded our experiments here. If copying had been perfect at such lengths, it would have warranted a deeper analysis at longer lengths. However, since we already observe failures in copying across models and settings at ~500 tokens, we consider it a noteworthy finding. We find this especially interesting as previous work has shown that models perform perfect accuracy while copying ordinary text (with semantic cues) [1] at thousands of tokens length, whereas our Lorem-Ipsum setup demonstrates failures on copying text without semantic cues at much more moderately sized lengths.

  Model	Unambiguous	Ambiguous
  llama3-70B-Instruct	0.99	0.96
  llama3-8B-Instruct	0.96	0.89
  qwen2.5-32B-Instruct	1.0	0.95
  qwen2.5-7B-Instruct	0.99	0.75

• Overall, we appreciate the reviewer's comment and will report these results at higher lengths more explicitly in the paper.

Fig 3 is not enough to conclude that the asymmetry was introduce by pretraining. A comparison against models without pretraining is needed for attribution

2. We would like to clarify our argument that the asymmetry is introduced by pretraining: We agree that the left-right asymmetry is shown most clearly in comparison against models without pretraining, and we would like to point out the from-scratch results in Appendix B. These show that no left-right asymmetry is present when training from scratch. Further important support for the asymmetry as a pretraining artifact is given in Section 4.4, where we give a mechanistic explanation to both fine-tuned and pretrained behavior, and attribute the asymmetry to an imbalance between induction and anti-induction circuits in pretrained models. Taken together, we agree with the Reviewer that the point is not made by Figure 3 -- rather, it follows from comparison to from-scratch models and from mechanistic interventions. We will revise to ensure this argument is made transparent.

The connection between the "copy and retrieval" task with prestraining ppl or downstream tasks performance is unclear. More specifically how does the learning from the model's behavior on this task generalize to other tasks and if any constructive guidances can be distilled from it. The significance of the findings needs more elaboration

3. We believe that copy and retrieval tasks are indeed fundamental to reasoning, as solving subproblems in the reasoning chain often involves some sort of retrieval and copying of the input [2]. However, we agree that this can be made more explicit, and we will add a more detailed discussion in the Appendix.

The paper can benefit from better presentation and story telling. e.g. a contribution summary at the end of the intro section, as well as the reason to choose copy and retrieval as a representative task would be helpful.

4. Thank you for your suggestions, we will incorporate them into the paper.

[1] Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-Context Models (Liu et al., EMNLP 2024) [2] Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (Zhang et al., 2025.)