The emergence of sparse attention: impact of data distribution and benefits of repetition

We thank the reviewer for their feedback and for clearly expressing their concerns about our paper. We provide some clarification below, which we believe address all the concerns raised by the reviewer:

1. Weakness: The single-location task is insufficient to analyze the training dynamics of real models. The reviewer is right that our analysis (in particular the theoretical one) does not fully capture all aspects of the learning dynamics of large language models, trained on more sophisticated tasks. However, we would like to nuance this point. Our analysis was designed with minimality as a main goal: we wanted to find a minimal setting in which emergence occurs during training and in which we could see the benefits of repetition. This allows us to identify, for the first time, a simple mechanism (sparse attention) that is easier to understand intuitively, while being practically relevant as it is ubiquitous in trained Transformers, and has close links with induction heads in in-context learning. By design, we are not trying to quantitatively capture realistic learning dynamics, but rather get an intuitive understanding of how different variables (here sequence length, data dimension, or repetition) affect learning. We believe our analysis provides insights about some non-trivial behaviors that this simplification makes much easier to analyze and conceptually easier to grasp. And we hope that, in future work, these findings can be applied to wider settings to gain additional understanding.
2. Weakness: Fixing the location of the relevant token limits the applicability of the results. The location is indeed fixed in all experiments provided in Section 2. The ablations in Figure 14 show that randomizing the token position (and providing a feature to the Transformer that allows it to select this position) has some effect on the exact scaling laws that the plateau length follows, but the overall finding remains unchanged.
3. Question: Values taken by $(w, a)$ depending on the B value? $w$ always takes the same final value ($1$) whatever the B value. When there is in-context repetition, the attention is equally shared between all relevant tokens, so we have the identity $\alpha B = 1$ at the end of training with $\alpha$ being the attention given to one relevant token. This implies that the attention to each of the relevant tokens decreases as $B$ increases. In terms of $a$, the difference between the $a$ for relevant tokens and the $a$ for non-relevant tokens goes to infinity, whatever the $B$ value. To summarize, increasing $B$ does not change the final function learned by the network, but only convergence speed.
4. Question: evidence for the claims of l.185-187? We thank the reviewer for pointing out this issue. These sentences summarize the findings of the section, and the rest of the section provides evidence for these claims. We have made that clearer in the current version of the manuscript.
5. Question: Differences and connections between grokking and emergence? Emergence is a broad concept that goes beyond emergence during training (the type we study). One can understand grokking as a specific form of emergence during training, in which the generalization abilities of the model suddenly improve much later than the ability of the model to solve the training task. However, there are other forms of emergence during training, for example, there are cases in which emergence already occurs looking at the training loss only (we study one such example here).

We hope that this answers the concerns raised by the reviewer, and remain available for additional clarifications. Given these answers, we would appreciate it if the reviewer could reevaluate our work.

We thank the reviewer for their appraisal of our paper!

We agree with the reviewer that the role of in-context repetition is rather intuitive. Please see our answer to BBS6 regarding the second weakness the reviewer mentions (specificity of our results to the in-context learning task). Given the generality of the mechanism we highlight, we expect cross-sample repetition to be useful beyond the in-context learning task we study; Zucchet et al. 2025 provide one such example.

The link with active learning is as follows: we show that varying diversity might be a powerful way to speed up emergence, but can be damaging for generalization. This insight provides a simple active learning algorithm: whenever an agent sees that it is stuck on a task, it decreases data diversity (which it can, as an active learner), and it increases it afterwards when learning finally kicks off.

We have made these points clearer in the current version of the manuscript.

Regarding the questions asked by the reviewer:

1. Question: why remove the layer norm? Note that we only remove it for the single-location linear regression experiments, and keep it for the in-context learning experiments. Our main hypothesis is that it is difficult to always output 0s at specific locations with layer norm, but this is possible without. As a consequence, the layers have to implement the selection mechanism together, a cooperation that may take longer to learn.
2. Question: linear growth of the value weights? We haven’t experimented with this and do not have a good argument for why this would happen, as it would not help to improve the selection mechanism nor the required linear transformation of token embeddings. That said, we are happy to run that experiment if the reviewer has a hypothesis for why this would occur.

We thank the reviewer for their positive feedback and thoughtful review. We address the weaknesses they raised and the questions they asked below:

1. Weakness: no discussion of which task could benefit from these effects. We thank the reviewer for pointing out this blind spot in our argumentation. We believe our findings to be general and not limited to tasks that rely on heavy in-context manipulations. We believe that some sparse context selection is needed in order to see the emerging behavior we characterize, which might be relatively common given the sparsity of trained LLMs. We expect cross-sample repetition to be particularly helpful in tasks in which learning the feedforward mapping is hard (high $d$ in our analysis, which this form of repetition effectively reduces). This includes learning facts – the findings of Zucchet et al. 2025 are one example of the benefits of cross-sample repetition – but may extend to some other tasks like reasoning-heavy tasks. In-context repetition mostly when learning from very long sequences (as they decrease the effective sequence length). It is a natural feature of language (it helps humans to process texts), and we expect that reducing it would actually considerably slow down the learning of language models. We have included more detail about these considerations in the discussion and we hope that our work will motivate future studies to investigate these questions.
2. Weakness: theoretical power laws don’t exactly extend to other tasks. Indeed, each task seems to have its own power law and its specifics are task-, architecture- and optimizer- dependent (our ablations of Appendix B.3 shows quite strong power law variations already on the single-location linear regression task only). We believe that the fact that there exists such a power law for more sophisticated tasks (see also Figure 2 right of Zucchet et al. 2025) is a good sign that the core phenomenon, though not the exact coefficients, is robust and may generalize to real-world applications.
3. Question: does the position of the selected token affect learning speed? We have no reasons to believe that it has a strong impact. When looking at Figure 11, which shows learning dynamics for different seeds (and thus different locations), all of them have similar dynamics.
4. Question: role of in-context repetition and does repetition causally drive sparse attention? The reviewer is right that the effect of in-context repetition is rather intuitive: if the same information is repeated multiple times in the context, it becomes more correlated with the token to predict and thus learning is faster. Our theory captures that analytically. Repetition affects the learning of sparse attention (as per the argument above), but it does not causally drive sparse attention in the sense that sparse attention would arise even without repetition (because it is useful to solve the task).

We hope this clarifies our results, and we remain available during the rebuttal period for additional clarifications.

We appreciate the reviewer’s positive feedback on our work. We agree with the weakness they raised (see our answer to reviewer oc9h for nuance on it), and answer their questions below:

1. Question: why does in-context repetition make the task easier? With in-context repetition, the information relevant for the prediction is repeated multiple times within the context, and is thus more correlated with the token to predict. This provides a stronger statistical signal to the model, and thus speeds up learning. An alternative way of seeing that is that in-context repetition reduces the effective sequence length (it’s mathematically similar to having a sequence of length $T/B$ with only one relevant token). Given that learning time scales with sequence length, this effectively makes the task easier and speeds up learning.
2. Clarification: takeaway from Figure 1 right and 2. Figure 1 right is the simulation of the simplified dynamics (Eq 3 and 4), which comes at the price of a few assumptions and on which we derive the theoretical exit time. Figure 2 consists of the learning dynamics of the model of Equation 2 (without the assumptions). The takeaway from these figures is indeed that linearizing the early dynamics of the simplified model captures the plateau length in the model of Eq. 2.
3. Clarification: signal and noise terms in equations 3 and 4. At a high level, there are two types of tokens in this task, the relevant ones (signal) and the non-relevant ones (noise). The derivation of Equations 3 and 4, which can be found in Appendix A.2 and A.3, uses this insight to group the contributions to the loss of the tokens together, and Eq. 3 and 4 reflect this split.
4. Question: hypothesis for why layer normalization makes the task harder? We have not thoroughly investigated this question as this is likely specific to the single-location regression task (we use layer norms in the in-context learning experiments). Our main hypothesis is that it is difficult to always output 0s at a specific location with layer norm, but this is possible without. As a consequence, the layers have to implement the selection mechanism together, a cooperation that may take longer to learn.
5. Question: how does the induction head map to equation 2? We would first like to clarify that equation 2 is the equation of our toy model and that it can not directly implement an induction head (it is not a sequence to sequence layer, it cannot use relative position information which is needed for the copying mechanism, it has no semantic similarity needed for the moving mechanism). That said, the toy model of equation 2 is relevant in the sense that it can be used to capture some of the dynamics underlying the learning of sparse attention and therefore of the induction head. This is because the attention needed for the copying mechanism of the induction head is sparse (attention on the last token), and the one for the moving mechanism is sparse (attends to the same query token in the context).

We hope this answers the questions of the reviewer and we remain available for additional clarification.

Thank you for your response and addressing my questions. I think this is a valuable contribution towards understanding training time emergence, and have maintained my initial rating of accept. I would encourage the authors to clarify the above points in their main paper as well, for reader convenience.