Interpreting the Repeated Token Phenomenon in Large Language Models

We thank the reviewer for their valuable comments. Next, we address the reviewer's questions and concerns:

“I feel there is a gap between "repeated tokens could diminish the influence of the preceding prefix" and "repeated tokens could lead to training data leakage". I am looking forward to the clarifications from the authors.”

We provide a brief proof below to theoretically validate the diminishing prefix influence (a full proof will be in the Camera-Ready version). This effect leads to the described attention sink divergence.

We have revised the paper to include a theoretical analysis of equation 4. In short, the output of an attention head ($o$) for a sequence with a prefix (of size $k$) proceeded by a set of repeated tokens (of length $n$) is:

$o = \sum_{i<k} \alpha_i v_i + \sum_{k <= i < n+k} \alpha _i v$

where $\alpha_i$ are the coefficients of the attention matrix (after softmax), thus: $\sum_{i}\alpha_i = 1$

$v_i$ is the value of token at index $i$, and $v$ is the value of the repeated token (from index $k$ to $n+k$)

RoPE affects key and queries, thus affecting the values $a_i$ but the values of repeats of the same token are fixed $v$.

For a long enough sequence the values of the repeated token will dominate the values of the prefix, this output will converge to $v$.

While we acknowledge the need for further investigation, our work establishes a crucial precursor to training data extraction. The identified mechanism of repeated token divergence leading to attention sinks creates the conditions under which the model is more likely to retrieve and potentially output memorized content.

“Do you have any intuitions on why certain repetition cannot lead to attention sinks / large norm of activations? The paper about massive activations [1] also show that besides the first token, several other tokens with low semantics could also induce the attention sink. Do you think this is related?”

We suspect certain repetitions does not lead to attention sinks for several reasons:

1. the context window is too short for the convergence to take place.
2. the circuit identifying the first token does not generalize to all models.

We suspect the model doesn't learn to identify <bos> but rather the first-token due to "packing", and the face that multiple <bos> tokens appeared in each context window, and believe that knowledge about the subtle training decisions will shed light on repetitions that do not lead to attention sinks.

We think that BoS is a token with no-entropy, since it is always the first token, it does not attend the rest of the sequence, it is updated in a fixed manner in each layer. The model leverages that to control and bias the attention function. We suspect that other tokens, especially those carrying little information about the context (conjunctions, such as: "the", "and", "but"), will be used by some attention heads to implement a no-op. This has been studied in [1].

[1] Outlier-Efficient Hopfield Layers for Large Transformer-Based Models, arxiv, 2024.

“Another weaknesses is that most of experiments are conducted using case studies. A systematic evaluation with metrics is lacking. Besides the attack success rate of mitigation approach I mentioned before, I find that when building the relationships between repeated tokens and attention sink, only some samples are provided. Why not include some statistical results to show the generalisation of this phenomenon?”

The causal relationship between the attention sink mechanism and the repeated token phenomenon is substantiated by our ablation experiments targeting specific 'sink neurons' (Section 3.2). Figure 3 provides empirical evidence that removing the contribution of these identified neurons effectively mitigates the high activation norms characteristic of repeated token divergence. This experimental approach, focusing on causal interventions within the identified circuit, is central to mechanistic interpretability and provides sufficient evidence to confirm the direct involvement of the attention sink mechanism in the observed phenomenon

We thank the reviewer for their valuable comments. Next, we address the reviewer's questions and concerns:

“The paper shows that not all tokens can effectively induce attention-sinks when repeated (Figure 5). This unexplained variability suggests additional factors at play that aren't fully captured by the current mechanistic account.”

As we mentioned in the limitations section, some additional factors can indeed be at play. Nevertheless, our goal is to understand why and how the previously identified phenomenon of model divergence due to repeated tokens, happens. Our current model is useful - it allows intervention that prevents the phenomenon, and therefore we believe that our analysis is valuable.

“While the paper identifies the mechanism behind repeated token divergence, it doesn't fully explain why this leads to training data extraction. The relationship between attention-sinks and the retrieval of memorized content remains unclear.”

While we acknowledge the need for further investigation, our work establishes a crucial precursor to training data extraction. The identified mechanism of repeated token divergence leading to attention sinks creates the conditions under which the model is more likely to retrieve and potentially output memorized content.

“The authors note that while there are shared motifs across LLMs, the first attention layer's specific behavior was unique to LLaMa2. This raises questions about how broadly applicable their exact mechanisms are across different model architectures.”

We suspect subtle differences in training, data or initialization induce different ways for the first attention layer to behave. However, it was important to us to show that by reverse-engineering one layer we can generate non-repeating sequences which also induce attention sinks.

“The validation of the patch focuses on model performance on standard tasks but doesn't directly measure its effectiveness in preventing the extraction of training data, which is the primary security concern.”

We have provided a few examples showing how coherency is maintained. It is rather difficult to measure training data leakage as most models are open-weights but not fully open-source, therefore there is very little information about their training data.

“In Claim 4.1 and its proof: the proof makes logical sense but relies on simplifying assumptions. It correctly identifies that with identical tokens, the value vectors in self-attention are equal. However, the proof doesn't fully account for the positional information encoded by RoPE: while RoPE affects keys and queries, the interaction between position-encoded representations could theoretically still allow differentiation between positions. The empirical observation in Equation 4 helps support this claim, but the theoretical argument could be more rigorous in addressing how RoPE specifically fails to distinguish positions in this context.”

We have revised the paper to include a theoretical analysis of equation 4. We argue that differentiation between positions of equivalent tokens is impossible with RoPE. In short, the output of an attention head ($o$) for a sequence with a prefix (of size $k$) proceeded by a set of repeated tokens (of length $n$) is:

$o = \sum_{i<k} \alpha_i v_i + \sum_{k <= i < n+k} \alpha _i v$

where $\alpha_i$ are the coefficients of the attention matrix (after softmax), thus: $\sum_{i}\alpha_i = 1$

$v_i$ is the value of token at index $i$, and $v$ is the value of the repeated token (from index $k$ to $n+k$)

RoPE affects key and queries, thus affecting the values $a_i$ but the values of repeats of the same token are fixed $v$.

For a long enough sequence the values of the repeated token will dominate the values of the prefix, this output will converge to $v$.

We thank the reviewer for their valuable comments. Next, we address the reviewer's questions and concerns:

“Rope affects the queries and keys, not keys and values, this claim: "thus symmetry between all tokens is preserved" shall be further clarified. Rope will affect attention output according to the relative positions, so even the repeated tokens will have different outputs, the claim: "all tokens in the repeated sequence will have the same output after the first attention layer." in line 299 is confusing.”

We agree and sorry for the confusion. We have revised the paper to include a proof to these claims. In short, the output of an attention head ($o$) for a sequence with a prefix (of size $k$) proceeded by a set of repeated tokens (of length $n$) is:

$o = \sum_{i<k} \alpha_i v_i + \sum_{k <= i < n+k} \alpha _i v$

where $\alpha_i$ are the coefficients of the attention matrix (after softmax), thus: $\sum_{i}\alpha_i = 1$

$v_i$ is the value of token at index $i$, and $v$ is the value of the repeated token (from index $k$ to $n+k$)

RoPE affects key and queries, thus affecting the values $a_i$ but the values of repeats of the same token are fixed $v$.

For a long enough sequence the values of the repeated token will dominate the values of the prefix, this output will converge to $v$.

“Although mentioned in the limitations, the first attention layer’s behavior was unique to LLaMa2, but the first attention layer mechanism poses an important finding in this paper, which hinders the generalization ability of this paper. How well this method can be used in other models is unknown.”

We suspect subtle differences in training, data or initialization induce different ways for the first attention layer to behave. However, it was important to us to show that by reverse-engineering one layer we can generate non-repeating sequences which also induce attention sinks.

“I wonder how will the mitigation influence the decoding efficiency as all sink neurons need to be figured out.”

The decoding efficiency after the mitigation is not significantly different from the original decoding efficiency. As can be seen by the pseudo-code (Listing 1), our patch changes one neuron for LLaMa2. Even with more neurons, the patch can be done in parallel on all of them as the set of neurons can be precomputed.

“The reason behind the high attention scores of the attention sinks and repeated tokens seems different. Repeated tokens are likely to have high activations because of self-attention between similar embeddings, but attention sinks are due to other different reasons. In addition, does the LLm distinguish attention sinks only by attention scores? Why can't model distinguish by positions?”

Our analysis indicates that the high attention scores observed in both phenomena are not due to distinct causes but rather stem from the same underlying neural mechanism. Specifically, the mechanism responsible for creating attention sinks (high attention on the initial token, crucial for fluency ) is inadvertently triggered by long sequences of repeated tokens.  

The core issue lies in the model's first attention layer, which struggles to differentiate a sequence of identical tokens from a single, initial token, particularly in architectures using RoPE. This layer misidentifies the repeated sequence, activating the same "sink neurons" that amplify the hidden states of initial tokens. Consequently, repeated tokens acquire high hidden-state norms, attracting disproportionate attention similar to the BoS token. While the model utilizes positional information (e.g., via causal masking ), this specific case of identical repetitions challenges its ability to distinguish position effectively, leading to the observed divergence.  

Therefore, our work demonstrates that repeated token divergence is mechanistically linked to the attention sink phenomenon, arising from a failure in positional differentiation under specific repetitive conditions. We hope this clarifies the connection established in our findings.

We thank the reviewer for their valuable comments. Next, we address the reviewer's questions and concerns:

“The main weakness of this work is that the task considered is very rare in the realistic application. Most users of LLMs will not let LLMs repeat some words. The lack of motivation discounts the importance of this work. It is hard to justify why people want the LLMs to repeat the token.”

The repeating tokens “task” is indeed not a common task. Nevertheless, once this behavior is triggered, it can cause a training data leakage – a serious security flow [1, 2]. Like many software vulnerabilities, the behavior that triggers the presented vulnerability is not a typical user input. Nevertheless, we assume here an adversarial user that aims to attack the model, and we find a patch that can protect the model from such an attacker.

[1] Nasr et al. Scalable extraction of training data from (production) language models, ICLR 2025

[2] Bye Bye Bye...: Evolution of repeated token attacks on ChatGPT models, https://dropbox.tech/machine-learning, 2024