Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings

We are grateful for the reviewer's thoughtful feedback highlighting areas that warrant further discussion and clarification. We address these concerns below.

---

Namely: what is the role of multiple attention heads? In particular, for a given input sequence, many attention heads are sinks, not just one.

Thank you for this insightful question. Our findings indicate that each attention head implements an independent instance of the catch, tag, release mechanism. Concretely, each head selects its own attention sink, captures a distinct subsequence of tokens, and injects a unique tag direction into those tokens’ representations via its value vector (potentially repeating this process for multiple sinks). In this sense, the sink location, the span of caught tokens, and the resulting tag are all specific to the head. While some heads may form sinks over overlapping regions, for instance, multiple heads attending to the BOS token, it is also common for sinks to appear at different positions, capturing different subsequences. This enables the model to segment and annotate the input in parallel, with each head providing a distinct structural or semantic signal. To illustrate this more concretely, the revised manuscript now includes a new figure showing that for a single prompt, two different heads induce distinct sinks corresponding to two different subsequences of the input.

---

How do the tags interact with each other? Is there constructive or destructive interference (or something more complicated), or are all the "tags" orthogonal?

Thank you for this excellent question. In Figure 2(c), each of the R, G, and B channels corresponds to a different top principal component of the attention head output. The emergence of clearly separated red and green coloring indicates that tokens tagged by different sinks project strongly onto different principal components. Since principal components are orthogonal by definition, this visual separation provides concrete evidence that the corresponding tags occupy distinct, non-overlapping directions in representation space.

While this is only a single example, it is representative of a broader trend we observe across models and prompts: tags created by different sinks tend to align with orthogonal directions, suggesting that the model avoids destructive interference by disentangling the subspaces in which tags reside. This supports the interpretation that tags from different sinks are not simply superimposed but are instead composed in a structured, non-interfering manner. To show this, we have measured the cosine similarity between distinct tags across all layers and heads showing that the tags typically exhibit very small cosine similarity implying that they are close to orthogonal. This will be displayed in a histogram which we present a summary of in the table below:

---

---

It is especially confusing since in Figure 2 the tokens later in the network seem to adhere strongly to the tags produced by the single discussed head earlier in the network, whereas multiple heads surely were attention sinks.

We appreciate the reviewer’s observation and agree that further clarification is warranted. While Figure 2(a) illustrates the behavior of a single attention head exhibiting a specific sink structure, in this particular example, other heads in the layer independently converge on the same sink structure. This results in the tags from multiple heads reinforcing one another and producing a coherent clustering pattern deeper in the network, as shown in Figure 2(d).

To clarify this, we have revised the caption of Figure 2(d) to read:

Tagged tokens, created by multiple attention heads that independently exhibit the same sink structure, propagate through the residual stream and cluster in a deeper layer based on their associated sink.

However, we emphasize that this alignment across heads is not guaranteed and does not always occur. To illustrate a contrasting case, we have added a new figure where different heads create distinct sinks, resulting in more heterogeneous tagging across heads. We hope this comparison clarifies that the example in Figure 2 is representative of one possible, and particularly clean, case rather than a general rule.

---

What is the role of multiple attention heads… in the synthetic setups considered in the work?

We deliberately designed a simple synthetic task that can be solved using a single attention head to isolate and study the catch, tag, release mechanism. A single head is sufficient in this case because it can form one attention sink and tag a single subsequence starting from a specific source token. However, this setup limits the model to tagging only one subsequence per source token. To tag multiple subsequences that all begin at the same token but end at different positions, multiple attention heads are needed -- each forming its own sink and injecting a distinct tag. For instance, a more complex variant of the task could require the model to extract and average several subsequences (e.g., tokens 3–7, 3–15, and 3–28, all starting from token 3) and then combine those averages. This would naturally encourage the use of multiple heads, each capturing and processing a different subsequence.

---

There is little-to-no discussion of applications to the practice of LLMs.

We have added the following paragraph discussing various applications of the mechanism:

The presence and utility of the `catch, tag, and release’ mechanism have important implications for practical use of large language models. Section 4 shows the potential for tags to be used to steer activations [1] which can be applied for model alignment and safety [2]. In prior work [3], attention sinks were linked to repeated token phenomena and jailbreaking, suggesting the potential for the mechanism to be used for both exploit detection and mitigation. In model compression, preserving this mechanism may be critical, as we hypothesize it can be captured within a low-rank subspace of the model’s parameters, which must be retained during pruning or quantization to maintain performance [4,5]. Furthermore, recent work, [6], has identified low-entropy tokens as central to LLM reasoning abilities, likely overlapping with the tag tokens we highlight, indicating this mechanism may play a foundational role in LLM reasoning.

---

What is the role of other non-attention components in the network (MLP, ...) in this mechanism?

We have added the following paragraph to discuss the potential role of the MLP in the transformer layers in light of the catch, tag, release mechanism:

Prior theoretical work studying transformers from a dynamical systems perspective has shown that groups of token representations tend to collapse to low-dimensional subspaces, or even single points, across layers [7]. This raises the question: which tokens collapse together, and what determines the target of this collapse? We hypothesize that the catch, tag, release mechanism plays a central role in both the grouping and the collapse target. Specifically, tokens attending to the same attention sink receive a common tag, which defines the direction and destination of their collapse. We further propose that the MLP layers progressively drive the collapse across layers by reinforcing the shared tag structure, effectively denoising the token representations toward the shared tag.

We also envision this mechanism being used in practice by researchers and practitioners to better understand and intervene in model behaviour. For example, one could input a prompt, extract the tags assigned to tokens, and highlight, for each tag, the tokens grouped by it. This would reveal how the model segments and jointly processes subsequences of input, offering a transparent window into its internal computation.

---

We sincerely thank the reviewer once again for highlighting areas of ambiguity and missing discussion in our manuscript. If the reviewer feels that our revisions have sufficiently addressed these concerns, we would be grateful for any reconsideration of the score.

[1] Turner et al. (2023). Steering Language Models With Activation Engineering.

[2] Wang et al. (2023). Trojan Activation Attack: Red‑Teaming Large Language Models using Activation Steering for Safety‑Alignment.

[3] Yona et al. (2025). Interpreting the Repeated Token Phenomenon in Large Language Models.

[4] Zhang et al. (2024). OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition.

[5] Makni et al. (2025). HASSLE-free: A Unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs.

[6] Wang et al. (2025). Beyond the 80/20 Rule: High‑Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.

[7] Geshkovski et al. (2023). A mathematical perspective on Transformers.

Thank you for your review and for providing additional ideas and suggestions as to how we can extend our experiments.

---

Are there other kinds of semantic or syntactic information encoded by these tags? For instance, in narrative text, could tags encode sentiment…

Indeed, the tagging mechanism we explore can potentially capture a broader range of semantic and syntactic information beyond just binary True/False classifications. To show this explicitly, we extended our experiments to assess whether tags can encode sentiment. Specifically, we construct prompts in the following format:

it's a charming and often affecting journey. This statement is: POSITIVE. 
unflinchingly bleak and desperate. This statement is: NEGATIVE. 
[SENTENCE]. This statement is:"

where [SENTENCE] is replaced with various examples from the SST-2 dataset. Using the same experimental setup described in Section 4, we evaluated whether tags could capture sentiment polarity. The results, presented in the table below, show that tags indeed encode positive and negative sentiment information effectively, supporting the broader applicability of the method to a variety of different semantic tasks.

---

Are there specific types of attention heads or MLP neurons in subsequent layers that specialize in processing tokens based on the direction of these tags?

This is an excellent and non-trivial question. In principle, one could examine the cosine similarity between tag embeddings and individual neurons (i.e., rows in the MLP weight matrices of subsequent layers) to identify whether certain neurons respond selectively to tags. However, as discussed in [1], it is unlikely that features, such as tags, are cleanly localized to individual neurons. Instead, model features tend to be superimposed across many directions in activation space due to the model’s limited dimensionality relative to the complexity of the data it learns to encode.

In this view, the directions corresponding to tags may not be easily detectable when analyzing individual neuron activations. Instead, we hypothesize that the representation of a tag is distributed and likely spans multiple directions in the activation space, potentially spanned by the top singular vectors of the relevant weight matrices. This suggests that tag-related information is embedded across a subspace rather than isolated to specific neurons.

Probing this hypothesis would require more advanced interpretability techniques—for example, analyzing representations in the singular vector basis, using sparse autoencoders, or applying subspace probing tools from mechanistic interpretability to recover the directions associated with specific functions. We believe this is a promising direction for future research.

[1] Toy Models of Superposition https://transformer-circuits.pub/2022/toy_model/index.html

---

How "clean" is this separation in practice? Does the tag component have minimal projection onto the token's original semantic direction?

We conducted additional experiments to better understand how distinct the tag component is from the token's original semantic representation. Specifically, we measured the cosine similarity between the tag (after being mapped back into the residual stream) and the token representations of non-sink tokens to which the tag is applied, quantifiying the degree of overlap between the tag direction and the token’s inherent semantics. We averaged the results over 200 prompts of 1024 tokens each, and across all layers of the model. As shown in the table below, the cosine similarity is consistently low, especially in the LLaMA models, which suggests that the tag subspace remains largely orthogonal to the native token representations. This indicates a relatively "clean" separation between the semantics introduced by the tag and the token's original meaning.

---

How sensitive are the quantitative results, such as the average number of sinks (Figure 3a) and the variance explained (Figure 3b), to the choice of this hyperparameter?

To assess how sensitive our results are to this design choice, we conducted a comprehensive sensitivity analysis. Specifically, we varied the threshold across twelve values ranging from 0.03 to 0.4 and evaluated the impact on two key metrics: (1) the average number of identified sinks and (2) the average variance explained by the tags. We will include a plot in the appendix where each point corresponds to a threshold, the x-axis represents the average number of sinks, and the y-axis reflects the variance explained.

Across all models tested, we find that the resulting curves are typically logarithmic-like: initially, decreasing the threshold adds new sinks and increases explained variance, but this gain saturates beyond a certain point. This suggests that smaller thresholds tend to capture less meaningful or redundant tokens. The table below provides representative values from this analysis to support this observation.

---

We are very grateful to the reviewer for their thoughtful comments, constructive suggestions for future work, and positive evaluation of our submission. We hope that our responses and additional results further clarify the contributions and are viewed as a meaningful step forward.

We would first like to thank the reviewer for their thorough and thoughtful analysis of the manuscript, its appendices, and the supplementary material, which we have leveraged to improve the overall quality of our work.

---

The analysis of models with QK normalization is limited, ... the paper does not verify whether the "catch, tag, release" mechanism is still present in these models.

We agree with the reviewer that the omission of measurements relating to models with QK norm was an oversight in our original manuscript. To provide evidence that the 'catch, tag, release' mechanism is still present in models with QK norm, we have now added measurements for Qwen 3 8B, and 14B showing their sink count (using violin plots), variance explained by tags (using box and whiskers plots), as presented in Section 3.2 Figure 3. We summarize these results in the table below, which averages across all attention heads.

Additionally, we also executed the probing experiments provided in Section 4 on the Qwen 3 models which are presented in the table below:

To summarize, the above results follow the same trends as the models without QK norm, suggesting that the 'catch, tag, release’ mechanism is still present in these models.

---

The authors do not go further to explain how this increased prevalence contributes to the models' reasoning abilities.

We thank the reviewer for highlighting the need for a deeper analysis supporting Claim A3. In response, we have added a new experiment to Section 5 that includes the most frequent attention sink tokens in both reasoning-tuned and base (non-reasoning) versions of the Qwen 2.5 14B model:

This table reveals not only a quantitative increase in sink prevalence, but also a qualitative shift in which tokens form sinks, indicating a meaningful structural adaptation in how the 'catch, tag, release' mechanism operates under reasoning fine-tuning as we describe in the additional paragraph that will be added to the manuscript:

In base models such as Qwen 2.5 14B, sink formation predominantly occurs around function words (e.g., Ġthe, Ġto, ĠI) and punctuation (such as . or ,), likely due to their high frequency and syntactic roles. In contrast, the DeepSeek Qwen 14B reasoning-tuned model forms attention sinks around semantically significant or task-structuring tokens, such as:

• IMAGE suggests segmentation for multimodal inputs, grouping the token with subsequent image-related descriptions.
• Doctor likely marks named entities relevant for reasoning over domain-specific content.
• Tokens like MON, CAP, IT, and GR are plausible abbreviations or categorical labels (e.g. weekdays, captions, grades) that suggest segmentation for structured data.
• SP may represent speaker or section markers, important in multi-step or dialog-based reasoning.

These sink tokens not only appear with high frequency but also explain a substantial portion of the variance, indicating their importance in structuring the model’s internal representation. The presence of such tokens, absent in the base model’s sink list, suggests that reasoning fine-tuning reorients the attention sink mechanism from syntactic attractors to semantically meaningful or task-specific units.

This new evidence strengthens Claim A3 by showing that attention sinks in reasoning-tuned models are not just more frequent, they are repurposed to segment, tag, and organize input in ways that directly support reasoning behaviors such as modality transitions, entity tracking, and logical clause separation. The 'catch, tag, release' mechanism thus adapts in both frequency and function to facilitate reasoning.

We also now explicitly acknowledge in the Limitations section that this connection remains observational:

While our findings in Section 5 strongly suggest a structural role for sinks in reasoning, they do not yet establish a causal link between the two. This remains an important direction for future work.

---

In line 580, the equation for the attention weight... defines the denominator as a sum over all keys from $k=1$ to $T$ .

Thank you for catching this error. The summation should indeed be from $k=1$ to $k=i$ instead of from $k=1$ to $k=T$ (which would change the first two lines of that derivation). However, this does not impact the later two equations in that derivation so the ultimate limit and result is still correct. We have also carefully gone over all other equations in the proof to ensure that this typo does not occur elsewhere.

---

Unable to reproduce the clear averaging behaviour shown in Figure 7(c)

If no clear averaging behavior emerges, this indicates that optimization failed to converge to a low-loss solution. We have observed this same issue in our own experiments. However, in all successful runs that do converge, we consistently observe the emergence of the 'catch, tag, and release’ mechanism and the clear averaging shown in Figure 7(c). To clarify these issues, we have added the following paragraph to the limitations section in Appendix B:

Although the theoretical model is simple, the optimization process does not always reach a low-loss solution. We found that convergence depends on how the $s_{\text{tag}}$ parameter is initialized, with larger initial values generally leading to more consistent success. The table below shows how often the model achieved a high fit ($R^2 > 0.95$) across 10 runs for different starting values of $s_{\text{tag}}$:

---

The learnable parameter, initialized at 10 as per Appendix D, does not grow large and instead decreases

The theoretical result provides sufficient conditions for the emergence of the mechanism, but these are not necessary. Empirically, we observe that the mechanism can still emerge under different dynamics. For instance, instead of $s_{\text{tag}}$ diverging while the weights remain fixed, it is also possible for $s_{\text{tag}}$ to remain fixed while the weights shrink. Fully characterizing all such cases is non-trivial. However, the key point is that the 'catch, tag, release' mechanism consistently emerges in this toy setting. Moreover, $s_{\text{tag}}$ is initialized to a value already $10\times$ larger than the second coordinate of the number tokens, which may reduce the need for it to grow further.

---

There is a typo in line 622

Thank you for bringing this to our attention, we have since revised the appendix to correct this typo from -10 to 10.

---

We extend our gratitude to the reviewer for their careful analysis of our paper and for highlighting issues that we have overlooked. Should the reviewer think that we have taken sufficient action to address their concerns, we would be most appreciative of any reconsideration of their score.

We thank the reviewer for their thoughtful review and for offering several valuable suggestions to enhance the comprehensiveness of our analysis.

---

Could you examine which tokens become sinks, whether they equally contribute to explained variance, and whether this changes after reasoning fine-tuning? Possible to create a taxonomy, using bigger and different dataset?

We think this is a great suggestion and have executed additional experiments that count the frequency of tokens that appear as sinks and measure the average explained variance of each token when it appears as a sink. To this end, we analyzed 200 prompts of length 1024 from the QuALITY dataset, using both the base and reasoning-tuned versions of the Qwen 14B model. The results, which are representative of broader trends we have observed, are now included in Section 5 and summarized in the table along with a paragraph for discussion below:

In base models such as Qwen2.5-14B, sink formation predominantly occurs around function words (e.g., Ġthe, Ġto, ĠI) and punctuation (e.g., . or ,), likely due to their high frequency and syntactic roles. In contrast, the DeepSeek-Qwen-14B reasoning-tuned model forms attention sinks around semantically significant or task-structuring tokens, such as:

• IMAGE suggests segmentation for multimodal inputs, grouping the token with subsequent image-related descriptions.
• Doctor likely marks named entities relevant for reasoning over domain-specific content.
• Tokens like MON, CAP, IT, and GR are plausible abbreviations or categorical labels (e.g., weekdays, captions, grades) that suggest segmentation for structured data.
• SP may represent speaker or section markers, important in multi-step or dialog-based reasoning.

These sink tokens not only appear with high frequency but also explain a substantial portion of the variance, indicating their importance in structuring the model’s internal representation. The presence of such tokens—absent in the base model’s sink list—suggests that reasoning fine-tuning reorients the attention sink mechanism from syntactic attractors to semantically meaningful or task-specific units.

---

The threshold-based sink definition is arbitrary, based on previous studies. Could you provide some intuition why this metric is indeed select all attention sinks

We agree that this is a limitation of our current methodology and that the threshold we adopt, based on prior work, does not guarantee the identification of all tokens functioning as attention sinks. To acknowledge this, we have added the following paragraph to the Limitations section (Appendix B):

The threshold-based metric used to identify attention sinks does not guarantee full coverage of all such tokens. Our choice of a 0.2 threshold, while consistent with prior studies, is not necessarily optimal. As discussed in [1], there is currently no principled method for determining this threshold.

To assess how sensitive our results are to this design choice, we conducted a comprehensive sensitivity analysis. Specifically, we varied the threshold across twelve values ranging from 0.03 to 0.4 and evaluated the impact on two key metrics: (1) the average number of identified sinks and (2) the average variance explained by the tags. We will include a plot in the Appendix where each point corresponds to a threshold, the x-axis represents the average number of sinks, and the y-axis reflects the variance explained.

Across all models tested, we find that the resulting curves are typically logarithmic-like: initially, decreasing the threshold adds new sinks and increases explained variance, but this gain saturates beyond a certain point. This suggests that smaller thresholds tend to capture less meaningful or redundant tokens. The table below provides representative values from this analysis to support this observation.

---

… compare to alternatives (e.g., entropy or KL divergence)

We initially considered using alternative metrics such as entropy or KL divergence to identify attention sinks. However, attention sinks are defined over columns of the attention probability matrix, that is, a sink corresponds to a token that consistently receives high attention across multiple rows. Since the columns are not normalized, the sink column does not represent a valid probability distribution. As such, entropy or KL divergence cannot be meaningfully computed over columns.

Likewise, using row-wise entropy to detect focused attention would be unable to capture attention sinks. Low entropy across rows merely indicates that each token is attending strongly to a few others, but it says nothing about which tokens are being attended to. For example, if the attention matrix is close to diagonal, each row has low entropy, yet no token consistently receives attention—i.e., no sink exists. For this reason, we opted for a threshold-based metric that directly evaluates whether certain tokens receive consistently high aggregate attention, better aligning with the conceptual definition of attention sinks.

---

How could it be that accuracy is still 50% after adding tags?

We have double-checked the data, and the result is indeed accurate for that specific attention head. One possible explanation is that the tag portion of the activation has a significantly smaller norm compared to the non-tag portion. As a result, the classifier may effectively ignore the tag component due to its negligible contribution, leading to no improvement in accuracy despite the presence of tags.

---

$\theta_{activation}$ is not defined, would be better to state it in the text

Thank you for pointing out a potential point of unclarity. We have included in the Appendix the full equations and descriptions of the probe.

---

We thank the reviewer for suggesting several valuable directions for additional measurements which have enhanced the comprehensiveness of our analysis. If the reviewer finds that these additions strengthen the paper, we would be grateful for any reconsideration of the score.

[1] Gu et al. (2024). When Attention Sink Emerges in Language Models: An Empirical View.