What are you sinking? A geometric approach on attention sink

We thank the reviewer for the accurate review.

$**Regarding the "minor notes":**$

• $\textsf{Out of date citations}$: Thank you for catching this. We will update all citations, including Moschella et al. (2023) and Xiao et al. (2024), to their correct, published versions.
• $\textsf{Prior work on sink-like behavior}$: While Xiao et al. coined the term "attention sink," the underlying behavior was indeed observed in earlier foundational work on BERT interpretability. We will absolutely add citations to Clark et al. (2019) and Kovaleva et al. (2019) in our related works section to provide a more complete historical context.
• $\textsf{Figure 1 visualization}$: Thank you for pointing out this potential confusion. The visualization was intended to show that connections in Centralized and Distributed frames are not layer-specific, unlike the Bidirectional frame. We will revise the figure and its legend to make this distinction clearer.

$**Regarding weakness 1: lack of background references**$

We will add a new section in the Appendix that provides an introduction on the key concepts from information geometry and topological data analysis.

$**Regarding weakness 1: study on static checkpoints**$

This is a key limitation that we acknowledge. A full analysis throughout the entire training process is computationally prohibitive for models of this scale. Your suggestion to study smaller models is very good and our Random Matrix Theory analysis on Pythia checkpoints (in Appendix B, due to space constraint) is a first step in this direction, showing that reference frames emerge in the earliest stages of training. We will highlight this finding more prominently and explicitly state in our limitations section that a more granular temporal study, perhaps on smaller, more efficiently trainable models, is a crucial direction for future work.

$**Regarding weakness 1: limited Scope (LLMs only)**$

Our work focuses on LLMs as this is the primary domain where attention sinks have been formally studied. However, we agree that our geometric framework is general and likely applicable to any Transformer-based architecture. We thought that this paper was rather dense in information already, so we decided to focus on text and save the other modalities for a possible future work.

$**Regarding question 1 and 2: causality (Softmax + Positional Embeddings) and "not an architectural phenomenon"**$

We agree our wording could be more precise.

• Causality: You are absolutely correct. Our central argument is that sinks emerge due to the interplay between the softmax function and the inductive biases of position encoding. The softmax creates the geometric pressure for sparsity on the probability simplex (as discussed in Appendix A), and the position encoding provides the specific bias that determines where these sparse points (sinks) will form. This is indeed a core finding, and we will revise the abstract and introduction to state this two-part mechanism more explicitly.
• "not an architectural phenomenon": Your critique of this sentence is fair, when we wrote this, we meant that sinks are not a bug or an arbitrary quirk specific to one model implementation (e.g., LLaMA vs. Mistral). Rather, they are a fundamental, predictable consequence of the core components of the Transformer architecture itself. In that sense, they are an architectural phenomenon. We will revise our language to be more precise, stating that reference frames are a fundamental geometric principle within the Transformer architecture, emerging predictably from the interaction of its core components. Thank you for pushing for this clarification.

$**Regarding question 3: interpretation of Moschella et al. (2023)"**$

Thank you for this very careful reading. Moschella et al. propose using relative representations to establish a shared coordinate system, they do not claim that such frames emerge naturally in standard training. Our intention was to cite their work as providing the conceptual inspiration for why reference frames are such a powerful and necessary structure in high-dimensional spaces, which motivated our search for their emergent counterparts. We will revise line 56 and related text to state that Moschella et al. demonstrated the utility and proposed the creation of such frames, which provided a key motivation for our work.

$**Regarding question 3: reference for performance degradation"**$

The primary reference for performance degradation upon removing the initial tokens (the attention sink) is the original paper by Xiao et al. (2024), "Efficient Streaming Language Models with Attention Sinks". They demonstrate that for streaming applications, where a fixed-size cache is used, preserving the attention sink tokens in the cache is crucial for maintaining model performance. We will add this citation explicitly at line 17. We also agree with your nuanced view: our claim is not that any architecture designed without sinks is inferior. Rather, simply removing the sink tokens post-hoc from a model that was trained to rely on them is detrimental. Designing an architecture from the ground up that accomplishes the same geometric stabilization without a sink is an exciting research direction. We will clarify this important distinction in the text.

$**Regarding question 3: definition of sinks"**$

$\textsf{On the use of thresholds and the 90th percentile}$

We acknowledge the reviewer's concern that any threshold-based definition carries a degree of arbitrariness. Our approach was guided by two principles: aligning with the conceptual understanding of the phenomenon and choosing a conservative measure to ensure the robustness of our findings.

• The term "attention sink" implies a token that attracts a disproportionately large share of the total attention budget. Our definition aims to capture this by looking for attention weights ($\alpha_{ij}$) that are outliers in the global distribution. The 90th percentile for $\tau$ was chosen as a standard statistical convention to isolate these extreme values, the top 10% of all attention links. This ensures we are not just flagging moderately high attention, but truly exceptional concentrations of it.
• The role of the frequency threshold ($\gamma$): We want to emphasize that the $\tau$ threshold does not operate in isolation. The second threshold, $\gamma$ (frequency), is critical and directly addresses the reviewer's point about consistency. A token is only identified as a sink if it exceeds the high attention threshold $\tau$ for a large fraction ($\gamma \ge$ 0.3) of all source tokens. This combination ensures that we identify tokens that are not just attended to strongly by one or two queries, but that act as consistent, sequence-wide attractors. While we did not conduct a formal false-positive study, this dual-threshold requirement is designed to be highly resistant to false positives; a token must be both an outlier in attention weight and highly consistent in its role as an attractor.

$\textsf{On the proposed alternative definition}$

The reviewer's suggestion to rank attention weights for each query and look for tokens that consistently appear in the top P% is an excellent and valid method for identifying tokens that are consistently important to other tokens. We believe this method would be highly effective for identifying stable semantic or syntactic relationships. However, we chose our definition because it is tailored to identify a slightly different phenomenon that is central to our paper's thesis: the existence of universal geometric anchors. The reviewer's method identifies tokens that are consistently high-rank. For example, for the query "cat," the token "mouse" might consistently be in the top 5% of attended-to tokens. This captures relative importance.

Our method identifies tokens that receive a high absolute quantum of attention, regardless of rank. A token could be the #1 attended-to token for every query, but if the attention distribution is very flat, its absolute weight might never cross the 90th percentile threshold. Conversely, a token like [BOS] might consistently receive 30% of the attention from every other token. It would always be highly ranked, but more importantly, it consumes a disproportionate, absolute share of the probability mass.

The idea is that attention sinks function as reference points in a coordinate system. For this, we need to find the "gravitational wells" of the attention map, like the points that command a large, absolute portion of the attention budget from across the sequence. Our definition, by focusing on outlier absolute weights, is specifically designed to locate these universal anchors. The reviewer's proposed method, while very good, is better suited for analyzing pairwise token importance.

We will add a note in the appendix clarifying the rationale for our definition in contrast to other valid alternatives, like the one proposed.

$**Regarding question 4: effects of softmax modifications"**$

Standard softmax creates the pressure for the kind of sparsity we observe in classic attention sinks. Modifying this function would not remove the fundamental need for a reference frame, but would change its implementation. Approaches like Softpick or sparse-softmax enforce a "harder" sparsity, often forcing the model to select a small, discrete set of tokens to attend to. This would prevent the formation of a "lazy" sink where one token passively collects a soft distribution of attention from all others. Instead, we hypothesize that these methods would force the model to implement a more dynamic reference frame. For any given query, the model would be forced to make a hard choice, selecting a small set of tokens to receive 100% of the attention. These tokens would become explicit, context-dependent reference points for that specific computational step.

We thank the reviewer for the accurate review.

$**Regarding weakness 1: complexity and necessity of the analyses**$

Our goal was to show that the evidence for reference frames is not an artifact of one particular analytical lens, but a consistent signal across topological, spectral, information-theoretic, and vector-space perspectives. In our revision, as detailed in our answer to Question 3, we will improve the methodology section to more clearly motivate why each tool was necessary to answer a specific research question, thereby guiding the reader through our analytical narrative.

$**Regarding weakness 2: scalability to larger models**$

Our analysis was computationally constrained to models up to ~12B parameters. However, as we elaborate in our answer to Question 1, our core theoretical argument is based on architectural "primitives" (softmax and position encodings) that are scale-invariant. We therefore hypothesize that our findings will generalize, and perhaps even become more pronounced, in larger models. We will explicitly state this in the limitation section.

$**Regarding weakness 3: theoretical vs. practical contributions**$

Our paper moves the conversation about attention sinks from "bug" to "feature," opening up new avenues for deliberate architectural design and training strategies. We outline several concrete examples of how this theoretical insight can translate into practical applications in our answers to Question 2 and Question 4.

$**Regarding weakness 4: the density of the presentation**$

We agree that the paper's density could be a barrier. We will add more explanatory text to contextualize the results in our tables (as discussed in our response to Reviewer uQoB) and restructure Section 4 to better motivate our methodology. We will also add a section to the appendix to provide more background on the key concepts from topological data analysis and information geometry.

$**Regarding question 1: scaling on bigger models**$

Unfortunately for this project we didn’t have the resources to test on models of that size. But, while our direct empirical analysis was computationally constrained to models up to ~12B parameters, we have theoretical reasons to believe our findings will generalize to much larger models.

• Our core thesis is that reference frames are not an emergent property of scale, but a fundamental consequence of architectural primitives. The geometric principles we identify, the simplex constraint of softmax and the mathematical properties of position encodings, are intrinsic to the architecture, regardless of parameter count. A 70B LLaMA-family model still uses standard RoPE, which mathematically privileges the first token. Therefore, we hypothesize it will continue to exhibit a centralized reference frame. In fact, as model capacity and the complexity of the learned representations increase, the need for a stable, efficient coordinate system to anchor the high-dimensional manifold likely becomes more critical, not less.
• Empirical observations from the community regarding larger models appear to support this. For instance, strong, persistent attention to the [BOS] token is a widely noted phenomenon in very large decoder-only models. Our framework provides a formal explanation for these observations.

So, we acknowledge that direct empirical validation on 70B+ models is an important next step. We will add to the paper that while our theory predicts scalability, this remains a hypothesis to be confirmed by future work with greater computational resources.

$**Regarding question 2: how the knoledge about the reference frame could be used in practice**$

Our findings suggest that the choice of position encoding is, in effect, a choice of geometric inductive bias. This allows for more principled architecture design. For example:

• For tasks requiring robust long-range dependencies anchored to a consistent context (e.g., summarization, few-shot prompting), an architecture promoting a centralized frame (like standard RoPE) may be optimal due to its efficient, hub-and-spoke information routing.
• For tasks requiring high contextual flexibility and sensitivity to local syntax (e.g., creative writing, complex reasoning), an architecture promoting a distributed frame (like NTK-scaled RoPE) might be better, as it creates a more adaptable, locally-aware coordinate system. This allows designers to match the geometric properties of the model to the geometric properties of the task.

Knowing that models dedicate significant capacity to forming reference frames (as shown by our Fisher Information analysis), we could devise strategies to accelerate this process. For example, a pre-training or warm-up phase could use a curriculum or a specific regularization term to explicitly encourage the formation of a stable reference frame, potentially leading to faster convergence on the primary task.

$**Regarding question 3: is the analysis redundant?**$

We view our analysis not as redundant, but as providing a form of cross-validation, where each method offers a unique and necessary piece of the puzzle. The complexity arises from the need to build a robust case from different angles.

Here is a breakdown of why each component was essential:

• Topological Analysis (Persistent Homology): This was necessary to understand the high-level shape of the attention graph. It answered the question: "What is the global connectivity structure?" For example, the discovery of high $Betti_1$ values (loops) in early layers was a unique signature that immediately distinguished bidirectional frames from the star-like ($Betti_0 \approx$ 0) structures of decoders.
• Spectral Graph Theory: This analysis quantified the properties of the connectivity structures revealed by topology. It answered: "How is information concentrated and distributed within that shape?". Metrics like the Fiedler value and degree centralization allowed us to quantitatively differentiate between centralized (single dominant node) and distributed (multiple smaller nodes) frames. The sign-flipping correlation pattern we identified became a key quantitative signature.
• Information Geometry (KL & Fisher): This moved beyond structure to function. It answered: "How critical are these reference points for the model's information processing?" The KL reduction experiment directly measured the functional cost of removing sinks, proving their importance. The Fisher information analysis revealed where in the network (i.e., which layers) the model invests its learning capacity to build these frames, highlighting the early-layer focus of centralized frames versus the mid-layer focus of bidirectional ones.
• Value Space Analysis: This final piece connected attention patterns to their ultimate purpose: geometric transformation. It answered: "How do these reference points actually orient other tokens in the value space?" The directional influence metric was crucial here, showing that reference tokens act as coordinate axes that actively guide other representations, rather than just passively aggregating information.
• Random Matrix Theory Analysis (in the Appendix): it clarified that the reference frames emerge in the first stages of the training, showing that the reference frames aren't just an artifact, but a useful feature to achieve proper processing.

We will, however, take the reviewer's feedback to heart and revise the methodology section (section 4) to more clearly state the unique question each analysis answers, and better justifying the inclusion of each and guiding the reader through the logic.

$**Regarding question 4: potential methods or experiments for "reference frame engineering"**$

Building on our answer to Question 2, we can envision several concrete experimental ideas:

• Explicit anchor tokens: it could be possible to introduce new, non-semantic "anchor" tokens into the vocabulary. During training, a regularization term could be added to the loss function to explicitly encourage (or discourage) attention to these tokens, effectively creating programmable reference points. An experiment could test if a model can be trained to use [ANCHOR_1] as a centralized sink and [ANCHOR_2] as a secondary, distributed sink.
• Attention head specialization: Our analysis shows that specific heads specialize in attending to reference points. it could be possible to perform experiments where, during fine-tuning, these specialized heads are either frozen (to preserve the original geometric structure) or selectively pruned/retrained (to force the model to adopt a new geometric strategy). This could be a powerful tool for efficient task adaptation.

Once again, we thank the reviewer for their deep engagement with our work. Their feedback has been valuable in helping us clarify our contributions and strengthen the paper.

We thank the reviewer for the review.

$**Regarding weakness 1: lack of experiments to prove causation**$

We agree that a perfectly controlled experiment, training multiple large-scale models from scratch where only the position encoding scheme is varied, would provide the most definitive proof of causation. However, the huge computational cost is beyond the resources available for this work.

Instead, we sought to build a strong case for a causal link through two complementary lines of evidence:

• Consistency across many architectures: our study was designed to show that the link between position encoding and reference frame type is not an artifact of a single model but a consistent principle. We demonstrate that all analyzed models using standard RoPE (LLaMA 3.1/3.2, Mistral, Gemma) develop centralized frames, all models with NTK-aware scaled RoPE (Qwen 2.5, Phi-2) develop distributed frames, and all bidirectional encoder models with absolute embeddings (BERT, XLM-RoBERTa) develop bidirectional frames. The consistency of this pattern across different model families, sizes, and training datasets provides robust correlational evidence that strongly suggests the position encoding scheme is the primary causal factor.
• A mechanistic explanation: our paper provides a direct mathematical explanation for why this causal link exists. In Section 3.4, we explain the underlying mechanism for each encoding type. For instance, we explain that standard RoPE's formulation results in an identity matrix transformation for the first token ($R_0 = I$), giving it a unique, privileged computational status that naturally leads to its emergence as a centralized reference point (lines 167-169). Similarly, we explain how the scaling factor in NTK-aware RoPE diminishes this advantage, creating the conditions for distributed frames (lines 181-184).

While we did not perform the interventional experiment suggested, we believe our combination of consistent, large-scale observational evidence and a clear, mathematically-grounded causal mechanism provides a compelling argument for the claims made. We will, however, add a note to the limitations section acknowledging that this type of controlled experiment would be a valuable direction for future research.

$**Regarding weakness 2: lack of theoretical explanation on the emergence of the phenomenon**$

We respectfully believe that our paper does propose a fundamental theoretical explanation, which is rooted in the interaction between two core principles of the transformer architecture. Perhaps we can state it more directly here and ensure it is properly written in the final version.

Our theoretical argument is that reference frames (and thus attention sinks) emerge as an optimal solution to a geometric problem created by the interplay of:

• The information geometry of the softmax function: as explained in section 3.2 (and further in Appendix A), the softmax function constrains attention outputs to the probability simplex ($\Delta^{n-1}$). We argue that this is not just a normalization step but a fundamental geometric constraint. This constraint, governed by the Fisher information metric, creates a curved representational manifold that privileges sparse distributions, that is, it creates a mathematical pressure to concentrate attention probability mass onto a small number of tokens. This is the general force that necessitates the creation of "sinks."
• The inductive bias of position encodings: while the softmax creates the pressure to form sinks, the specific mathematical structure of the position encoding scheme provides the inductive bias that determines the pattern of these sinks. Section 3.4 is dedicated to this point, providing a detailed theoretical account for each reference frame type:

1. Centralized frames emerge because standard RoPE's math gives a unique computational advantage to the first token.
2. Distributed frames emerge because scaled RoPE modifies this math to weaken the first token's advantage, allowing other tokens to serve as local anchors.
3. Bidirectional frames emerge because absolute position embeddings explicitly inject position markers that the encoder architecture can leverage as stable start and end anchors.

So, our theory says that attention sinks are not an accident, but a necessary geometric feature. The softmax creates the "reason" (a need for sparse, stable reference points on the probability simplex), and the position encoding provides the "way" (the specific geometric biases that shape the final structure).

We will revise the introduction and abstract to make this two-part theoretical argument more explicit.

We thank the reviewer for the accurate review.

$**Regarding weakness 1 and 3: info dump and missing simple baselines**$

We agree with the reviewer's opinion. In our effort to be comprehensive, the presentation in sections 4 and 5 became overly dense.

Our Plan for Revision:

We will revise Section 4 (Methodology) to explicitly frame our analysis around a series of research questions. This will guide the reader and justify why each tool was necessary, transforming it from an "information dump" into a clear investigative path. The structure will be:

• RQ1: What is the global shape of the attention graph? -> Tool: Topological Analysis (Persistent Homology, Betti numbers).
• RQ2: How is information concentrated within that shape? -> Tool: Spectral Graph Theory (Fiedler value, centralization).
• RQ3: How functionally critical are these structures? -> Tool: Information Geometry (KL Divergence, Fisher Information).
• RQ4: How do these structures orient representations in value space? -> Tool: Value Space Analysis (Directional influence, etc.).

As suggested, we will add a new subsection in Appendix (due to space constraints) dedicated to a more detailed, foundational analysis of attention sink patterns. This new section will include the requested visualizations, showing the distribution of attention sinks for each reference frame type across our test dataset.

$**Regarding weakness 2: metric values are not contextualized for readers.**$

This is a fair and important criticism. We thank the reviewer for pointing this out. To stick to page limits, the tables in the main body of the paper present a high-level summary of our findings. We would like to note that the complete, detailed tables for all experiments are provided in the appendix for full transparency.

That said, we agree that the summarized values in the main text must be better contextualized to be interpretable. We will revise Section 5 to ensure that for each value presented, we explicitly explain its significance, for example what constitutes a "high" versus "low" value and why the observed numbers support our conclusions.

Example for Table 1: When discussing $Betti_1$ values, we will add a clarification: "A $Betti_1$ value near zero, as seen in Llama-3.2, indicates a tree-like or star-like graph with very few cycles, which is the topological signature of a centralized structure. In contrast, the significantly higher initial Betti₁ of 19.69 for XLM-RoBERTa indicates a highly complex graph with numerous loops, a distinctive feature of the initial layer connectivity in bidirectional frames." We will apply this principle to all key metrics in the main text.

$**Regarding Question 1: what do you mean by "optimal"? - definition of a "specialized head"**$

By "optimal," we do not mean a globally optimal solution in a mathematical sense, but rather an emergent optimum found by the model during training via gradient descent.

The objective being optimized is the standard cross-entropy loss, which, as we argue in Section 3.2 (citing Tishby et al., 2000), can be viewed through the lens of the information bottleneck principle (Equation 7). The "optimality" of reference frames arises from the interplay of two things:

• The softmax constraint: This pushes attention distributions towards sparsity on the probability simplex to efficiently compress information.
• The positional encoding bias: This creates a specific "loss landscape" where certain tokens are computationally cheaper to use as anchors.

A reference frame is "optimal" in the sense that it is the most efficient strategy the model discovers to satisfy the need for a stable, sparse coordinate system (from the softmax) within the specific geometric constraints imposed by its architecture (the position encoding). We will revise the abstract and introduction to state this more precisely.

A head is defined as "specialized" if it has learnt to consistently form attention sinks across our dataset. So, we classify a head as specialized if, for a high percentage of the test samples (for example, >80%), it produces an attention pattern that meets the $\textsf{sink}(j)$ definition from Equation 1. This ensures we are analyzing heads that have adopted a stable, structural role rather than ones that only occasionally exhibit sink-like behavior.

$**Regarding question 2: meaning of "Token Specialization".**$

This metric in Table 1 describes the behavior of the specialized heads we identified.

• "100% on BOS token" for a centralized frame means that every single one of the specialized heads in that model uses the Beginning-of-Sequence token as its primary, consistent attention sink.

• "65.4% on ','" for the distributed frame means that among that model's specialized heads, the comma token is the most common anchor, accounting for 65.4% of their observed sink behavior. The remaining percentage is distributed among other tokens that serve as secondary anchors, such as articles ("the", "a") or periods.

It's interesting to note that in the second case the "special tokens" are function-dependent, not position-dependent. A comma acts as a reference point because of its syntactic role, regardless of where it appears in the sentence.

We are gonna add a proper explanation regarding this point in the paper.

$**Regarding question 3: extra charts for attention sink distribution.**$

We agree that providing these visualizations would greatly strengthen the paper and make the concept of a distributed frame much clearer. We plan to add a new section (probably in the appendix) dedicated to this. It will include charts showing the distribution of tokens that become sinks for each reference frame type across our dataset.

$**Regarding question 4: how thresholds were set.**$

The thresholds $\tau$ (attention weight) and $\gamma$ (frequency) in Equation 1 were set empirically to be conservative and robust.

• $\tau$ (90th percentile): This was chosen as a standard statistical convention to isolate only the most extreme, outlier attention weights, ensuring we focus on truly disproportionate attention.

• $\gamma$ (0.3-0.5): This frequency threshold ensures that we only identify tokens that are consistent attractors across a large portion of the dataset, filtering out anomalies that might appear in only a few samples.

Our method identifies tokens that receive a high absolute quantum of attention, regardless of rank. A token could be the #1 attended-to token for every query, but if the attention distribution is very flat, its absolute weight might never cross the 90th percentile threshold. Conversely, a token like [BOS] might consistently receive 30% of the attention from every other token. It would always be highly ranked, but more importantly, it consumes a disproportionate, absolute share of the probability mass. The idea is that attention sinks function as reference points in a coordinate system. For this, we need to find the "gravitational wells" of the attention map, like the points that command a large, absolute portion of the attention budget from across the sequence. Our definition, by focusing on outlier absolute weights, is designed to locate these anchors.