Thank you for your thoughtful comments and positive assessment of our work. Below, we provide individual responses to the comments and questions you raised.

I wonder how MLPs would impact the current analysis.

Thank you for the question. Our analysis focuses on the self-attention mechanism, as it is the primary component responsible for information exchange across tokens. In contrast, MLP layers are typically applied independently to each token and do not directly contribute to inter-token communication. As such, they are unlikely to affect position bias in the same way that attention mechanisms do.

That said, we acknowledge that MLPs, being universal function approximators, can theoretically induce position-dependent effects under specific conditions. For instance, one could construct a scenario where an MLP maps $x_i$ to $x_{n-i+1}$ for a sequence $x_1, x_2, x_3, x_4, …, x_{n-1}, x_n$​, thereby indirectly altering token interactions. However, such behaviors would require highly specific weight configurations that are unlikely to arise under standard training, and they would generally lack robustness to input variation.

Our goal in this work is to isolate and understand the role of attention masks and positional encodings in shaping position bias—factors that more directly govern how positional information is integrated across tokens. Nonetheless, extending our framework to investigate whether and how MLP layers might interact with or amplify these biases is a valuable direction for future work, and we will add a discussion of this point in the revised manuscript.

What do the authors think of other families of "sparse" softmax, like sparsemax or fenchel-young derived softmax. Would it make sense to study them through the lens of avoiding attention sinks?

Thank you for the question. Our current findings suggest that sparser attention mechanisms may help mitigate attention sinks by slowing the convergence of attention flow. In our framework, this is captured by Theorem 4.2, which shows that increased sparsity (e.g., shorter sliding windows) causes center nodes to accumulate influence more gradually, making them less likely to dominate early on. Consistently, we observe empirically that attention sinks are less likely to emerge under sparser masking conditions (Figure 10).

This connection suggests that sparse alternatives to softmax could act as implicit regularizers, dampening the formation of attention sinks and promoting more balanced information flow across tokens. Extending our analysis to formally validate and characterize these effects would be a valuable direction for future work, and we will note this in the revised manuscript.

Could the authors elaborate on larger-scale experiments that could validate their theory or that can benefit from their findings?

Thank you for the question. A key contribution of our theoretical framework is that it helps identify and quantify the sources of attention bias introduced by masking and positional encodings. These insights can inform the design of alternative transformer architectures, by motivating positional encoding schemes or masking strategies that encourage more uniform or task-aligned attention distributions.

Larger-scale experiments could build on our findings to evaluate how different architectural choices impact position bias and downstream performance on different tasks. Conversely, one could also explore whether alignment between a model’s learned position bias and natural language structure (e.g., recency effects in Futrell et al.) correlates with improved language modeling performance.

Given that the graph-theoretic framework of attention was introduced in Wu et al. (2024), I think that the mention of "novel graph-theoretic framework" in the abstract is not adapted.

Thank you for the thoughtful comment. We agree that the use of graph-theoretic tools to analyze attention mechanisms has been explored in prior work, notably by Abnar et al. and Wu et al. We will revise the abstract to more accurately reflect our contribution. We will revise the wording in the abstract to more precisely reflect our contribution.

The novelty of our work lies not in the use of graphs itself, but in how we apply this perspective to analyze the flow of information and the emergence of positional bias across multiple layers of attention. Our framework leverages the structure induced by causal and other attention masks to provide theoretical insight into how positional preferences arise independent of semantic content. We will clarify this distinction in the revised abstract to better situate our contributions.

We sincerely appreciate your feedback and welcome any further suggestions.

---

References

Abnar et al. (2020) Quantifying Attention Flow in Transformers.

Futrell et al. (2015) Large-scale evidence of dependency length minimization in 37 languages.

We appreciate your positive assessment and constructive comments, which have helped strengthen our work. Below, we provide responses to your comments.

The paper primarily analyzes the attention mechanism without deeply exploring how other transformer components might interact with position bias.

Thank you for your thoughtful comment. We agree that these other transformer components could contribute to position bias in an nontrivial way. We view this as an important direction for future work.

That said, our focus on the attention mechanism in this work is aligned with a common practice in the theoretical analysis of transformers: isolating the attention mechanism to better understand its intrinsic inductive biases. Many prior studies have adopted similar simplifications by omitting or abstracting away certain components in order to enable precise analysis [1-5].

Nevertheless, our analysis can be naturally extended to account for residual connections. Following the renormalization technique in [6], we can redefine the attention matrix at layer t as $A^{(t)}\_{res} = 0.5 A^{(t)} + 0.5I$. This ensures $A^{(t)}_{res}$ remains a valid stochastic matrix and thus retains interpretability. Under this adjustment, our theoretical results still hold; but the convergence rate would slow down, aligning with findings in [1] that residual connections slow down rank collapse rate under attention. We will add a remark to the paper detailing how to handle residual connections in our analysis and their effect.

...would benefit from more direct empirical validation with actual LLM attention patterns rather than just the controlled experiments.

Thank you for the comment. Natural language lacks ground-truth annotations for position bias, making it difficult to disentangle architectural effects from semantic content. To enable precise control, we adopt the synthetic setup from Reddy, which allows us to study positional bias in isolation. Despite this abstraction, our setup reproduces key behaviors observed in real LLMs, such as the “lost-in-the-middle” effect (Sec. 5.2) and attention sinks (App. K.2). Tab 1 further shows alignment between our results and empirical observations on position bias reported in the literature.

We agree that validating these results directly in real models with natural language data is an important next step. As noted in App K, one direction is to quantify position bias in LLMs and relate it to known linguistic structures [7]. We will highlight these directions in the updated version.

The experimental design has some minor limitations.

Thank you for your comment. The choices n = 8 and d=64 follow from Reddy. In line with your suggestion, we have added more variants for the decay masks and RoPE for the experiment in Sec 5.1 under the same setup:

Decay

RoPE

The results align with our findings in Sec 4, with greater $m$ or $\theta$ inducing greater decay, while deeper attention amplifying the bias toward earlier tokens.

We also present here the standard deviations for the results in Fig 2:

The standard deviations for Fig 3 show a similar trend that they are small compared to the averages. This supports the robustness of our reported trends.

We sincerely appreciate your feedback and welcome any further suggestions.

---

References

1. Attention is not all you need: pure attention loses rank doubly exponentially with depth

2. Signal propagation in transformers: Theoretical perspectives and the role of rank collapse

3. A mathematical perspective on transformers

4. The emergence of clusters in self-attention dynamics

5. On the Role of Attention Masks and LayerNorm in Transformers

6. Quantifying Attention Flow in Transformers

7. Large-scale evidence of dependency length minimization in 37 languages

We appreciate your positive assessment and constructive feedback, which have helped strengthen our work. Below, we provide individual responses to the comments you raised.

The paper misses some critical literature in interpretability and analysis of language models.

Thank you for the pointers to these important references. In particular, we will clarify how our approach aligns with, yet differs from, the “attention rollout” and “attention flow” methods proposed by Abnar et al. (2020). While those methods conduct post-hoc graph-based analyses on specific inputs to understand information flow, our work takes a more general and theoretical perspective: we formalize attention masks themselves as directed graphs (Sec. 3) and analyze how their structure shapes the flow of information across layers—independent of semantic content.

This allows us to rigorously trace how positional information is integrated across general sequences, highlighting the inductive bias introduced by masking alone. We hope this theoretical framework—which connects architectural design to emergent behavior—offers a useful new tool for understanding and probing attention mechanisms in future research.

We appreciate your feedback and will work diligently to ensure that our final version properly suits our contribution within the broader literature on understanding attention.

The theoretical analysis ignores residual connections, which are integral to modern transformers.

Thank you for your comment. One way to incorporate residual connections into our analysis is to follow the renormalization approach used in Abnar et al. (2020), where each attention matrix $A^{(t)}$ is adjusted to $A^{(t)}\_{res} = 0.5 A^{(t)} + 0.5I$. This ensures $A^{(t)}_{res}$ remains a valid stochastic matrix and thus retains interpretability. Under this adjustment, our theoretical results still hold; but the convergence rate would slow down, aligning with results in Dong et al. (2021) showing that residual connections slow down the rank collapse rate in token representations. We will add a remark to the final manuscript detailing how to handle residual connections in our analysis and their effect.

Should matrices $W_q$ and $W_k$ have size $d_{QK}$ instead of $d'$?

Good catch! Indeed, in many practical implementations, $d'$ matches $d_{QK}$​, However, we keep them distinct in our formulation to accommodate more general scenarios where these dimensions may differ (e.g. Wang et al. (2024) proposes adjusting $d_{QK}$​​ at inference time to handle longer contexts more effectively).

Attention sink and center node: do your analysis does not actually justify the existence of attention sinks, right? Or does it?

Thank you for the thoughtful question. Our analysis does not directly justify the existence of attention sinks—defined as single-layer heads that concentrate all attention on a single token. Rather, our goal is to provide structural insight into where attention sinks tend to emerge and why they become more prominent in deeper layers.

• In particular, we observe that the positions where attention sinks are most likely to appear (e.g., the first token under causal and sliding window masks, or the first K tokens under prefix masks; see Xiao et al., (2024); Gu et al., (2025)) coincide with the center nodes in the attention-mask graph defined by our framework. These nodes act as structural attractors in the multi-layer attention dynamics, helping to explain why they accumulate disproportionate influence over time.

• Thus, while attention sinks remain a per-layer phenomenon, our graph-theoretic perspective offers a complementary explanation for their emergence in specific positions and their amplification in deeper layers.

We appreciate your questions and comments very much. Please let us know if you have any further questions.

---

References

Wang et al. (2024) Length Generalization of Causal Transformers without Position Encoding.

Xiao et al. (2024) Efficient Streaming Language Models with Attention Sinks.

Gu et al. (2025) When Attention Sink Emerges in Language Models: An Empirical View.

Dong et al. (2021) Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth.

Abnar et al. (2020). Quantifying Attention Flow in Transformers.

We thank the reviewer for providing thoughtful feedback. Below, we provide detailed responses to the comments.

Theorem 4.1 only yields an upper bound on the weight given to a token.

Thank you for the comment. We agree that Thm 4.1 provides an upper bound rather than a strict inequality, and we do not claim it guarantees a strict bias toward earlier tokens in every setting. Rather, it characterizes an emergent tendency, rooted in the causal mask, for earlier tokens to accumulate more attention as the number of attention layers increases.

• Importantly, while the upper bound is not always tight, it is not vacuous: it decays exponentially to $0$ for tokens $i\geq 2$ as attention deepens, meaning it cannot be arbitrarily large. This suggests that—regardless of specific token embeddings—later tokens become progressively less influential, while earlier tokens retain more aggregate attention. One can derive more precise and stronger results by imposing additional assumptions regarding token embeddings or model weights. For example, for sequences of identical tokens, the upper bound becomes exact.

• Moreover, our empirical results support these core insights from Thm 4.1. As shown in Fig. 2, even in the absence of explicit positional bias in the data, models with causal masking exhibit clear early-token bias—an effect that strengthens with depth. This empirical observation closely aligns with our theoretical findings, suggesting that the upper bound meaningfully captures the inductive bias induced by causal masking.

Attention patterns look at how attention is distributed in a single layer.

Thank you for the comment.

• While attention patterns are often analyzed at the single-layer level, such analyses may miss how information propagates and accumulates across layers—a key aspect of how transformers build contextual representations. A central contribution of our work is to model this global effect of attention across layers. In your example, while token 2 does not directly attend to token 0, it inherits information from token 0 through token 1. Such multi-hop influences are not captured by per-layer attention patterns but are critical for understanding deep model behavior.

• Our graph-theoretic framework is designed precisely to capture these multi-step dependencies by modeling the global effect of attention across layers. This complements prior layerwise analyses and aligns with recent work (Abnar et al., 2020; Barbero et al., 2024; Wu et al., 2024) that emphasizes the importance of analyzing multi-layer attention compositionally.

By tracing these cumulative token interactions, our framework offers a complementary and more holistic view of attention dynamics. We hope that our methodology can serve as a useful tool for the community and help guide future investigations into multi-layer attention mechanism.

In Kazemnejad et. al. the authors explicitly show a set of weights for a transformer model that lead to it learning positional indices whereas the paper is claiming this can not be achieved based on experiments.

Thank you for the comment. We would like to clarify the relationship between our work and the findings of Kazemnejad et al.:

• Kazemnejad et al. provide a theoretical construction showing that positional information can, in principle, be encoded using only causal masking via hand-crafted weights. However, their result does not reflect what typically emerges through standard training.

• Our work builds on this insight by asking a complementary question: Do transformer models actually learn positional encodings in practice when trained without explicit positional signals? We address this through a controlled task where position is critical and observe whether models develop positional awareness under causal masking alone.

• As shown in Sec. 5.2 and Fig. 3, models trained with explicit positional encodings (sin PE or RoPE) capture both start and end-of-sequence biases, while models using only causal masking consistently fail to capture end-of-sequence patterns. This suggests that while position can theoretically be represented via causal masking, it does not seem to emerge naturally in practice, even when position is highly informative. Our theory supports this by showing that causal masking induces an inductive bias toward earlier positions.

Thus, rather than contradicting Kazemnejad et al., our work provides a more practical and deeper understanding of how position bias actually manifests in Transformers under different PEs. We will clarify this distinction more explicitly in the final version of the manuscript.

We appreciate your questions and comments very much. Please let us know for any further questions.

---

References

Abnar et al. Quantifying Attention Flow in Transformers

Wu et al. On the Role of Attention Masks and LayerNorm in Transformers

Barbero et al. Transformers need glasses! Information overSquashing in Language Task