Rope to Nope and Back Again: A New Hybrid Attention Strategy

Thank you for the valuable feedback. We address the comments as follows:

1. Training overhead analysis: We discuss relevant aspects in Section 5.3, Impacts on Training and Inference. Our findings indicate that introducing hybrid attention can significantly accelerate both training and serving. Additional benefits arise when combining it with specialized kernels such as ring attention, which reduces communication overhead.

2. Comparison with p-RoPE: Thank you for pointing this out. In the second paragraph of Section 3, we briefly mention variants that combine NoPE and RoPE along the head dimensions (e.g., GPT-J/NeoX). The work of p-RoPE further analyzes RoPE, focusing on the impact of different frequency domains and concluded that low/high frequencies focus on semantic and positonal attention respectively, which is similar to our findings that NoPE focuses more on information retrieval and RoPE more on recency bias. The difference is that p-RoPE removes low frequencies in the head dimensions (like partial rope used in prior work) and we combine RoPE/NoPE across layers (which has more engineering benefits as pointed out in section 5.3).

We are probably not able to conduct another pretraining from scratch with this variant as the resource constraint is very demanding for this experiment (similar to why we didn't do NoPE + SWA). However, we will include a reference to the p-RoPE paper and expand our discussion of its relationship to our method in the same paragraph in the final version.

We appreciate your thoughtful feedback and look forward to further discussions to enhance our work.

Thank you for the authors' response. However, as the authors acknowledge in their rebuttal, the novelty of the findings remains limited when compared to [1]. Furthermore, the set of baselines used for the experimental comparison is not sufficiently comprehensive. Strengthening these aspects would significantly improve the paper. Given that the authors have not demonstrated a convincing distinction or provided further specific experimental results, I will maintain my score.

[1] Round and Round We Go! What makes Rotary Positional Encodings useful?

We would like to respectfully clarify that the claim—“the authors acknowledge in their rebuttal, the novelty of the findings remains limited when compared to [1]”—is not supported by anything stated in our previous responses. At no point did we concede that the novelty of our work is limited relative to [1], and such a characterization is inaccurate.

To clarify again the distinction between our work and [1] (as well as other partial RoPE approaches), we emphasize that our method introduces a layer-wise combination of NoPE and RoPE, which leads to two key contributions:

1. Division of Labour: Our analysis shows that different layers naturally specialize in distinct roles—such as retrieval versus local aggregation—leading to an emergent functional separation (see Section 2) and the subsequent efficiency gains. To our knowledge, this insight has not been demonstrated in prior work. We also hope that our findings can inspire further discussion and exploration of this phenomenon within the research community.

2. Efficiency Gains: While head-wise or dim-wise mixing of RoPE and NoPE can preserve performance, they do not yield improvements in training or inference efficiency over dense models. In contrast, our proposed RNoPE-SWA design achieves up to 4× speed-up and substantial KV-cache savings (see Section 5.3).

Comparing with [1] or other "partial rope" approaches: On the analysis part, [1] offers a valuable analysis of RoPE and NoPE characteristics and the role of frequency components. Our paper did similarly but with focus on layer-wise composition which resulted in significant efficiency improvements over [1]. Moreover, our work includes full training from scratch of a substantial 8B model, further supporting the robustness and reproducibility of our approach—something not covered in [1].

We believe these points underscore the novelty and practical value of our contributions beyond what is explored in [1].

We agree that the relationship between “partial RoPE” and relevant prior work deserves further discussion. We appreciate this valuable point and will include an expanded discussion in the paper to better contextualize our approach.

[1] Round and Round We Go! What makes Rotary Positional Encodings useful?

Thank you for the valuable feedback. We address the comment as follows:

1. Analysis of RoPE, NoPE, and QK-Norm, and rationale for interleaving RoPE and NoPE for improved performance:

   In Figure 1, we analyzed the attention mass on the needle tokens across sequence lengths and model variants. From these observations, we infer that NoPE exhibits an advantage in its ability to focus on relevant information, based on the following points:

• Attention mass differences at fixed sequence length: For example, in Figure 1 (sequence length 32k), the attention mass of RoPE, NoPE, and QK-Norm is 0.02, 0.03, and 0.01, respectively. While the absolute values are small, the relative differences are notable: RoPE ≈ 2× QK-Norm, and NoPE ≈ 3× QK-Norm.

• Attention mass differences across sequence lengths: When comparing the attention mass over needle tokens across sequence lengths, we observe that QK-Norm drops the most (from 0.017 to 0.005, >3× reduction), RoPE drops from 0.033 to 0.015 (>2× reduction), while NoPE drops from 0.045 to 0.025 (<2× reduction). This suggests that NoPE degrades less when sequence length increases.

• Needle score and prior findings: From Table 2 (and corroborating prior studies), QK-Norm performs poorly on long-context tasks, likely due to its reduced ability to attend to relevant tokens—a hypothesis we further analyze in Appendix B.

These differences led us to hypothesize that RoPE and NoPE could be complementary where RoPE explicitly models positional information and recency bias and NoPE improves long-context retention and reduces degradation across long sequences.

Motivated by this, we explored interleaving RoPE and NoPE to combine their strengths. This design decision were made as an exploration. We also conducted several other attempts to improve the model's capabilities, which we did not include as they either performed poorly or were less relevant. As we also stated in the paper, other attempts were also made by prior work like GPT-J/Neox of combing NoPE and RoPE (partial RoPE) but not across layers.

Thank you for pointing this out. We will make our reasoning clearer in the final version by including numeric comparisons and percentage differences.

---

2. Comparing with baseline with larger RoPE theta values and sliding window.

Thank you for this important suggestion. We plan to include experiments utilizing varying θ values for RoPE in the final version. Due to computational constraints, these may be limited to the fine-tuning (SFT) stage. Our current baseline uses a θ of 8 million, which aligns with industry-standard long-context training regimes. For example:

• LLaMA 3 adopts θ == 500 k for support of an 8 k context window
• Qwen‑2.5, supporting up to 128 k tokens, employs θ around 10 million
• Cohere Command R, also designed for 128 k context length, uses θ on the order of 8 million

While we acknowledge that these models also incorporate techniques like Dual‑Chunk Attention or the Yarn mechanism to enhance long-context performance, the overarching trend of scaling θ to a few million for extended context remains consistent.

---

3. On including baseline models with sliding‑window attention

Thank you for this perceptive suggestion. We agree that adding baseline comparisons that also utilize sliding‑window attention (SWA) would strengthen the completeness and fairness of our evaluation. However, we did not include those in the current version for the following reasons:

• Resource constraints: Pretraining models from scratch or fine-tuning major variants with SWA is computationally expensive and falls beyond our current resource budget.

• Low practical impact of pure NoPE‑only variants: In our preliminary experiments, models using only NoPE—regardless of whether with SWA—showed significantly worse performance, including large increases in perplexity. This suggests limited practical value for pure NoPE baselines in long‑context regimes.

• Focus on core contributions: We prioritized experiments directly aligned with our hypothesis: the hybrid combination of RoPE and NoPE. As such, we ran QK‑Norm with sliding‑window attention—but omitted it from the paper because it did not illuminate the central mechanisms we study.

The goal of this paper is that we want to show:

• Analysis that different attention mechanisms and combinations across layers can result in interesting attention patterns and further understanding this pattern can help us improve the model.
• Experiments that focus on demonstrating that RNoPE‑SWA can easily outperform prevailing long-context recipes with lower compute and serving overhead without tuning the θ values. This "free-lunch" phenomenon also corresponds to findings of other recent works on hybrid attention.
• We understand that there are possibly other ways to combine different mechanisms to improve, even just between NoPE and RoPE. We think it will be good for further work to explore and refine on top of this approach.

---

We appreciate your thoughtful feedback and look forward to further discussions to enhance our work.

Thank you for your comment. We have explained our rationale for the experiments in responses to other reviewers, but to summarise: our work began with the analysis in Section 2, where we explored different positional embeddings and identified interesting attention patterns. These findings directly informed our modelling choices. We believe that the analysis in Sections 2 and 3 is valuable in its own right, because:

• Among the many works employing hybrid-attention architectures, ours is one of the very few that investigates the underlying mechanisms of their effectiveness. For example, the findings of "division of labour" across different layers can be interesting and informative for future model design.

• Our discussion of QK-Norm, to our knowledge the first in the community, provides a plausible explanation for why models like Gemma 3, which uses a similar interleaved global–local attention architecture with solely RoPE, underperform on long-context evaluations such as RULER. For example, Gemma 3 27B scores 0.91 at 8k but drops to 0.66 at 128k—a ~30% loss—while also lagging behind other open-source models. In contrast, RNoPE-SWA, despite requiring significantly less compute and being almost 4× smaller, shows only ~17% and ~7% drops on RULER QA and RULER Retrieval, respectively. While this is not an exact apples-to-apples comparison, the large gap highlights the likely role of QK-Norm, which our paper analyses in detail.

We also want to point out that our proposed method—derived from this attention analysis—offers clear efficiency advantages in speed and memory usage, while achieving better performance than other methods that combine NoPE and RoPE such as [1] or a dense baseline that utilize full attention. This "free lunch" phenomenon also stems from the analysis we performed in previous sections.

Given the scope and length of the analysis in Sections 2 and 3—and the substantial resources required—we found it difficult to allocate additional space or compute for more ablations and baselines in Sections 4 and 5. Instead, we ensured that our baseline is strong, well-trained, and uses hyperparameters (such as RoPE θ) validated by industry practice. While we would like to add further experiments, the review timeline makes it unlikely that this can be completed before the discussion period ends.

Another reason we structured our experiments in this way is that long-context evaluations can be highly sensitive to the amount of training data. With insufficient training tokens, the conclusions drawn can change dramatically. For example, the ALiBi paper [2] reported extraordinary context extrapolation capabilities, but subsequent works [3, 4] found that part of this effect was attributable to insufficient pretraining tokens and a limited receptive field. Given limited compute resources, we wanted to ensure that—even with a limited number of experiments or baselines—each run received sufficient training tokens to produce meaningful and robust results.

We kindly ask the reviewer to take these factors into account when assessing the novelty and potential impact of our work, and looking forward to future discussions.

[1] Round and Round We Go! What makes Rotary Positional Encodings useful?

[2] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

[3] BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model

[4] Dissecting Transformer Length Extrapolation via The Lens of Receptive Field Analysis

Thank you for your thoughtful feedback. Below we clarify several misunderstandings and explain the scope and focus of our work as well as answering questions.

First we want to emphasize that the goals of this paper are:

• Illustrate with analysis that different position embeddings can demonstrate different attention distributions. We then use this insight to hypothesize that mixing them may combine their strengths.
• Understanding these patterns and combining some position embeddings in a certain way (RNoPE-SWA) can easily outperform the prevailing SOTA long-context scaling method based on RoPE-NTK-Scaling and its variants. And RNoPE-SWA can achieve this with much less compute during long-context training and serving. Overall, RNoPE-SWA without much hyper-parameter tuning easily outperforms SOTA baseline with much less compute required -- a "free-lunch" phenomenon observed in other hybrid attention works (minimax-text-01, jamba, yoco, etc.) as well.
• Note that our aim is not to provide exhaustive ablations for RNoPE‑SWA, but to demonstrate the interesting phenomenon that reduced compute can yield better results under some design choices.

To give a brief summary of how we approached the problem:

• In section 2.1 and 2.2, we analyzed the evaluation score and attention distributions of variants with RoPE, NoPE and QK-Norm. Based on the analysis, we formed some intuitions and made some hypothesis on how we can combine different types of position embeddings to potentially improve the model performance.
• In section 2.2.3, we showed that a model with interleaved NoPE and RoPE layers demonstrate very interesting attention distribution and distinct attention characristics among layers with different position embedding types. The combination also showed excellent long-context performance which corresponds to our analysis and conjectures that models with this attention distribution might work better on long-context tasks than existing SOTA baselines trained with RoPE-NTK. We also shrinked the attention span of RoPE layers based on the observations we had on the attention layers to signifcantly bring down compute requirement during long context training and serving.
• Based on the above analysis, we chose to interleave NoPE and RoPE with a certain ratio and sliding window attention as our final model design.
• We compare our model architecture with the prevailing method (a dense architecture with RoPE-NTK training) where most open-sourced SOTA models adopted more or less (llama3, Qwen-2.5, DeepSeek-v3, Command R, etc.)

Note that our final method and architecture were derived not from exhaustive ablations but from empirical observations and analyses in several small‑scale pre‑experiments. The primary goal of this paper is to demonstrate that this specific approach outperforms the prevailing SOTA long‑context extension training recipe based on RoPE NTK scaling. Determining whether other combinations or hyper-parameter tuning can surpass our proposed method is beyond this work’s scope and left for future research.

---

We will answer the questions with the above paper goals in mind:

1. Why didn't the experiment show NoPE as a baseline?

Prior work (e.g., [37], [71]) already shown pure NoPE lags behind RoPE on standard benchmarks, despite strong extrapolation ability. In Sec. 2.1–2.2 we independently confirm that NoPE underperforms RoPE‑NTK on both retrieval accuracy and perplexity, which motivated focusing on mixed architectures rather than repeating a known weak baseline. We eliminated QK-Norm from subsequent experiments due to the same reason.

---

2. The paper should include all the below variants in the ablation:

• RoPE
• RoPE + SWA
• NoPE
• NoPE + SWA

Note that we derived our final architecture RNoPE-SWA based on analysis from section 2 rather than exhaustive architecture searches on architecture combinations. During the analysis, we pointed out a few interesting attention patterns and already eliminated some bad design choices like pure NoPE or QK-Norm. There could potentially exist other combinations of position embeddings that achieve on-par oir better performance and same or different attention patterns compared to RNoPE-SWA. Several potential variants can be tested such as RoPE + SWA, Alibi+SWA, partial RoPE or Yarn + NoPE + SWA and we think they are good to be explored in the future. However, the goal of the paper is to illustrate on the findings from the analysis and the experimental results that even with much less compute, this particular combination -- RNoPE-SWA -- can easily outperform the SOTA methods which uses a dense architecture and RoPE-NTK. Note that his "free-lunch" phenomenon also corresponds to findings of other recent works on hybrid attention and the analysis conducted in the paper can serve as a step towards explaining this behavior.

To sum up the reasons that prevents us from including some experiments:

• Resource constraints: Pretraining models from scratch or fine-tuning major variants with SWA is computationally expensive and falls beyond our current resource budget.

• Low practical impact of pure NoPE‑only variants: In our preliminary experiments, models using only NoPE—regardless of whether with SWA—showed significantly worse performance, including large increases in perplexity. This suggests limited practical value for pure NoPE baselines in long‑context regimes.

• Focus on core contributions: Rather than prove RNoPE‑SWA is optimal among all hybrids, we aim to show it can already surpass SOTA long‑context extension architectures/recipes.

---

3. The paper should include why evenly stacking RoPE and NoPE instead of assigning early layers with RoPE and later layers with NoPE

Similar to our reasoning above, this can be one of the explorations for fututre work. It may or may not work but comparing all available variants is not the goal of this paper.

---

4. Layer interleaving ratio ablations are not shown explicitly in the paper.

Thanks for pointing this out. We will include more information about this in the Appendix. We originally left them out because we felt they were not central to our core message, for the following reasons:

• At the end of section 2, we resutled in RNoPE-10k-SWA with 1:1 interleaving. We ablated and found 1:3 to be a good number with a decent level of sparse layers. Going with 1:7 seems to be streching too much given the model size and number of layers. This turns out to be different again when we further scaled the model parameters later on. It seems that the model can allow higher level of sparsity as it grows bigger with more layers, which makes sense intuitively as well. However, this is less relevant to the core concept the paper wants to convey.
• Any 1:X interleaving yields a theoretical compute cost of 1/(1 + X) compared to a dense model at very long sequence lengths. Thus, the “free‑lunch” trade‑off—reduced compute with better accuracy —holds regardless of the exact ratio.

While exploring how far we can push sparsity is an interesting avenue (and subject to future work), omitting these details does not undermine our main narrative: that a simple sparse‑dense interleaving outperforms dense RoPE‑NTK baselines without needing to fine‑tune hyperparameters for optimal sparsity.

---

Answer to Questions:

1. What does qc mean in the figure 2? -- Thanks for spotting this, we will rectify the typo accordingly in the figure.
2. Can you include more ablation on the design choices? -- With similar answers from above, we refined our design and obtained the final architecture based on analysis in section 2 and part of section 3 instead of massive ablations.

We appreciate your thoughtful feedback and look forward to further discussions to enhance our work.

Thank you for the valuable feedback.

1. Interleaving ratio ablation omitted: Thank you for highlighting this. We agree that this is an important point, and we will include the full ablation results in the appendix in the revised version. Briefly, our findings are as follows:

In Section 2, we presented the RNoPE-10k-SWA model using a 1:1 interleaving ratio. We later ablated different ratios and found that 1:3 offered a favorable trade-off between performance and the number of sparse layers. In contrast, a 1:7 ratio introduced excessive sparsity given the model size and number of layers, leading to a noticeable drop in the Needles score. Interestingly, this trend shifted when scaling the model further: larger models with more layers tolerated higher levels of sparsity, which aligns with our intuition that deeper networks can better distribute and specialize functionality.

We initially omitted these details because we felt they were tangential to the core contributions. Regardless of the exact ratio, any 1:X interleaving configuration yields a theoretical compute cost of 1/(1 + X) compared to a dense baseline model at very long sequence lengths. Thus, the “free lunch” phenomenon—achieving reduced compute with improved accuracy—holds across a range of interleaving ratios.

---

2. Thanks for bringing this up, we think this is also one of the very interesting findings of the paper. We actually gave this lots of thinking but ended up not including too much details about this in the original version of the paper. We will try to add as much as possible in the revised version. Our ideas are actually similar to yours (flatter attention distributions from RoPE interfering with the sharp, similarity-based attention of NoPE):

• This paper has shown that the original dense, rope-based transformer architecture has lots of noise in its attention mechanism after training. These noise prevents the model from forming useful attention patterns, especially at longer context. We think transformer models try to "fight" this design flaw during gradient descent but is not able to completely denoise its attention. This could be why when we visualize the attention distributions of the needles, there are still significant mass on the irrelevant contextual tokens (ex. with QK-Norm being the highest).

• From our understanding, the "division of labor" that happens in Figure 2 indicates that by constructing the model with interleaved NoPE/RoPE layers, gradient descent will "push" the NoPE layers to learn information retrieval more than RoPE layers since NoPE doesn't have recency bias built in at all, whereas the RoPE layers will be "pushed" to aggregate local information. We think how well this system can be trained following this setting depends on how much inductive bias we give to each type of layers. increasing RoPE’s theta value expands its receptive field, increasing its tendency to attend to distant tokens and introducing more noise into the system. Cutting the context off with sliding window is strictly imposing this inductive bias to RoPE layers to make sure they solely focus on recent contexts and introduces less obstacle for the model to follow the "division of labor" phenomenon during training.

We believe this phenomenon warrants further theoretical analysis, including the development of formal bounds or derivations to better explain the observed effects. Some related insights can be found in the minimax-text-01 report (https://arxiv.org/pdf/2501.08313), particularly Section 2.2.4, which we found to be highly thought-provoking.

We sincerely appreciate your thoughtful review and look forward to incorporating this feedback to strengthen our paper.