Round and Round We Go! What makes Rotary Positional Encodings useful?

We are happy to hear that you found our work well-written and clear. We are also happy that you appreciated one of our important claims in the paper which provides evidence against the common claim present in important works that RoPE helps to decay with distance the signal. We would like to answer your questions and comments.

The claim that high-frequency components are for positional information is unclear...However, it doesn’t necessarily mean the high-frequency components contribute to the off-diagonal pattern. So the function of high-frequency components remains unclear.

We thank you for the great comment. In our paper we mathematically explain why the highest frequencies are the most effective in constructing the positional attention patterns, but indeed we do not make claims that they are necessary. What we instead claim is that given some fixed budget of vector “norm” the highest frequencies are precisely the most useful to have the sharpest patterns. For the details see the paragraph below Theorem 5.3 which explains exactly why the highest frequencies are the most useful when constructing these sharp attention patterns. Finally, we also left a discussion of this in the Appendix E.1.

In practice, we believe that we provide ample evidence for this behaviour in our work – see e.g. Figure 14, Figure 15, and Figure 16, where we show how the heads that are diagonal and off-diagonal have very high frequency usage. The qualitative difference can be contrasted with Figure 17 and Figure 18 in which this high frequency usage is very clearly not as present. In Figure 4 this perhaps is even more clear when looking at a single layer: the two heads that mostly use the highest frequencies are heads 5 and 8 which correspond to a diagonal and off-diagonal head respectively. We also validated this behaviour in Llama in our new revision in Figure 11. We hope that this new figure in particular can help to convince you that this is indeed a wide-spread mechanism. We are happy to provide more evidence if you believe more is necessary, but we believe there is already a sufficient amount of evidence in the work (both theoretical and empirical).

Some prior works provide evidence that NoPE can still encode positional information [1][2]. So the fact that NoPE cannot learn a diagonal or off-diagonal pattern may not imply it cannot encode positions.

We agree with this point and in fact we do not make claims that NoPE is unable to encode positional information – we have clarified this in the paper. Our claim is instead that RoPE provides an efficient way to construct sharp diagonal or off-diagonal attention patterns (more generally positional attention patterns) through the use of the highest frequencies in particular. The difference between works such as [1, 2] is that we prove that specifically an attention head in isolation with NoPE is unable to learn these specific attention patterns, but we do not make any claims on what a deep model is able to learn. We however believe that studying what a single attention head can do is important because of an argument of efficiency: RoPE allows an attention head to implement something that would instead require more than one layer or component to achieve otherwise without it.

This claim comes from an analysis on a 2-dimensional case. However, it is possible that the 2D case is too restrictive and a higher dimensional case could have a different result.

We believe this 2-dimensional case to be quite applicable because as we show in our experiments, we find that the large norms often focus on distinct 2-dimensional bands. As such, we have indicative evidence that our theorem applies in the practical situations we study over several potent pre-trained LLMs. Of course, proving the more general case would be very interesting, but we believe this to be mathematically quite challenging – although we do not have reason to believe this to be untrue in the general case.

I appreciate the authors' thoughtful responses and the additional efforts put into conducting more ablation studies. After reviewing them, I believe these enhance the quality of the paper. Therefore, I have raised my score.

We thank you for your very interesting questions. We would like to address your points in full.

Figure 1 illustrates vectors with identical frequency differences, which does not fully align with the paper’s focus[...] heat maps, which may appear monotonous and lack of expressivenes

We believe that Figure 1 illustrates nicely one of the main mechanisms which we present in Proposition 3.1 and later use in Theorem 5.3. The figure depicts a single frequency as we believe that visualising multiple frequencies might be too cluttered and detract from the main point of the construction. To address this point we have amended the caption to clarify that the figure is focusing on a single frequency of RoPE as we agree with you that this might otherwise not be clear.

Regarding the heat maps, we believe that they offer a very clear way to visualise our arguments. We also would like to point out that these types of heatmaps are very common in these types of visualisations [1, 2]. As we are mostly interested in visualising activations when we take a mean over different sequences, there is often no clear 1:1 correspondence to natural language. We would be happy to accommodate your request and modify our visualisations if you could point out which specific figure(s) you are referring to and how you would improve their message.

The experiments are limited to decoder-only models...

In this work we focus on Large Language Models such as Gemma and Llama that generate text in an auto-regressive manner. These language models do not use encoder-Transformers. As such, it is out of scope of this work to study encoder Transformers. We are further unaware of open-sourced and pre-trained LLMs that use an encoder architecture and use RoPE. We hope that given our comments convince you that this is not in fact a limitation of our work. Finally, it is also quite common for works to focus solely on studying decoder Transformers e.g. [3, 4]. We have made this more clear in the text.

...While the study advances understanding of RoPE, its applicability to other positional encoding methods, such as Alibi, remains limited, potentially constraining its relevance for models with alternative encoding schemes...

We are happy to read that you believe our study advances the understanding of RoPE. This is in fact the main goal of our work. As far as we are aware RoPE is by far the most popular positional encoding used in the training of autoregressive LLMs today – used in Gemma, Llama, and many others. For this reason we choose to focus only on understanding RoPE in this paper. Alibi is definitely a very interesting technique, but as far as we are aware much less used in frontier LLMs and mathematically very different. For these reasons, Alibi falls outside the scope of this work. We do however comment in the Appendix (Section B.2) on how we believe our work applies to different positional encodings.

The paper states that higher frequencies correspond to positional attention, while lower frequencies correspond to semantic attention. However, the authors provide an ablation study only for the lower frequencies, without addressing both low and high frequencies. This aspect is insufficiently explained in the ablation experiments.

We thank you for the interesting suggestion. We have added this as an ablation in Table 2 and are happy to report that removing the lowest frequencies indeed seems to outperform by a significant margin the removal of the highest frequencies. We hope that you find this result interesting.

We sincerely thank you for your review and valuable comments. We hope that in light of our revisions (see general comment) and our response and additional ablations that you agree that our work has now improved. We would be grateful if you could consider upgrading your score under this new light. We are more than happy to answer any further questions during the rebuttal period.

[1] Randomized Positional Encodings Boost Length Generalization of Transformers. ACL 2023.

[2] Penzai + Treescope: A Toolkit for Interpreting, Visualizing, and Editing Models As Data. Johnson. Arxiv 2024

[3] Transformers need glasses! Information over-squashing in language tasks. NeurIPS 2024. Barbero et al.

[4] The expressive power of Transformers with Chain of Thought. ICLR 2024. Merrill et al.

We are very happy to see you have enjoyed our work. We are particularly happy to read that you believe “this study can help researchers better understand the mechanism of popular transformer architectures” – which is exactly the point of this paper.

The empirical study is limited to a single Gemma architecture. While this is unlikely, some results may be artifacts of the specific model selected.

We thank you for this comment. We are excited to have now added results with Llama3.1 8B which show similar patterns. Interestingly, Llama3.1 8B has a 500k wavelength parameter and Grouped Query Attention. We find that the findings are very similar to Gemma, helping to support the generality of our work. The results can be found in the Appendix (Section C).

In Section 3 authors train 2B model and show improvements on validation perplexity. While these results are positive, perplexity improvements do not always results in overall improvements in model's abilities. Authors could provide evaluation results on popular benchmarks to build more convincing picture.

We have spent some time to add additional ablations to answer other reviewers which we hope you will find valuable. We agree that our evaluation of p-RoPE could be more extensive. The focus of the paper was not that of proposing a new type of positional encoding, but rather that of understanding RoPE. We found that our analysis immediately translated to improvements such as p-RoPE, but we saw p-RoPE more as an interesting ablation to verify our intuition. We are of course happy to see that you have already pointed out that our work helps to better understand RoPE.

Pre-training model from scratch is expensive and not always feasible. I was wondering if you can instead continuously train other (Gemma-7B, Llama-3 1/3/8B, etc) models for fewer steps but with p-RoPE approach? Do you think it would work or no and why?

This is a very interesting point and something which we found to be possible – although we found that performance was better when training from scratch. We believe it is possible as cutting off the lowest frequency rotations provides the “least amount of change” when re-computing the activations.

Please, correct me if I'm wrong, but in the proof of Proposition 3.1: for k>1, g_k=theta^{-2(k-1)/d} is not necessarily rational, but algebraic. Therefore, Lemma A.1 should be stated not for rational g, but for algebraic g, and it should use the fact that pi is transcendental. line 680, cos(j-i-r) -> cos(j-i+r) line 680, the comma in the displayed equation should be a period. line 684, cos(j-i-r) -> cos(j-i+r)

Yes you are exactly right. Thanks for catching this and for carefully checking our proofs! We have corrected this.

We thank you again for endorsing our work and we hope that our additional results on Llama help to increase your confidence in the generality of our results. We are of course more than happy to answer any further questions.

Thank you so much for your thorough review and the great suggestions!

We address W3 and W4 first as W1 and W2 are also repeated in the questions.

W3 The discussion of related work is insufficiently thorough

We agree with your point that our related work discussion was a bit limited. We have added a new section in the Appendix (Section B.1) with a pointer in the main text covering a large number of works. We are of course happy to include works you believe we may have missed and hope that our additional literature review addresses your concern.

W4 ... the authors conceal a core assumption: that attention scores are interpretable or meaningful. I want to point out that this assumption is still being debated, and I suggest that the authors make it explicit ...

We thank you for the additional reference, we have added this as a note in the main text.

Q1 In practice, do the phenomena observed on gemma-7b apply to other LLM backbones, such as llama-2 or llama-3?

Thanks for the great comment, we agree that this was missing in the original version of the manuscript. We are excited to have added new results on Llama3.1 8B, showing similar results to Gemma. This new section can be found in the Appendix (Section C). Importantly the Llama model uses a different wavelength of 500k and grouped query attention – leading to very interesting insights which we discuss in the appendix and allowing us to further support our claims. In particular, we find that Llama still prefers the lower frequencies, but now leverages a greater spread of them due to the increased max wavelength. We also find diagonal attention patterns being constructed through the highest frequencies. We hope that you find these new additions valuable.

Q2 The phenomena observed in this paper are based on what scale and type of data? Could the findings be validated across various tasks, such as language modeling, code, or QA?

Our investigations rely on the publicly available pretrained Gemma models (and now also Llama) that are trained on a very large corpus. We believe the details of the training data are not public but for instance in Gemma they report “data from web documents, mathematics, and code” [1]. Llama is also trained on “5% multilingual data” [2]. As such we believe our findings to be rather broad due to the scale, breadth, and amount of training present in these models.

Q3 In Line 190, the authors claim that NoPE has strong OOD capabilities, but Kazemnejad et al., 2024 only validated this for sequences of length up to 50...

We thank you for the additional references, we have now included these as a disclaimer in the main text. We would like to clarify that we do not make claims about NoPE being the best method to generalise to OOD, but simply that NoPE provides a mechanism to construct certain attention patterns in a way that is perfectly robust to relative distance – as by construction NoPE is invariant to relative distance. An example of this is attending to the BOS token robustly, which we have now added a more detailed discussion in the Appendix (Section E.1).

Q4 In Line 497, the authors state, "p-RoPE is in spirit similar to the idea of increasing the wavelength of RoPE from 10,000 to 500,000," so I suggest that Table 2 should include RoPE with a base of 500,000.

We agree that this ablation would be valuable in our work. We have now added the results with a base of 500k in Table 2 and are happy to report that p-RoPE outperforms this baseline as well.

Q5 How do the observations in Section 4 change across different RoPE variations, such as 0.25-RoPE, 0.75-RoPE, RoPE_10000, and RoPE_500000? For example, in Figure 3, how does the norm distribution of low frequencies vary across these models?

This is a very interesting question. With the new Llama model, we now are able to see the effect of increasing the wavelength to 500k. In particular, it is precisely what we predicted given our investigation. When comparing Figure 10 (Llama 500k wavelength) and Figure 13 (Gemma 10k wavelength), we see that Gemma is much more limited to the very lowest frequencies, while the increased wavelength allows the model to use many more frequencies in Llama. We have provided a much more detailed discussion in the Appendix (Section C) on why we believe this is the case. Overall, we believe these new results to heavily back up our findings and thank you for the suggestion on this comparison.

Due to logistical reasons, it is hard for us to perform the same analysis on the models we trained ourselves, but we hope that you can appreciate the comparison we have now added between Gemma at 10k and Llama at 500k. We also believe that these results are likely to be more interesting as these models have of course been trained for much longer.

Thank you for your detailed response. I have no further questions. I raised my score (from 5 to 8).

Thanks for your excellent points! We would like to address your comments below.

The paper hopes to adapt to any context length, but the actual experimental results only have one result on 8K. And the evaluation of PPL is not comprehensive enough. ...

Thanks so much for the great points regarding how to improve our experimental evaluation. We invested time in improving our evaluation of p-RoPE and have added 2 new ablations, which have also been requested by other reviewers.

We completely agree with your point that the experimental evaluation could be more extensive, but we would like to stress that the point of this work is not that of proposing a new positional encoding, but rather to better understand RoPE. As such we saw the proposal of p-RoPE as a way great way to validate the intuition we develop in our paper. We believe that the 2 new ablations we provide help to support such a claim.

In other words, while we offer p-RoPE as a practical solution, this is not the main focus of the paper and is only there to validate experimentally our reasoning in Section 6 that the low frequency channels could be removed as they are not robust. We however also believe that our improved experimental section is still rather interesting as we train 2B parameter Gemma models, showing improvements in perplexity. Of course we agree with you that perplexity has its limitations as well and we have covered this in the Appendix (Section B.5).

We are happy to overall see that you seem to appreciate our contributions towards the understanding of RoPE and kindly request you to evaluate our work on whether it provides a better understanding and novel insights on why RoPE is useful. We believe that addressing common misconceptions such as the claim that RoPE helps to decay activations with distance to be important – as pointed out also for example by Reviewer rv9e in their strengths. Many works tend to in fact propose new positional encodings, but very few if any attempt to truly understand why the existing widespread positional encodings we have to date work so well.

I have some doubts about the display of Figure 2. Normally, for the same qk, different relative distances should have a gradually decreasing effect. But does different qk introduce different variables, making the comparison unfair?

We thank you for the great point. We agree with your perspective and we have added in the Appendix (Section B.4 – see Figure 9) another synthetic experiment where the queries and keys are “constant” from a Gaussian – i.e. we sample a single query and a single key from a Gaussian and then repeat them up to the sequence length. It is clear also in this case that there is no clear decay. We hope that this addresses your point.

Is there any empirical experiment on how many frequencies are cut off for the best effect?

In our experiments, we found a value of 25% cutoff (0.75-RoPE) to be the best performing. We report in Table 2 results for a cutoff of 0.25 and 0.75, of course we believe that a finer grid search is likely to yield even better results. Please also compare our ablation with 0.75-RoPE_{reversed} in Table 2, where we cut-off the highest frequencies. We show that this does not work as well, aligning with the intuition derived in our work.

Will removing low-frequency RoPE make the effect on short text worse?

Thanks for the interesting question. We have no reason to expect shorter texts to fare worse given our experiments. The datasets we use include a number of short documents with less than $1,000$ tokens.

We would like to thank you for your excellent comments. As most focused on our p-RoPE experiments, we hope that you could appreciate that we believe most of the contributions are actually towards the understanding of RoPE, rather than the proposal of a new type of PE. In this light, we hope that, with our new ablations and the additions we mention in our global comment, that you kindly consider upgrading your score. We are more than happy to keep engaging if you have further questions.

Thanks for pointing this out, we have updated line 423 to reference this work.

We are indeed happy to see evidence for our conjecture.

Best, Authors