Organization

Because there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews. We will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).

Weaknesses

The authors find that built-in ...

It is a good question, though to be pedantic, Spherical RoPE is not unconstrained; it is contrained to not be commutative. It is true that the models learn to be almost completely translation equivariant (see the Questions section for elaboration).

While it is an interesting question, we don't think it is relevant enough to cram into the main body of this paper. The point still stands that the hard constraint does not appear to be why RoPE performs well.

The presented results seem to be from ...

We have included additional experiments below.

Similarly, for the extrapolation experiment in Figure 3, the statistical significance of the results is unclear and care should be taken when drawing conclusions such as "Spherical RoPE generalizes better to larger image sizes compared to other RoPE variants".

We will include results from three runs when those results are available (expect Saturday/Sunday).

The experiments are limited in scope; ...

Our claim is that RoPE does not need to be strictly equivariant, that is the benefits from RoPE are not caused by equivariance. We would like to make it clear that we are not claiming that "equivariance is not needed for vision (in general)."

We have included segmentation experiments on VOC Segmentation, however this tends to require a pretrained backbone for transformer based architectures to have good results. Even in this very low data regime, we see that Spherical RoPE performs on par which is contrary to the main motivation of shift invariance as an inductive bias.

The authors compare their methods ...

This has now been included.

While the authors' findings ...

RoPE is currently the most popular way of doing positional encodings and the question of how to extend it to N-D is a very relevant and active topic right now. It is very relevant for the field to understand why RoPE works to move forward. It is currently assumed that it is because it is a relative positional encoding (see blanket statement).

Questions:

Have the authors tried to interpret ...

Yes we have. The Learned Spherical RoPE learns to push all the $y$ frequencies to zero in the last layers which collapes the rotation to the identity. This is a special case where Spherical RoPE becomes strictly equivariant to $x$. We will try to provide a small section in the appendix because it is interesting.

However, this requires more testing such as whether this is determined by the order of the roll/yaw matrix. Moreover, whether it learns to be approximately equivariant or not isn't relevant for this paper's focus question of whether it is necessary. Indeed, it almost needs to learn approximate invariance for these datasets since we do random cropping augmentations. Because of the history and perceived importance of equivariance in these datasets, our results are even more evidence that equivariance is not the core reason RoPE works.

The authors should report error bars on the results...

We have provided these results below, though some results are in progress.

The authors should re-run experiments for LieRE using their training framework, so that the different results in Table 2 can be compared more directly.

CIFAR100 provided. ImageNet in progress.

In Figure 3, it seems that the highest ImageNet ...

This is a standard result and one of the draws of RoPE-like embeddings as it allows for training on lower resolution images which can be done much faster [1-5]. We are unfamiliar if there is any proper explanation of this phenomena, but the layman intuition is that the larger resolution provides more information and more patches as context for the classification. The change in zoom in the content of the patches is not an issue because of random cropping while training.

The paper uses the term "equivariant" ...

Thanks for pointing this out. We discuss this concept in detail in our discussion of equivariance in the Appendix, but we have now added a sentence stating that we assume equivariance -> shift equivariant explicitly in the main body.

We have thought about shift equivariance. To make a rotationally equivariant RoPE, one simply needs to parameterized position in log-polar coordinates. However, SE(2) is much more complicated. One needs a position and orientation [6] of the patch and the orientation is not a known quantity. More generally, one could construct a bi-invariant, which has been shown to be equivalent to the steerability property in [7].

[6] Kofinas, M., Nagaraja, N., & Gavves, E. (2021). Roto-translated local coordinate frames for interacting dynamical systems. Advances in Neural Information Processing Systems, 34, 6417-6429.

[7] Knigge, D., Wessels, D., Valperga, R., Papa, S., Sonke, J. J., Bekkers, E., & Gavves, E. (2024). Space-time continuous pde forecasting using equivariant neural fields. Advances in Neural Information Processing Systems, 37, 76553-76577.

Typos and other minor formatting issues (these do not influence the rating):

Thank you for the diligence!

Blanket Statement

Significance

As we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is not that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.

Because RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform slightly better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem necessary for the main benefits of RoPE.

We believe that this is significant in context of very recent work. Many contemporary works seek to impose constraints, such as commutativity, to "extend" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:

[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string. (ICML July 2025 Spotlight Poster)

[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).

[3] Liu, H., & Zhou, H. (2025). Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding. (Preprint)

Additional Experiments

We will add results when they are available.

We improved the frequency initialization for learnable RoPEs and have run these additional evaluations:

Below we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.

Thank you for the clarification.

Yes, you are correct. It is very important for the model to be able to represent equivariance. We have observed in follow up work that making it impossible for the model to learn relative position is severely detrimental to the performance. If the query and the key are rotated in opposite directions, then the model can only represent the mean position rather than the relative position. Our rudimentary experiments observed that this formulation performs as badly as no positional encodings.

As for why we should drop the strict requirements if it needs to learn it anyways: Unlike many N-D systems in science which are perfectly equivariance, vision is not strictly equivariant but only approximately equivariant. This was shown in [1] where softened weight sharing gives better results than strict equivariance. This makes vision an interesting task for assessing positional encodings. Furthermore, we focus on vision because it is the standard benchmark in all related N-D RoPE extensions.

We revised the paper with the below changes to try to weaken the claim and add nuance. I hope this addresses your concerns on the nuance.

1. Abstract :

Line 9-10 : "However, we question the necessity of equivariance." -> "However, we question [whether strict equivariance plays a large role in RoPE's performance]."

2. Introduction:

Line 67-68 : "We conclude that there is evidence that a rotary positional encoding need not be equivariant for vision." -> "We conclude that [our evidence points to the performance of RoPE over traditional embeddings not being explained by equivariance]."

3. Table 1 :

"Equivariant" -> "Strictly Equivariant"

4. Spherical RoPE:

Line 237: In fact, their generators are strictly non-commutative, $\mathcal{A}_x\mathcal{A}_y \ne\mathcal{A}_y\mathcal{A}_x$.

->

In fact, their generators are strictly non-commutative, $\mathcal{A}_x\mathcal{A}_y \ne\mathcal{A}_y\mathcal{A}_x$. [While this does not mean Spherical RoPE is incapable of learning or approximating equivariance throughout the network, it is the component of LieRE removed by Mixed RoPE and works which enforce commutativity such as [1] and [2].]

5. Conclusion:

Line 324 : We conclude that Mixed RoPE is a very general solution for M -D data if equivariance is a necessity. However, we see little evidence that relative positional bias is impactful for vision transformers.Thus, we conclude that we do not need equivariance for rotary-based positional encodings to have a performance boost over APE methods."

->

"We conclude that Mixed RoPE is a very general solution for $M$-D data if equivariance is a necessity. However, we see little evidence that strict relative positional bias is impactful for vision transformers. Thus, we conclude that [evidence suggests that RoPE does not need stict equivariance constraints to] boost performance over APE methods."

5. Limitations : Moved to main body.

"While our results do not show relative embeddings to be detrimental, we believe them to be evidence that equivariance is not the reason for RoPE's success. However, our experiments were performed in Vision where the number of tokens is limited compared to the long context lengths of NLP. Moreover, the datasets are not what many believe to be "at scale". While Spherical RoPE and LieRE would intuitively favored at scale over Axial RoPE, as they have less inductive bias, it is unclear whether inductive bias and equivariance is favored at scale \cite{brehmer2024does}.

[It has also been shown that vision is not a purely equivariant task and benefits from symmetry breaking [1]. Our results do not show that equivariance is not useful in tasks that are grounded in physics and obey strict symmetries.]"

Organization

Because there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews. We will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).

Reviewer 3 :

Weaknesses

I think the main weakness of this paper is its limited evaluation and experiments. While the theoretical contributions are solid, only the smallest ViT-S model and only two datasets CIFAR100 and ImageNet was used. As the main claim of this paper hinges on the fact that Spherical RoPE performs similarly to Mixed RoPE, the conclusion ("equivariance is not necessary") requires more thorough evaluation at multiple model sizes and with various other datasets.

We have provide more evaluations on various training splits, segmentation tasks, and have included error bars from multiple runs. These can be found below.

A smaller weakness of this paper is that although the authors show that equivariance is not necessary for performance, there is no explanation or hypothesis as to why this may be true. This would greatly strengthen the paper in my opinion.

We would like to refrain from being too speculative. In the Discussion, we do hypothesize this may be because the small number of tokens, so it may not hold for language. However, there has been evidence in the past that equivariance as a hard constraint is too strict for vision, specifically relaxed weight sharing has beneficial for vision in [6].

[6] Elsayed, G., Ramachandran, P., Shlens, J., & Kornblith, S. (2020, November). Revisiting spatial invariance with low-rank local connectivity. In International Conference on Machine Learning (pp. 2868-2879). PMLR.

Blanket Statement

Significance

As we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is not that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.

Because RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform slightly better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem necessary for the main benefits of RoPE.

We believe that this is significant in context of very recent work. Many contemporary works seek to impose constraints, such as commutativity, to "extend" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:

[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string. (ICML July 2025 Spotlight Poster)

[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).

[3] Liu, H., & Zhou, H. (2025). Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding. (Preprint)

Additional Experiments

We will add results when they are available.

We improved the frequency initialization for learnable RoPEs and have run these additional evaluations:

Below we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.

Organization

Because there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews, we will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).

Weaknesses:

Lie RoPE isn't equivariant if the generators don't commute, making the spherical RoPE redundant in terms of figuring out whether equivariance is important (we can just compare Lie RoPE to itself with/without commutative generators)

By Proposition 2, all LieRE with commutative generators are theoretically equivalent to some form of Mixed RoPE, as we show it is the spectral decomposition of the generators. Thus, to compare "unconstrained" LieRE to LieRE with commutative generators, we can theoretically compare the numbers to Mixed RoPE which is done in their [4] or add positional embeddings as done in [5].

However, these do not isolate equivariance. Spherical RoPE has three main advantages as an ablation:

• LieRE is capable of learning commutative generators where (fixed) Spherical RoPE is strictly unable to (in one attention layer). Similarly, modifying Mixed RoPE with APE as done in the Mixed RoPE paper (Heo et al.(2024)) makes the model not equivariant, however it is not a precise ablation. In a similar way to how Mixed RoPE constrains to LieRE to be commutative generators, Spherical is constrained to be non-commutative generators -- outside of the degenerate case where the frequencies are learned to be zero. This is why we choose it to evaluate whether equivariance is why RoPE performs well. In summary, the important property is Spherical RoPE is strictly the component of LieRE which breaks the "relative positional encoding".
• LieRE does not have as comparable of an implementation. LieRE requires exponentiating skew symmetric matrices which: 1. Requires more parameters 2. Introduces numerical caveats. Spherical RoPE on the other hand can be implemented in an almost identical way to Axial and Mixed RoPE making it more appropriate as an ablation.
• The learned behavior of Spherical/Mixed/Axial RoPE is more concise. We will add a section visualizing the weights learned to the appendix in each layer by Spherical, Axial and Mixed RoPE. We observe that that they learn to set many frequencies to zero. This also means weight decay on the frequencies will bias the model to be position invariant. This type of intuition is more difficult when learning Lie matrices with LieRE or other commutative variants.

[4] Ostmeier, S., Axelrod, B., Moseley, M. E., Chaudhari, A., & Langlotz, C. (2024). Liere: Generalizing rotary position encodings. arXiv preprint arXiv:2406.10322.

[5] Heo, B., Park, S., Han, D., & Yun, S. (2024, September). Rotary position embedding for vision transformer. In European Conference on Computer Vision (pp. 289-305). Cham: Springer Nature Switzerland.

Experiments are shown without standard deviation; thus results aren't clear, the results themselves are quite close to each other (other than in comparison to APE).

This is one of the main points. Even completely removing equivariance by constraining the generators to be non-commutative the results remain close outside of Uniform RoPE and APE. We have improved results for learned PEs and included error bars from multiple runs (see above evaluations).

Axial RoPE (which is equivariant) performs consistently better or at the same level as Spherical with fewer degrees of freedom and parameters, so it may indicate that equivariance is important...

Spherical RoPE outperforms Axial RoPE, neither of which require any parameters. Here is the table for training on subsamples. Spherical RoPE performs consistently better.

Questions:

Can't we look at experiments without the Spherical RoPE, as it is just an example of a RoPE that isn't equivariant, while Lie RoPE is already a known one that isn't equivariant?

See response to weaknesses

The experiments don't show IMHO that equivariance is not beneficial, it seems Axial RoPE performs better with fewer parameters (or rotational degrees of freedom).

See response to weaknesses

Empirical evaluation is quite limited. Are there any more experiments performed? (Suggestion: Lie RoPE with commuting generators vs arbitrary generators. This might show the inverse result, that equivariance might be beneficial.)

In our paper, we show that Lie RoPE with commuting generators is the same (mathematically) as Mixed RoPE, so your suggestion boils down to comparing their performances. See first response for more detail.

Blanket Statement

Significance

As we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is not that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.

Because RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform slightly better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem necessary for the main benefits of RoPE.

We believe that this is significant in context of very recent work. Many contemporary works seek to impose constraints, such as commutativity, to "extend" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:

[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string. (ICML July 2025 Spotlight Poster)

[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).

[3] Liu, H., & Zhou, H. (2025). Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding. (Preprint)

Additional Experiments

We will add results when they are available.

We improved the frequency initialization for learnable RoPEs and have run these additional evaluations:

Below we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.

Organization

Because there seemed to be consistent confusion about the goal and significance about our work we have included a blanket statement at the end, after the individual responses. We have also included the additional results in that section as it was a consistent concern across reviews. We will provide the remaining numbers as soon as possible (expect Saturday for ImageNet).

Weaknesses:

The experiments (image classification) seem appropriate, but there is not clear outperformance of the proposed methods.

Our point is not to outperform, but rather to ablate whether relativity is necessary.

It is not clear to me how important these results are for classification performance. The results seem relevant and interesting, but in many cases the differences in performance seem small, especially on ImageNet. It might be nice to find evaluation domains/benchmarks that illustrate these differences more precisely. In particular, it would be interesting to know if there are particular types of (image) classification problems for which these methods are particularly relevant.

We have included a segmentation benchmark in the appendix. Once again, our point is to ablate whether relative position is important and not to show dominance of Spherical RoPE. In our view, ImageNet and image classification are biased toward relative PEs as the translations equivariance of convolutions have proved powerful inductive biases in the past. Thus, the comparable or better performance illustrates our point more than an anti-equivariance tailored datasets. These would not provide evidence that equivariance is not a powerful inductive bias for vision. In addition, image classification is the most common benchmark for comparing 2D RoPE methods in the closely related works of Mixed-RoPE[5], LieRE[4], ComRoPE [2] and STRING[1].

Related to the point above, I wonder if it might make sense to create some ablation studies here that could help pinpoint which aspects of the rotary embedding are most helpful

There seems to be a misunderstanding. Spherical RoPE and Uniform RoPE are meant to be ablations to "pinpoint which aspects of the rotary embedding are most helpful". We have modified the tables from "(Ours)->(Our Ablation)" to make that more clear and to emphasize Spherical RoPE not necessarily as a practical method, but as an ablation.

It struck me odd that Prop 1 and 2 did not have formal proofs. I would convert the "explanation" text into a proof or at least include the proof in the appendix (if I missed it in the appendix, please ignore this comment).

Thanks for calling it out. We will add more formal proofs to the appendix.

Limitations:

See my comments in strengths and weaknesses. I am most concerned that the paper does not make the right comparisons and does not zero in on the aspects of the rotary embedding that is most important.

If this is not resolved please clarify and I will try to respond.

Blanket Statement

Significance

As we feel like several reviewers may have misunderstood the goal of our work, we would like to clarify on this point: Our goal is understand why RoPE works, and specifically whether equivariance is the dominant contributer as is commonly believed. Our claim is not that one should use Spherical RoPE or Uniform RoPE. These are ablations to move towards understanding what is important for RoPE's success.

Because RoPE has been incorporated into every large model, it is a very relevant question for this year to understand why exactly it works. It is strongly believed that the improvement is because RoPE is a relative positional encoding. We believe we clearly show that this is strictly not true. Whether Liere, Mixed RoPE, or Spherical RoPE perform slightly better is not relevant as the understanding is RoPE is better because it is equivariant. As Spherical RoPE (which is not relative) maintains its gap over APE where Uniform and Axial RoPE (which are relative) has a weakened performance, equivariance does not seem necessary for the main benefits of RoPE.

We believe that this is significant in context of very recent work. Many contemporary works seek to impose constraints, such as commutativity, to "extend" LieRE. We show by Proposition 2 that if we need to create a rotary embedding which is commutative, then whatever we do will theoretically boil down to Mixed RoPE. We also show through experiments that reversibility [3] (Axial RoPE) produces worse performance. To illustrate the impact, we point the reviewers to three recent papers doing exactly this that have been accepted or submitted in the last three months:

[1] Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Avinava Dubey, Ayzaan Wahid, et al. Learning the ropes: Better 2d and 3d position encodings with string. (ICML July 2025 Spotlight Poster)

[2] Yu, H., Jiang, T., Jia, S., Yan, S., Liu, S., Qian, H., ... & Yuan, C. (2025). ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 4508-4517).

[3] Liu, H., & Zhou, H. (2025). Rethinking RoPE: A Mathematical Blueprint for N-dimensional Positional Encoding. (Preprint)

Additional Experiments

We will add results when they are available.

We improved the frequency initialization for learnable RoPEs and have run these additional evaluations:

Below we show the time it took for each method to run 100 forward passes to compare relative speed. The experiment was performed by running a dummy input of dimension (B=256, C=3, H=224, W=224) 100 times with a ViT backbone on one A100 gpu. This is simulated training time, so the rotation matrices were recalculated with every pass.