Thanks for the valuable comments from the reviewer, we want to deliver further explanations accordingly.

Comparison with more baselines on more benchmarks

It is a nice suggestion to compare FoPE with more baseline methods, thus we supplement the following experiments:

• We validate the effectiveness of FoPE by evaluating it on 10+ additional benchmarks and comparing it with 3 more baselines. These supplementary experiments are conducted on OLMo-1.2B training on C4 datasets, with all experimental settings kept consistent with Sec 5.2 (line 293-329).

• While FoPE is primarily designed for length generalization and long-context tasks, it also performs comparably or even surpasses all baseline methods. But on MMLU, all methods have weak performance, perhaps caused by the training data. In the future work, we may conduct more experiments on datasets containing math data.
• We also conduct evaluation similar as Sec 5.2 (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Supplementary_Main_Experiments.pdf). FoPE still demonstrate the best accuracy on passkey retrieval and the second-best ppl on C4 (only slightly behind ALiBi).
• As time is limited, we have not conducted experiments on RWKV, xLSTM and Mamba. But it is really a nice suggestion to consider the length generalization capabilities of models with different architectures. We would like to cite these papers in related work in our future revisions.

Clarification of the hyper-parameters' influence on FoPE

We agree with the reviewer that hyper-parameters optimization is crucial for the large-scale training, but we would like to make some clarifications: 1.In a wide range of parameters we tested, FoPE consistently demonstrates a significantly better performance than RoPE. 2. The performance of FoPE is not highly sensitive to hyper-parameters (see ablation in Fig 6 and Sec 5.5), although an elaborate hyper-parameter selection of FoPE may lead to a better performance.

Thus, even without a careful selection of hyper-parameters, FoPE is still competent for replacing RoPE in Transformers.

Additionally, we emphasize that hyper-parameter tuning is an inevitable procedure before pre-training nowadays, many other hyper-parameters also need to be selected (i.e. hidden_dim, num_heads, ...). As for FoPE, the ablation studies in Fig 6 and Sec 5.5 suggested that:

• The optimal $\sigma$ for FoPE increases as hidden_dim and num_layers grow, which is expected, as larger models suffer more from spectral damage.
• The best $D$ tends to be slightly bigger than the head_dim of each attention head. This is because too few frequency components cannot represent the spectrum damage well, , while too many exceed the model’s representational capacity.

But it is a nice suggestion to have a better selection strategy, we would like to continually research this problem.

Influence of clipping low frequencies

It is an insightful suggestion to consider the negative influence of clipping low frequencies, we want to make some clarification below:

• "Clipping low frequencies" is quite similar to the "interpolation" used by many extrapolation methods like YARN[1]. As this operation is mainstream solution for LLMs' length generalization, we suppose "Clipping low frequencies" does not have significant negative influence in most tasks.
• Also, FoPE has a consistent performance on diverse benchmarks, . Thus, the answer to this problem is still unclear. But we would like to continually consider this issue.

[1] YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024.

Thanks for the elaborate comments, we will address them according to their proposed order.

Clarification for "undertrained components" and the impact of poor threshold

• We have jointly defined the "undertrained components" and "floor frequency" in Fig 2, Sec 3.3 and Sec 4, using both visualization and formula.
• Also, these two concepts are well-known in the area of length generalization, mentioned by many papers [1, 2].
• As for the impact of poor threshold, we add ablations on 60M OLMo trained with 512 context length (similar setting as Sec 5.2). Our findings suggest that selecting a floor frequency no-less than $2\pi/L$ is necessary, but a higher threshold does not have a significant influence. |threshold (f=$2\pi/L$)|0|0.5f|0.75f|f|1.25f|1.5f| |-|-|-|-|-|-|-| |ce loss on 8192 length|6.86|6.54|5.97|5.84|5.86|5.85|

Empirical analysis of the Spectrum Damage on well-trained models

• In Sec 5.4, our experiments are based on SmolLM-1.7B, a famous model well-trained by HuggingFace for academic purposes.
• In Sec 5.6, Fig 3 and Fig 7, we have a detailed empirical analysis of the influence of Spectrum Damage on LLaMA-2-7B.
• To further illustrate this phenomenon, we conduct another empirical analysis on LLaMA-2-7B (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Empirical_Analysis_on_Spectrum_Damage.pdf). We force every token only has one frequency component in the 1st layer. Then, we compare the spectrum of the 1st and 2nd layer after DFT on the attention map. Many other frequency components appear in the 2nd layer, which demonstrates the happening of Spectrum Damage.

Advantage of FoPE over ALiBi and the well-known negative influence of linear declined attention

• Our results in Sec 5.2 clearly show FoPE's advantages compared to ALiBi, it is unclear why the reviewer got the opposite conclusion.
• We agree that "the mutual information between tokens behaves as a power-law distribution", and we appreciate this perspective.
• However, this property is only useful for short context modeling, but not for long-context modeling. For example, the "linear declined attention" hinders the retrieval of long-distance information, thereby lacking the crucial ability for long-context modeling.
• Not only do our results in Fig 1 demonstrate this phenomenon, but many well-known work also have the similar viewpoint [2, 3, 4] that "linear declined attention" and ALiBi is fall short in long-context applications.

Clarification for Sec 5.4 (the maximum sequence length and base model performance)

• We have provided the maximum sequence length in Sec 5.4 (line 328 and 373).
• The performance of SmolLM-1.7B-base is as follows, which do not influence our conclusions: |length|gov_report|multi_news|trec|triviaqa|samsum| |-|-|-|-|-|-| |0-4k|9.10|5.06|42.0|68.53|17.97| |4-8k|9.50|7.06|51.0|69.21|16.70| |8k+|8.02|6.01|38.0|65.81|20.17|

Advantage and clarification of "smaller models' performance decay faster than larger ones"

• Firstly, this phenomenon implies FoPE is a scalable method, which should be considered as an advantage.
• Secondly, it is not odd that larger models have better generalization.

FoPE is valuable for many reasons, although current LLMs achieve 100k+ context length

• It is well-known that current Transformer-based LLMs achieve 100k+ context length using extrapolation as YARN [1]. But in Fig 4, we have shown FoPE can also be used for extrapolation and has better performance than YARN.
• With FoPE, models achieve better length generalization trained with a much shorter context length. Thus, FoPE can significantly improve the training efficiency of models (the longer length, the much slower training). This is valuable for saving time and money.

Comparison with more baselines on more benchmarks

Please check our reply to Reviewer LYp1 for detailed results.

Influence of residual connections, LayerNorm and specific activation functions

• We have modeled the influence of activation functions in Sec 3.2.
• Considering the residual connections and LayerNorm only deliver linear transform in frequency domain, we did not include their analysis in our paper. But for clarity, we will contain them in future revisions.

Generalization ability of FoPE on non-training data

• We have evaluated the OOD generalization in Sec 5.2. (trained on Gutenberg Books and evaluated the ppl on C4)
• Our evaluations on Sec 5.4 include datasets from SCROLLS (See Sec 5.4).

Clarification for Figure 6

The description is clearly presented just under Fig 6 in Sec 5.5.

[1] YaRN: Efficient Context Window Extension of Large Language Models. ICLR 2024.

[2] FIRE: Functional Interpolation for Relative Positions Improves Long Context Transformers. ICLR 2024.

[3] CLEX: Continuous Length Extrapolation for Large Language Models. ICLR 2024.

[4] Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs. NeurIPS 2024.

Thanks for the detailed comments from the reviewer, we would like to make several clarification below.

Computation cost of FoPE is similar to RoPE

We agree with the reviewer that the efficiency and computation cost are essential for Position Embedding, thus:

• FoPE keeps the rotary matrix having the similar shape as RoPE.
• FoPE keeps the similar pipeline as RoPE: ① pre-compute the rotary matrix and save the matrix in cache before training; ② take out the matrix for rotation during training.

Therefore, FoPE is as efficient as RoPE, independent of the model scale. (Reviewer FqQv also acknowledges that FoPE is lightweight and effective in "Other Strengths And Weaknesses")

Evaluation on more tasks and baselines

It is a nice suggestion to evaluate FoPE on more diverse tasks, thus we supplement the following experiments:

• We validate the effectiveness of FoPE by evaluating on 10+ more benchmarks and comparing with more 3 more baselines. These supplementary experiments are conducted on OLMo-1.2B training on C4 datasets, keeping all of the settings similar to the experiments in Sec 5.2 (line 293-329). We do not show the results on GSM8K as all methods achieve nearly 0 acc, which may be caused by the lack of math data in C4.

• Although FoPE is primarily designed for length generalization and long-context tasks, it also performs comparably or even surpasses all baseline methods. But on MMLU, all methods have weak performance, perhaps caused by the training data. In the future, we may conduct more experiments on datasets containing reasoning and math data.
• We also conduct evaluation similar to those in Sec 5.2 (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Supplementary_Main_Experiments.pdf). FoPE still demonstrates the best accuracy on passkey retrieval and the second-best ppl on C4 (only slightly behind ALiBi).

Thanks for the insightful comments from the reviewer, we are going to make clarifications for the concerns above.

Clarification for the "heuristic approach" and "practical approach" of RoPE/FoPE

It seems this is the major concern of the reviewer, we acknowledge the importance of clarifying this point:

1. The Rotary Matrix implies the real part of the complex number in Matrix Space, which is also the main difference between the "heuristic approach" and "practical approach" of RoPE. The derivation is in Sec 3.4 of Roformer's original paper [1], we also provide a derivation under our understanding: In the 2D case of RoPE, the vectors $q$ and $k$ can be viewed either as 2D plane vectors ($\mathbf{q}=(q_x,q_y)^T$) or as vectors in the complex plane ($\mathbf{q}=\|\mathbf{q}\|e^{i\theta_q}$, $\|q\|=\sqrt{q_x^2+q_y^2}$, $\theta_q = \arctan{\frac{q_x}{q_y}}+k\pi$ ). The inner product of the two vectors is equal to $\langle\mathbf{q},\mathbf{k}\rangle=Re[\mathbf{qk}^*]$ (the proof is at the end of this reply).

2. As taking the real part does not change the frequency-domain property of a vector, thus: ① RoPE's actual implementation still achieves Eq.2; ② we can still write a strict Fourier form equation.

3. As a result, our theoretical analysis and implementation are well-aligned. To provide better clarity, we will explicitly include a derivation in the future revisions.

4. Our further empirical results also demonstrate that the actual implementation of RoPE properly achieves its theoretical motivation, also well-aligning with our frequency-domain analysis (See https://anonymous.4open.science/r/FoPE-Supplementry-Material/Empirical_Analysis_on_Spectrum_Damage.pdf). In this experiment, we force every token only has one frequency component in the first layer. Then, we compare the spectrum of the first and second layer with DFT on attention map (aligned with the derivation in Sec 2.2).

Relationship between fourier/signal modeling and multiple stacked blocks

• Firstly, we want to clarify that, as the Fourier Transform is a linear transform [2], it does not change the main properties of models. Thus, the multiple stacked blocks are still in charge of modeling more diverse/complex and higher-order functions/signals, which improves the expression ability of models.
• Secondly, based on our modeling from a signal/frequency-domain perspective, different layers actually process signals of different frequency distribution. This is useful for improving the expression ability of models. However, RoPE wrongly regards the frequency distribution as same in different layers, leading to drawbacks in length generalization.
• Lastly, our modeling in frequency-domain can analyze the periodicity of attention mechanism, providing insights into the drawback of RoPE in length generalization.

Typical range of values for $\omega_m$

We have shown these statistics in Appendix D.3 (line 727-728). The frequencies of RoPE are between (0, 1] and samples more densely near 0.

Comparison with more baselines on more benchmarks

We also conduct more supplementary experiments, please check our reply to Reviewer LYp1 for detailed results.

Reference

[1] Jianlin Su, et al. RoFormer: Enhanced Transformer with Rotary Position Embedding.

[2] Oppenheim, et al. Signals and Systems.

Additional Proof

$\langle f_q(\mathbf{x_m},m),f_k(\mathbf{x_n},n)\rangle$ $=\langle \mathbf{q}e^{im\theta},\mathbf{k}e^{in\theta}\rangle$ $=\text{Re}[\mathbf{qk}^*e^{i(m-n)\theta}]$ $=\text{Re}[\|q\|\|k\|e^{i[\theta_q-\theta_k+(m-n)\theta]}]$ $=\|q\|\|k\|\cos(\theta_q-\theta_k+(m-n)\theta)$ $=\|q\|\|k\|\left[ \cos(\theta_q-\theta_k)\cos(m-n)\theta - \sin(\theta_q-\theta_k)\sin(m-n)\theta\right]$ $=\|q\|\|k\|\left[(\cos\theta_q\cos\theta_k+\sin\theta_q\sin\theta_k)\cos(m-n)\theta-(\sin\theta_q\cos\theta_k-\cos\theta_q\sin\theta_k)\sin(m-n)\theta \right]$ $=(q_xk_x+q_yk_y)\cos (m-n)\theta - (q_yk_x-q_xk_y)\sin (m-n)\theta$ $=[q_x, q_y][\cos (m-n)\theta, -\sin (m-n)\theta; \sin (m-n)\theta, \cos (m-n)\theta][k_x, k_y]^T$ $=[q_x, q_y][\cos m\theta, \sin m\theta; -\sin m\theta, \cos m\theta][\cos n\theta, -\sin n\theta; \sin n\theta, \cos n\theta][k_x, k_y]^T$ $=(\mathbf{R_{\theta,m}q})^T(\mathbf{R_{\theta,n}k})$ $=\mathbf{q^T R_{\theta,m-n}k}$