Base of RoPE Bounds Context Length

Q1:The Desiderata 2. The similar token gets more attention in Section 4 seems intuitively correct but may not be empirically correct. A thorough empirical verification is a must-have.

A1: In our paper, the similarity between tokens measured by the cosine similarity(A) of their corresponding hidden-states, and the amount of attention is determined by the attention score(B). "Similar token gets more attention" means that a larger (A) leads to a larger (B).

To validate this desiderata, we conducted experiments on Llama1-7B, Llama2-7B, and Llama3-8B. We extracted 200 segments from PG19, each consisting of 1024 tokens, and calculated the Spearman’s rank correlation coefficient between (A) and (B). A positive Spearman’s rank correlation coefficient indicates that as (A) increases, so does (B), and the higher the absolute value of the coefficient, the stronger the positive correlation.

The results are shown in Figure 1 in PDF of Global Response. The positive Spearman’s rank correlation coefficient validates the property of "similar token gets more attention". And we observe that this positive correlation is stronger in Llama3-8B compared to Llama2-7B and Llama1-7B, suggesting that more powerful models may better learn this positive correlation.

Q2:The motivation section is not well-written or organized. It confuses me while I read this part. These are just some previous empirical observations and are not deeply discussed.

A2: Thank you for your suggestions. We will reorganize the motivation in the revised version as follows:

Recent advancements in long-context language models have seen widespread adoption of NTK-based methods [6, 12, 13]. However, a curious trend has emerged: practitioners often employ significantly larger base values than those originally suggested by NTK-aware approaches. This discrepancy raises critical questions about the efficacy of current theoretical frameworks. Why do practitioners deviate from the recommendations of NTK-based methods? Is the out-of-distribution (OOD) theory underlying these methods insufficient to fully unlock long-context capabilities?

On the other hand, recent research [14], driven by OOD theory, proposes using a much smaller base for RoPE to extend context length. However, our findings, as illustrated in Figure 3, suggest that this approach may only provide superficial long-context capability[21]. While achieving low perplexity even at 128k context length (explicable by OOD theory), the model fails to retrieve relevant information for context lengths as short as 1k—well below its pre-trained length. The observation suggests that the small base determined by OOD theory can't unlock true long-context capability.

These phenomena motivate us to delve deeper into the relationship between RoPE's base and context length. To address the gap between OOD theory and our observations, we conduct a theoretical exploration in the next section, aiming to uncover the underlying mechanisms of effective long-context modeling.

Q3: For the Caption of Figure 6, it would be better to show how to derive the value of the dotted line.

A3: Thank you for pointing out this. We apologize for not providing specific procedures and pseudocode for calculating the values in the paper. We will provide detailed information in the revised version.

In our paper, we adopted an imprecision search. Specifically, we traversed the base values of $a.b\times 10^x$, where a, b, and x are all Arabic numerals, e.g. $2.3\times 10^6$. We then observed whether $B_{\theta, m}$ can always be non negative within the window length and select the minimum base that meets the conditions. The python code is in the in PDF of Global Response.

Q4:Will the author plan to release the models? This would benefit future work.

A4: We are happy to open source these models and hope that they can benefit future work. However, due to the requirements of rebuttal, we are unable to provide a download link here. After the decision is made on the paper, we will release all the models used in the paper.

Q5: I wonder about the negativeness and positiveness of the in equations (8) and (9). If the values are negative, say -2 and -1, which one indicates more attention?

A5: In equations (8) and (9), -1 indicates more attention.

Q6: For section 5.3, I would like to regard this as a conjecture rather than an explanation.

A6: It seems that we didn't provide much explanation in section 5.3. In section 5.3, we train a 2B model with small base from scratch and find that it has a pool long context capability, which is consistent with our theoretical perspective.

Perhaps you are referring to section 5.4, so we answer according to the content of section 5.4 below.

Thank you for pointing it out. We agree that this is not a strict explanation. There are many different perspectives on positional encoding, and we are providing an interpretation for the strange phenomenon "superficial long context capability for small base" from the theory perspective proposed in our paper. This isn't a strict proof, and further research is needed on the performance of models and positional encoding.

Q7:Moinries: The upper case and lower case of the title are not consistent, such as the title of Section 4.1 and Section 2.2; Line 225 "method2" -> "Method 2"

A7: Thank you for carefully reviewing our paper. We will address the formatting and grammatical errors in revisions.

Q1:Regarding the "Desiderata 2 The similar token gets more attention", recently StreamLLM shows that there exists "attention sink" in popular LLMs. Namely most of tokens attend to the first few tokens. This somehow contradicts with the principle that "the similar token gets more attention". Could you provide your thoughts on this statement?

A1: Firstly, we think that they are not in conflict in phenomenon. We think that the sink token can be considered as a form of regularization to focus on and serves as an anchor point for positional information. “most of tokens attend to the first few tokens” and "the similar token gets more attention" are two distinct properties refer to different aspects. The former means "current token tends to pay more attention to the first few tokens than the tokens in other positions", the latter means "At the same relative position, current token tends to pay more attention to the similar token than random tokens". They can actually both be satisfied at the same time. For example, when similar token exists, the current token give a weight of 0.8 to first token and 0.2 to similar token. When similar token does not exist, the current token give a weight of 1.0 to first token and 0 to other tokens.

More importantly, as StreamLLM say "While StreamingLLM improves the efficiency of LLMs in streaming contexts, it does not extend the models’ context window or enhance their long-term memory capabilities. " "most of tokens attend to the first few tokens" may lead to low perplexity, but can't extend the context window. While based on "the similar token gets more attention than random ones", we can adjust RoPE's base to achieve good performance on more challenge benchmarks rather than only low perplexity.

Q2: Anthropic's blogs reveal that different heads may have different functionality in in-context learning. How may this interplay with the Rope base? Do you think different heads may have different optimal rope base?

A1: According to our research, for example, in the induction heads mechanisms proposed in Anthropic's blogs, the head that retrieves long-distance information may require a larger base, while a copy head that simply copies the information of the previous token and only focuses on nearby tokens may only require a small base. So we believe that different heads require different optimal base. However, setting different bases for different heads is a highly challenging task, perhaps a search method similar to LongRoPE[1] can be used.

[1] LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

Q:Extensively test the model using benchmarks such as RULER. Provide more empirical observations on the relationship between the base of RoPE and model performance on those challenging benchmarks.

A: We greatly appreciate your suggestion. We evaluated Llama2-7b on RULER, and the evaluation results are shown in Table 1. We can observe that when the base value exceeds the lower bound 6e5 given in our paper for 32k context, the model can achieve better performance on the RULER. And when base is greater than 6e5, the improvement is slight. The detailed evaluation results on various sub tasks are presented in Table 2. We will include these results in the revisions of our paper.

Table1. Comparison on Ruler, when fine-tuning Llama2-7b to a length of 32k ( the low bound base is 6e5) under different settings of RoPE's base

Table2. The detail results on sub-tasks of RULER. Niah_single is short for ns. Niah_multikey is short for nm.