What is Wrong with Perplexity for Long-context Language Modeling?

We appreciate Reviewer KQKC for giving highly positive feedback on the solidness of our paper and the persuasiveness of our experimental results. Now we address your concerns in the following points.

---

Q1. I don’t fully agree this the claim « the evaluation of long-context capabilities still heavily relies on perplexity (PPL) as the de facto metric » as we see now most papers relying on long-context benchmarks or use the needle-in-the-haystack test to evaluate long-context LLMs.

A1. Thanks for pointing out the inaccuracies in the expression in our paper. We have modified it to “the evaluation of long-context capabilities still widely uses perplexity (ppl) as the de facto metric.”

---

Q2. The reference to ppl is not the good one (Bengio did not invent perplexity!), you should credit people who actually were the first to introduce this metric. Perplexity was originally introduced in 1977 in the context of speech recognition.

A2. Thanks for pointing to the original reference. Following your suggestion, we have updated the paper with the correct citations.

---

Q3. I’d like to learn more about the computational implications of calculating LongPPL and LongCE, especially since the LPG score needs to be computed for each token (see my question below). I'm particularly curious about the costs associated with LongCE, as described in Section 3.2. While the Implementation details of LongPPL in the Appendix were helpful, I still want to understand more about how computationally expensive this process is.

A3. The calculation of both LongPPL and LongCE involves two steps: (1) One forward pass on the full text, and (2) several forward passes on short texts (to obtain the LPG score for each token). Step 1, which involves calculations for both PPL (CE) and LongPPL (LongCE), has a time complexity of $O(N^2)$ , where N is the input sequence length (N=32k in this paper). Step 2 represents the computational overhead of LongPPL (LongCE), with a time complexity of $O((N-K)K^2/d)$, where K is the length of the short context(K=4k in this paper), and d is the sliding window length (d=1k in this paper). Under the parameter settings in this paper, the computational overhead for LongCE is approximately 80% of the cost of CE.

Specifically, if we calculate the LPG for each token according to its original definition in Eq. 5, then a total of N-K forward passes of length K are needed, resulting in a computational complexity of $O((N-K)K^2)$. To make this process more efficient, we utilize the sliding window algorithm described in Appendix A.1. When the sliding window is set to d , we can obtain the LPG values for d tokens in a single forward pass of length K + d , whereas previously, we could only obtain the LPG value for one token in a forward pass of length K. Under this algorithm, step 2 only requires approximately $\frac{N-K}{d}$ forward passes on a sequence, reducing the complexity of the algorithm to $O((N-K)K^2/d)$.

Additionally, in terms of memory usage, LongPPL incurs extra overhead from loading the evaluator model. However, since LongCE uses the same model for both evaluation and training, it only introduces intermediate storage overhead from the inference of short context.

We have included the above discussion in Appendix A.1.

---

Q4. Computing longPPL and longCE (as opposed to PPL) must be more costly; how much more costly (especially a you need to compute the LPG score for each token at training time) ?

A4. In A3, we have already elaborated on the theoretical computational complexity of LongPPL (LongCE). Next, we will further explain their specific time overhead.

For LongPPL, we have added the following table detailing the time costs.

It can be observed that the time cost of calculating LongPPL using the 8B model as the evaluator is approximately 3~4 times that of calculating PPL, while the overhead for using the 72B model is much higher.

Although the computational overhead of LongPPL is non-negligible, we believe that such a computational cost will not have a substantial impact on the practicality of LongPPL. On the one hand, if users employ LongPPL as a benchmark, key tokens can be calculated offline, resulting in no online computation overhead. On the other hand, if LongPPL is used as an evaluation metric during training, its computational overhead is negligible compared to the overall training cost (as evaluation steps are typically sparse during training). We have included the above results in Appendix B.7.

For LongCE, we conducted an additional set of ablation experiments. In the following table, we present the long-context performance of models finetuned with LongCE under different values of K and d, which directly influence its computational efficiency.

First, it can be observed that increasing K or decreasing d significantly improves the efficiency of LongCE (from +79% to +36%/+43%). Second, under these settings, although the model’s performance on real-world tasks (LongBench) slightly decreases, it achieves substantial improvements on synthetic tasks (LongEval, RULER). This suggests that LongCE still holds potential for further efficiency enhancements.

We have included the above result in Appendix B.2.

---

Q5. Do you fine tune with longCE only or with a combination of CE and longCE ?

A5. We fine-tune with LongCE only. To maintain a balance between general language modeling ability and long-context capability, one could also consider combine CE and LongCE during training.

---

Q6. What about using the needle-in-the-haystack test for your evaluation of long context models ?

A6. To evaluate the long-context capabilities over 32k, we conduct the standard Needle-in-a-Haystack (NIAH) evaluation in Appendix B.5. We first test the models obtained in our main text, which are fine-tuned on 32K-length texts. As shown in figure 11, LongCE achieves a score of 10 on 5 out of 6 questions at the 40K length and 2 out of 6 questions at the 48K length, outperforming CE, which achieves a score of 10 on 2 out of 6 and 0 out of 6 questions, respectively. Therefore, LongCE demonstrates a longer effective context length.

Additionally, to demonstrate the generalization ability of LongCE on longer context lengths, we extend the context window of both models by increasing their RoPE base from 500K to 2M. The corresponding NIAH results are shown in Figure 12. The results show that model finetuned with LongCE answers all questions correctly at the 64K length and achieves a score of 10 on 32 sequences with lengths of ≥32K, while CE only achieves this on 26 sequences. This indicates that LongCE can generalize well at longer lengths.

---

Thanks again for your detailed comments. Please let us know if there is more to clarify.

Q5. Why does finetuning with LongCE demonstrate such different results for mistral-7b and llama2 on RULER, with one hardly improving (<1) and one significantly improving (~10)?

A5. Since LongCE uses the model itself as the evaluator, during the early stages of training, the quality of LongPPL re-weighting is closely related to the model itself. Therefore, the difference in performance between Mistral-7B and Llama2-13B is likely related to their initial differences in model capabilities.

---

Q6. Why does figure 6 only contain correlation between longPPL and longbench score? What is the correlation between standard PPL vs longbench score?

A6. Figure 6 shows the correlation between LongPPL and LongBench score when calculated using an evaluator model different from that in Figure 1 (right). Standard PPL, however, does not change with different evaluator models. The correlation between PPL and LongBench is in Figure 1 (left). We now added the clarification in the caption of figure 6.

Additionally, we have summarized their correlation coefficients in the table below, to further clarify the relationship between LongPPL, PPL, and the benchmarks. The results indicate that the correlation between LongPPL and the benchmark remains consistently high.

---

Thanks again for your careful reading and detailed review. Please let us know if there is more to clarify. We are happy to take your further question in the discussion stage.

Q4. The LongCE loss seems to improve mostly the retrieval-based test (LongEval), however, demonstrate little gains on RULER, which has a few non-retrieval tasks, and almost zero improvement on LongBench, which contains quite a few non-retrieval tasks (Figure 7). It is unclear to what extent the proposed method benefits non-retrieval tasks.

A4. We present the detailed LongBench scores of our main experiments of LongCE in the following table. The results show that LongCE not only significantly outperforms CE on retrieval tasks (correspond to “synthetic tasks”), but also excels in summarization and QA tasks. On the contrary, LongCE lags behind in few-shot learning. The reason for this may be that, in few-shot learning tasks, the information in the prompt is denser, and the importance of each token tends to be more equal. In principal, LongCE is better suited for scenarios where the importance of prompt tokens is not equally distributed. The results are added in Appendix B.3.

---

We thank reviewer At6Q for providing detailed comments, and we will address your concerns in the following points.

---

Q1.

The proposed LongPPL is limited by the evaluator model. Concretely, the validity of the evaluation is strictly bounded by the maximum effective length that an evaluator model can reliably handle.

A1.1. Although the performance of LongPPL depends on the capability of the evaluator model, our experiments show that even an 8B-level model is also capable of handling it. Furthermore, since the choice of evaluator model is flexible, the boundaries of LongPPL’s capability will expand as the long-context capability of open-source models improves.

Additionally, the two thresholds, which depend on both the evaluation data and the evaluator model, can further complicate the evaluation process. These limitations weaken the universality of the proposed metric.

A1.2. We conduct an ablation study on the two thresholds in LongPPL (with Qwen2-72B-Instruct) in the following table. We find that when \beta=1, \alpha=1 or 2, the correlation between LongPPL and benchmarks even improves. In fact, we simply reused the hyperparameters summarized from the previous experiments without any overtuning. The results show that LongPPL’s performance is not sensitive to the choice of hyperparameters, with the correlation coefficient being greater than 0.8 in most cases.

We have included the above results in Appendix B.2.

---

Q2. The key token selection of LongPPL is motivated by observing large gains in log probability of the ‘answer’ tokens in LongEval, a retrieval-based synthetic test. It raises the question of whether the gains primarily come from the improved prediction of less frequent n-grams that re-appear later in the context, as studied by [1][2]. If so, the motivation of leveraging an evaluator model is obscure since one can simply evaluate those re-occurred n-gram tokens, which are much more convenient to find and overall a more universal setup than LongPPL.

A2. Following your suggestion, we calculate perplexity solely on the re-occurred N-gram (word-level, N>2) in the inputs. We present the correlation coefficients between the new perplexity metric and the benchmark scores in the table below.

The results show that PPL on re-occurred N-grams has much weaker correlation with model’s long-context capabilities. This indicates that LongPPL’s powerful ability to capture long-context-related information cannot be simply explained by N-grams. However, the experimental results show that N-grams do have some effect, and we will include this discussion in the manuscript (Appendix B.6). Thank you for your constructive insight.

---

Q3. The LongCE loss should be compared with the standard focal loss [3] baseline to understand if it is really necessary to add the contrasting term of long and short context perplexity. The standard focal loss does not require extra forward pass for computing short context log probability, thus can potentially improve training efficiency.

A3. Following your suggestion, we finetune Llama-2-7B with FocalLoss ($\gamma$=2), and the results are shown below. It turns out that FocalLoss is inferior than LongCE, but performs better than CE in LongEval and RULER. This indicates that, although general re-weighting methods like FocalLoss can improve the model’s long-context capability to some extent, the re-weighting method based on long-short contrast in LongCE is more effective in long-context scenarios.

---

Q3. How does the performance of LongPPL and LongCE change when using a weaker model for key token identification? Have you tried using the same model for both evaluation and key token identification?

A3. For LongCE, we exactly use the same model as the evaluator (as explained in line 300). For LongPPL, in Table 5, we presented the LongPPL values when the evaluator model is the same as the model being evaluated, showing no correlation with long-context benchmarks.

Intuitively, if the evaluated model is used to selects key tokens, then under the long-short contrastive mechanism, it can only identify key tokens that it generates well, missing those it cannot generate well. As a result, all models compute perplexity only on tokens they perform well on, which fails to reflect differences in the long-context capabilities of different models. To clarify, we have referenced the content of Table 5 in the main text of the revised version.

---

Q4. The paper discusses the overhead of using LongCE in LLM fine-tuning compared to the original Cross Entropy. How does the evaluation time of using LongPPL compare to traditional perplexity (PPL) metrics?

A4. Please see A2.

---

Thanks again for your constructive suggestions. Hope our explanations can ease your concerns. Please let us know if there is more to clarify.

We thank Reviewer KLF8 for acknowledging the novelty and solidness of our work. Below, we address your main concerns.

---

Q1. LongPPL metric relies on a relatively strong model (medium-sized Qwen2-72B-Instruct) as an evaluator to identify key tokens. This dependence may limit the applicability of the metric in scenarios where such a strong model is not available or practical to use.

A1. We assume that by “strong” and “weak”, you are referring to the model size. Actually, LongPPL also supports using a smaller model as the evaluator model. In Figure 6, 8, 9, we use Llama-3.1-8B as the evaluator model to calculate LongPPL, and achieves high correlation as well.

---

Q2. The additional computational cost associated with identifying key tokens may still be significant. This could be a barrier to adoption in resource-constrained environments.

A2. We first introduce the theoretical complexity of LongPPL (LongCE), and then present their empirical overhead.

Theoretically, the calculation of both LongPPL and LongCE involves two steps: (1) One forward pass on the full text, and (2) several forward passes on short texts (to obtain the LPG score for each token). Step 1, which involves calculations for both PPL (CE) and LongPPL (LongCE), has a time complexity of $O(N^2)$ , where N is the input sequence length (N=32k in this paper). Step 2 represents the computational overhead of LongPPL (LongCE), with a time complexity of $O((N-K)K^2/d)$ , where K is the length of the short context(K=4k in this paper), and d is the sliding window length (d=1k in this paper). The detailed derivation can be found in Appendix A.1.

Empirically, we conduct additional experiments to demonstrate the time cost of LongPPL and LongCE.

LongPPL

We test the time cost of LongPPL in the following table.

It can be observed that the time cost of calculating LongPPL using the 8B model as the evaluator is approximately 3~4 times that of calculating PPL, while the overhead for using the 72B model is much higher.

Although the computational overhead of LongPPL is non-negligible, we believe that such a computational cost will not have a substantial impact on the practicality of LongPPL. On the one hand, if users employ LongPPL as a benchmark, key tokens can be calculated offline, resulting in no online computation overhead. On the other hand, if LongPPL is used as an evaluation metric during training, its computational overhead is negligible compared to the overall training cost (as evaluation steps are typically sparse during training).

We have included the above results in Appendix B.7.

LongCE

In section 4.2, we have already discussed the tradeoff between performance and computational cost of LongCE. To better illustrate this, we conducted an additional set of ablation experiments. In the following table, we present the long-context performance of models finetuned with LongCE under different values of K and d, which directly influence its computational efficiency.

First, it can be observed that increasing K or decreasing d significantly improves the efficiency of LongCE (from +79% to +36%/+43%). Second, under these settings, although the model’s performance on real-world tasks (LongBench) decreases, it achieves substantial improvements on synthetic tasks (LongEval, RULER). This suggests that LongCE still holds potential for further efficiency enhancements. Therefore, we believe that the computational overhead of LongCE is entirely acceptable.

We have included the above results in Appendix B.2.

---

Dear Reviewer KLF8,

We have prepared a response to address your additional questions. Would you please take a look and let us know whether you find it satisfactory?

Thank you! Wishing you a great day!

The Authors

Q6. Is the main reason perplexity fails due to its inability to correctly identify answer/key tokens?

A6. Yes. Since PPL cannot identify key tokens, it treats all tokens equally. However, because key tokens only constitute a small portion of natural text, PPL does not give them enough attention, and thus fails to reflect the ability to handle long texts.

---

Q7. Could you clarify what "task-aware" means in this context? Does it refer to distinctions between short-context and long-context tasks, or could it apply to other task definitions as well?

A7. In the beginning of section 3.1, we mention that the FocalPPL (Eq. 4) is task-aware by modifying the influence function I(x). The term “task” in this article refers to long-contexts, but it can also represent other general tasks (such as toxicity, safety). Here, we use the notion of FocalLoss to make the topic more general, and we will leave these other tasks for future work.

---

Q8. You mention that "even though the Llama-3.1 model we use is only 8B in size, it still obtains LongPPL with a strong correlation, suggesting high compatibility with powerful long-context models, even with smaller parameter sizes." Are you implying that LongPPL would show higher correlation with larger models? If so, could you explain why?

A8. We initially thought that an 8B-level model would not be suitable as an evaluator model, but experiments have shown that it works well. Therefore, the statement you mentioned is actually saying that Llama-3.1-8B works well, rather than suggesting that LongPPL would show higher correlation with larger models.

---

Thanks for your careful reading and detailed review. Please let us know if there is more to clarify. We are happy to take your further question in the discussion stage.

We thank Reviewer kFZS for providing constructive questions for our work. We will address your concerns in the following points.

---

Q1. Some settings in the figures lack clear explanations, which affects the reliability of the figures and the conclusions drawn from them. For example, in Figure 2, the specific prompt lengths for each data point are not provided, making it difficult to interpret the statement, "Each point represents the results obtained from testing at a specific prompt length."

A1. Following your suggestion, we have added the detailed experimental data from Figure 2 in Appendix B.4, including the prompt length corresponding to each point and its associated LongEval Acc and PPL. Additionally, we have included “ranging from 2k to 28k” in the caption of Figure 2 to clarify the range of prompt lengths. We hope these can enhance the clarity of our experiments, and please let us know if there is more to clarify.

---

Q2. Although the correlation arguments seem promising and qualitatively sound, they would benefit from added statistical rigor, such as p-values from hypothesis testing to confirm the significance.

A2. Following your suggestion, we have added the p-values in Figure 1, 5, 6, 8, 9. We summarize them in the following table. The results show that the p-values between LongPPL and the benchmarks are all less than or equal to 0.01, indicating a significant linear correlation. In contrast, the p-values between PPL and the Benchmark are all greater than or equal to 0.49, indicating no significant linear correlation. These results are consistent with the trends reflected by the correlation coefficients.

---

Q3. The naming of Log Probability Gain (LPG) and Log Probability Values (LPV) is not intuitive, which may hinder understanding and adoption of these concepts.

A3. Following your suggestion, we change the naming of LPG and LPV to Long-Short Difference (LSD) and Long-Context Likelihood (LCL), respectively. Hope this will make these concepts easier to understand.

---

Q4. Additionally, the experiments are conducted with a sequence length of 32K, a length considered relatively manageable by recent research. Testing on longer sequences could introduce challenges that may not align with current trends, potentially impacting the generalizability of the results.

A4. To evaluate the long-context capabilities over 32k, we conduct the standard Needle-in-a-Haystack (NIAH) evaluation in Appendix B.5. We first test the models obtained in our main text, which are fine-tuned on 32K-length texts. As shown in figure 11, LongCE achieves a score of 10 on 5 out of 6 questions at the 40K length and 2 out of 6 questions at the 48K length, outperforming CE, which achieves a score of 10 on 2 out of 6 and 0 out of 6 questions, respectively. Therefore, LongCE demonstrates a longer effective context length.

Additionally, to demonstrate the generalization ability of LongCE on longer context lengths, we extend the context window of both models by increasing their RoPE base from 500K to 2M. The corresponding NIAH results are shown in Figure 12. The results show that model finetuned with LongCE answers all questions correctly at the 64K length and achieves a score of 10 on 32 sequences with lengths of ≥32K, while CE only achieves this on 26 sequences. This indicates that LongCE can generalize well at longer lengths.

---

Q5. Lastly, the criteria for selecting experimental models are not clearly defined, which introduces some ambiguity regarding the rationale behind the choice of models used in testing.

A5. We are not sure whether you are referring to LongPPL or LongCE, so we will address your questions separately for each.

• LongPPL. As shown in Appendix D, the models we evaluated cover most of the mainstream long-context models available today. The vast majority of them, such as Qwen, Mistral (Mixtral), Llama, Yi, Phi, FILM, and Yarn, are long-context LLMs listed on RULER. These models encompass different model families as well as various long-context extension techniques (e.g., modification on RoPE, long-context fine-tuning, sliding window attention, etc.).
• LongCE. In Table 2, we use the widely used Llama-2-7B as our base model. In Table 3, we first use Mistral-7B as the base model to show that LongCE is effective across different model families. We then use Llama-2-13B to demonstrate that LongCE is also effective across different model sizes.

---

Dear Reviewer kFZS,

We have prepared a response to address your additional questions. Would you please take a look and let us know whether you find it satisfactory?

Thank you! Wishing you a great day!

The Authors

Q3. Can we use equation (5) directly for LongCE instead of (8). so only differentiate on key tokens, would that be doable?

A3. Following your suggestion, we finetuned Llama-2-7B with LongCE using equation (5) as the reweighting function. The results are given in the table below.

The results show that when directly using the hard criteria in Eq.5 to re-weight LongCE, i.e., calculating gradients only on key tokens, the model’s performance is inferior than using the soft criteria in Eq.8. In fact, the key tokens selected using hard criteria typically account for only a small proportion (approximately 1–2%). Training exclusively on these tokens would result in very low data utilization efficiency. In contrast, methods like LongRecipe [2], which also focus on a subset of tokens for long-context training, use at least 30% of the tokens. Therefore, it is more reasonable to use soft criteria to weight the tokens.

[1] Chen, Guanzheng, et al. "Clex: Continuous length extrapolation for large language models." arXiv preprint arXiv:2310.16450 (2023).

[2] Hu, Zhiyuan, et al. "Longrecipe: Recipe for efficient long context generalization in large languge models." arXiv preprint arXiv:2409.00509 (2024).

[3]Peng, Bowen, et al. "Yarn: Efficient context window extension of large language models." arXiv preprint arXiv:2309.00071 (2023).

[4] Young, Alex, et al. "Yi: Open foundation models by 01. ai." arXiv preprint arXiv:2403.04652 (2024).

[5] Zhang, Yikai, Junlong Li, and Pengfei Liu. "Extending LLMs' Context Window with 100 Samples." arXiv preprint arXiv:2401.07004 (2024).

[6] An, Shengnan, et al. "Make Your LLM Fully Utilize the Context." arXiv preprint arXiv:2404.16811 (2024).

[7] Wang, Suyuchen, et al. "Resonance rope: Improving context length generalization of large language models." arXiv preprint arXiv:2403.00071 (2024).

[8] Chen, Shouyuan, et al. "Extending context window of large language models via positional interpolation." arXiv preprint arXiv:2306.15595 (2023).

---

Thanks again for your constructive suggestions. Please let us know if there is more to clarify.

We thank reviewer aS6q for appreciating the idea of our paper, and we will address your concerns in the following points.

---

Q1. I wonder how would fine-tuned model perform on normal context length benchmarks, will it remain roughly at the same level or regress a bit?

A1. Please see A2 for experimental results on normal context length.

---

Q2. How long is considered as "long-context". In experiments, 32k is used. I wonder how much this method would affect on other context lengths.

A2. Based on our observations of recent research, the term “long context” primarily refers to context lengths that significantly exceeding the 4K length of models like Llama-2-7B. Although recent research has extended text lengths to 64K [1], 128K [2-3], and even longer [4], 32K is still widely accepted as the standard “long context” setting in this field [5-8].

To further address your concerns about the generalizability of LongCE to other context lengths, we conducted the following experiments.

Normal context length (4k/8k). First, we present the performance of models fine-tuned with CE and LongCE on normal context length (4k/8k), as shown in the table below (”0 step“ refers to the vanilla Llama-2-7B). In the default setting in our paper, we used a context length of 4096 for computing the short-context likelihood $P(x_i|s_i)$ in the LongCE loss, aiming at extending to a context longer than 4096. Here, to improve the performance at 4096 as well, we use a shorter context length of 1024 for computing $P(x_i|s_i)$ instead.

Experimental results show that LongCE achieves consistent performance improvements over CE in both normal-length and long contexts. This indicates that, by shortening the length of the short context in LongCE, the model achieves a balance between its capabilities on both long context and normal-length context.

Longer context lengths(>32k). Second, we conduct the standard Needle-in-a-Haystack (NIAH) evaluation in Appendix B.5 to evaluate models’ long-context capability when context lengths is greater than 32K. We first test the models obtained in our main text, which are fine-tuned on 32K-length texts. As shown in figure 11, LongCE achieves a score of 10 on 5 out of 6 questions at the 40K length and 2 out of 6 questions at the 48K length, outperforming CE, which achieves a score of 10 on 2 out of 6 and 0 out of 6 questions, respectively. Therefore, LongCE demonstrates a longer effective context length.

Additionally, to demonstrate the generalization ability of LongCE on longer context lengths, we extend the context window of both models by increasing their RoPE base from 500K to 2M. The corresponding NIAH results are shown in Figure 12. The results show that model finetuned with LongCE answers all questions correctly at the 64K length and achieves a score of 10 on 32 sequences with lengths of ≥32K, while CE only achieves this on 26 sequences. This indicates that LongCE can generalize well at longer lengths.

---