A Controlled Study on Long Context Extension and Generalization in LLMs

We thank reviewer for the question, and we clarify our contributions below,

Infrastructure contribution: Our study’s primary contribution is not a new algorithm but a standardised, fully reproducible testbed that eliminates three confounding factors that have hampered prior work: (i) heterogeneous base checkpoints, (ii) mismatched pre-training corpora, and (iii) inconsistent hyper-parameters during fine-tuning.

By releasing (a) the unified data pipeline, (b) a set of five identical base models exposed to a single 1 B-token long-context corpus, and (c) templated evaluation scripts for both intrinsic and extrinsic tasks, we provide the first public “apple-to-apples” suite.

Unified mathematica view : we summarize and yield an explicit frequency-scaling equivalence that mathematically links PI ↔ NTK ↔ YaRN ↔ CLEX (Eq. 7–10), making cross-method trade-offs transparent.

Findings: novel take-aways now highlighted in our paper:

(i) Exact finetuning vs. approximate attention – exact RoPE variants preserve retrieval depth up to 4× their finetuned window, whereas all approximate schemes collapse beyond 1× (Fig. 1), clarifying when efficiency shortcuts lose fidelity.

(ii) Perplexity correlates with downstream accuracy in limited length to some extent once confounders are removed (Kendall t = -0.72 on RULER; Table 6) – resolving contradictory claims in prior work. And we will add some recent experiments and discovered requested to further elaborate the limitation of PPL and downstream task accuracy.

We thank reviewer for their feedback. We agree “near-infinite” was overstated; we have replaced it with “beyond-training” throughout.

In our study, we define generalization as the model's ability to perform well across all tasks that extend beyond the training context length. Specifically, we have evaluated our models on tasks where the input lengths exceed 64k tokens, such as RULER, with sequences up to 128k tokens in Table 1. We will revise our writing to highlight this in our manuscript. In our original paper, we found that NTK-Dynamic yields the best performance beyond 32k.

Table 1: Generalization of NTK beyond 32k on RULER

To further evaluate the generalization, we evaluated sequences up to 128k tokens on HELMET with Llama-3 (8B). Results are shown below.

Table 2: Llama3 with up to 128k evaluation on HELMET

Take-away: even the strongest exact method degrades when extrapolated 4 × beyond its fine-tuned window, confirming the reviewer’s intuition. This indicates that even for the best generalized methods we discovered in our controlled setting, their generalization becomes weaker when the context length is much larger than the fine-tuned length.

Dear Reviewer uEtu,

As this is nearing the end of the author response period, we kindly ask if there is anything additional that you would like us to address. We appreciate your valuable suggestions and feedback.

Sincerely,

Authors

“NTK-F” stands for NTK-Frozen, i.e. NTK positional rescaling applied to a frozen base model with no further fine-tuning. We will expand the table caption and glossary to spell this out and avoid confusion.

We thank reviewer for the question, and we clarify our contributions below,

Infrastructure contribution: Our study’s primary contribution is not a new algorithm but a standardised, fully reproducible testbed that eliminates three confounding factors that have hampered prior work: (i) heterogeneous base checkpoints, (ii) mismatched pre-training corpora, and (iii) inconsistent hyper-parameters during fine-tuning.

By releasing (a) the unified data pipeline, (b) a set of five identical base models exposed to a single 1 B-token long-context corpus, and (c) templated evaluation scripts for both intrinsic and extrinsic tasks, we provide the first public “apple-to-apples” suite.

Unified mathematica view : we summarize and yield an explicit frequency-scaling equivalence that mathematically links PI ↔ NTK ↔ YaRN ↔ CLEX (Eq. 7–10), making cross-method trade-offs transparent.

Findings: novel take-aways now highlighted in our paper:

(i) Exact finetuning vs. approximate attention – exact RoPE variants preserve retrieval depth up to 4× their finetuned window, whereas all approximate schemes collapse beyond 1× (Fig. 1), clarifying when efficiency shortcuts lose fidelity.

(ii) Perplexity correlates with downstream accuracy in limited length to some extent once confounders are removed (Kendall t = -0.72 on RULER; Table 6) – resolving contradictory claims in prior work. And we will add some recent experiments and discovered requested to further elaborate the limitation of PPL and downstream task accuracy.

Our observation about generalization is particularly evident on RULER: only NTK and CLEX are able to maintain reasonable performance as the evaluation length increases. In contrast, Landmark’s performance degrades significantly with longer contexts, suggesting limited generalization ability in this setting.

Table 1: Llama2 on RULER Benchmark

We also notice that the perplexity scores of LongLoRA in Table 2 are significantly larger than those of other methods.

In Appendix L, we conduct a detailed study and find that hyperparameters can significantly influence the performance of different context extension methods, particularly approximate attention methods like LongLoRA. We sweep key hyperparameters such as batch size and learning rate.

The results are summarized below:

Table 1: Perplexity Results of LongLoRA on PG19 and Proof-file

PG19

Proof-file

We draw the following observations:

High Sensitivity: Approximate attention methods like LongLoRA are highly sensitive to hyperparameter settings. Small changes in batch size or learning rate lead to significant changes in performance.

Robustness Comparison: In contrast, NTK and YaRN demonstrate strong robustness across hyperparameter choices, maintaining stable perplexity across settings.

Optimization Difficulty: LongLoRA often requires much more careful hyperparameter tuning to reach optimal performance, making training more costly and less predictable.

We find that LongLoRA exhibits a similar trend to PI and YaRN: these methods effectively maintain low perplexity within the pretraining context length but fail to generalize to longer sequences.

We will acknowledge the well-known limitations of perplexity (PPL) as a long-context metric, citing the analyses of Fang (2025)[1] and Gao (2024)[2]. We will add new Section 5.3, “When does PPL succeed / fail?”, to address this request. Specifically,

First, we will summarize Fang et al.’s observation that token-averaged PPL can mask large errors on a handful of key tokens, making it an incomplete proxy for downstream quality.

Next, we will report that in our controlled setting—where both the model architecture and the fine-tuning data are held constant—PPL remains a weak predictor of task performance, with a correlation of -0.6 on the HELMET benchmark.

We further computed the Kendall correlation between downstream task performance and perplexity, and found that HELMET exhibits a stronger correlation compared to benchmarks such as RULER and LongBench.

Table 1: Kendall correlation of downstream task performance and PPL

Finally, we emphasize that future evaluations should pair PPL with richer suites such as HELMET so that both token-level and task-level behaviors are captured.

References

[1] Fang L, Wang Y, Liu Z, et al. What is Wrong with Perplexity for Long-context Language Modeling?[J]. arXiv preprint arXiv:2410.23771, 2024.

[2] Gao T, Wettig A, Yen H, et al. How to train long-context language models (effectively)[J]. arXiv preprint arXiv:2410.02660, 2024.

We thank the reviewer for the references to improve the comprehensiveness of our paper. While we are not able to make changes to our related work section at the moment, we are going to add the following references covering recommended papers from reviewer.

References (newly cited)

[1] HELMET – Yen et al., How to Evaluate Long-Context LMs Effectively and Thoroughly, 2024.

[2] NSA – Native Sparse Attention: Hardware-Aligned & Natively-Trainable Sparse Attention, 2025.

[3] MoBA – Mixture of Block Attention for Long-Context LLMs, 2025.

[4] DuoAttention – Efficient Long-Context LLM Inference with Retrieval Heads, 2024.

[5] ProLong – Gao et al., How to Train Long-Context LMs (Effectively), 2024.

[6] LongSkywork – Zhao et al., A Training Recipe for Extending Context Length, 2024.

[7] ABF – Xiong et al., Effective Long-Context Scaling of Foundation Models, 2023.

[8] PPL limitations – Fang et al., What Is Wrong with Perplexity for Long-Context LM?, 2024.

Thank you for your dedicated review of our work. In the following, we will carefully respond to your questions.

“Evaluation protocol (“out-dated benchmarks”)”

We appreciate the reviewer’s insightful comments regarding our evaluation choices. We agree that our original evaluation setup is somewhat outdated. To address this concern, we have extended our evaluation to include the HELMET[1] benchmark, which offers a more comprehensive and challenging suite of long-context tasks. In particular, we evaluated both LLaMA2 and LLaMA3 variants using HELMET’s standard protocol. Table 1: Llama3 on HELMET

Table 2: Llama2 on HELMET

Our Take-away: The newer HELMET tasks reinforce our original conclusion — full attention + NTK fine-tuning remains dominant; gains even widen on the 128 k slice (see Table 1).

[1] HELMET – Yen et al., How to Evaluate Long-Context LMs Effectively and Thoroughly, 2024.

“Compute budget / efficiency?”*

We conducted efficiency analysis under controlled conditions using the same hardware setup in Appendix E.

As shown in Table 1, we observed that approximate attention methods are indeed faster, achieving a speedup of approximately 1.5x to 2x compared to LLaMA when the context length is short; however, when the context length gets longer, we didn't see a significant margin.

We hypothesize that the discrepancy between the theoretical FLOPs-based comparisons and the observed speedup arises due to differences in hardware characteristics and CUDA implementations of the respective methods.

Table 3: Efficiency analysis of prefill stage time cost, decoding speed, and memory usage The prefill time cost represents the time required to generate the first token. The decoding speed (seconds / per token) is averaged over 100 token inferences at each sequence length. Memory consumption corresponds to the peak GPU memory usage during inference. All methods, except for LM-Infinite and Landmark, utilize Flash-Attention 2 for enhanced computational efficiency.

“Experiments stop at 64 k; not ‘near-infinite’ ”

We agree “near-infinite” was overstated; we have replaced it with “beyond-training” throughout.

In our study, we define generalization as the model's ability to perform well across all tasks that extend beyond the training context length. Specifically, we have evaluated our models on tasks where the input lengths exceed 64k tokens, such as RULER[1], with sequences up to 128k tokens in Table 1.

We will revise our writing to highlight this in our manuscript. In our original paper, we found that NTK-Dynamic yields the best performance beyond 32k.

Table 1: Generalization of NTK beyond 32k on RULER

To further evaluate the generalization, we evaluated sequences up to 128k tokens on HELMET[2] with Llama-3 (8B). Results are shown below.

Table 2: Llama3 with up to 128k evaluation on HELMET

Take-away: even the strongest exact method degrades when extrapolated 4 × beyond its fine-tuned window, confirming the reviewer’s intuition. This indicates that even for the best generalized methods we discovered in our controlled setting, their generalization becomes weaker when the context length is much larger than the fine-tuned length.

References

[1] Hsieh C P, Sun S, Kriman S, et al. RULER: What's the Real Context Size of Your Long-Context Language Models?[J]. arXiv preprint arXiv:2404.06654, 2024.

[2] Yen H, Gao T, Hou M, et al. Helmet: How to evaluate long-context language models effectively and thoroughly[J]. arXiv preprint arXiv:2410.02694, 2024.

We thank the reviewer for the thoughtful feedback and for recognizing the rigor of our controlled protocol (fixed base models, data, and hyper-parameters).

“Analysis … not deep, limited insights.”

We thank reviewer for the question, and we clarify our contributions below,

Infrastructure contribution: Our study’s primary contribution is not a new algorithm but a standardised, fully reproducible testbed that eliminates three confounding factors that have hampered prior work: (i) heterogeneous base checkpoints, (ii) mismatched pre-training corpora, and (iii) inconsistent hyper-parameters during fine-tuning.

By releasing (a) the unified data pipeline, (b) a set of five identical base models exposed to a single 1 B-token long-context corpus, and (c) templated evaluation scripts for both intrinsic and extrinsic tasks, we provide the first public “apple-to-apples” suite.

Unified mathematica view : we summarize and yield an explicit frequency-scaling equivalence that mathematically links PI ↔ NTK ↔ YaRN ↔ CLEX (Eq. 7–10), making cross-method trade-offs transparent.

Findings: novel take-aways now highlighted in our paper:

(i) Exact finetuning vs. approximate attention – exact RoPE variants preserve retrieval depth up to 4× their finetuned window, whereas all approximate schemes collapse beyond 1× (Fig. 1), clarifying when efficiency shortcuts lose fidelity.

(ii) Perplexity correlates with downstream accuracy in limited length to some extent once confounders are removed (Kendall t = -0.72 on RULER; Table 6) – resolving contradictory claims in prior work. And we will add some recent experiments and discovered requested to further elaborate the limitation of PPL and downstream task accuracy.

“Why does approximate attention degrade?”

While the purpose of this paper is not to improve over existing methods, we have the following hypotheses,

1. Information bottleneck (two-hop routing).
   Block-wise schemes such as Landmark Attention compress every B tokens into a single landmark; a query must therefore travel token → landmark → token, discarding fine-grained cues. Pointer-style tasks (e.g., Needle-in-a-Haystack) are the first to collapse[1].

2. Phase aliasing with RoPE.
   Sparse or chunked heads act as a low-pass filter on RoPE’s complex phases: high-frequency components (large \omega) are pruned, so accumulated phase error grows with hop count and the dot-product logits vanish behind the softmax. A recent analysis of RoPE’s failure modes confirms this aliasing effect[2].

3. Length-generalization failure.
   During fine-tuning, approximate kernels see a maximum relative distance d_train; at inference they are queried at d’ ≫ d_train. LM-Infinite[3] apply the sliding window mechanism to discard distant contexts, which keeps the input length does not exceed the context window. However, LM-Infinite can only attend to the tokens within the local window, which leads to a rapid decline in its performance as the sequence length increases[4][5].

4. Hardware-bound indirection costs.
   Many “sub-quadratic” kernels rely on gather/scatter or landmark sorting; on modern GPUs these memory-bound operations dominate wall-time, erasing the FLOP advantage once context length exceeds ≈ 32 k. Systems work on million-token inference using Context Parallelism observes the same bottleneck[6][7].

References

[1] A. Mohtashami & L. Alibeigi. Landmark Attention: Random-Access Infinite Context Length for Transformers. arXiv:2305.16300, 2023.
[2] I. Y. Men et al. Round and Round We Go! What Makes Rotary Positional Encodings Work—and Fail—in Long Contexts. OpenReview, 2025.
[3] C. Han et al. LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models. arXiv:2308.16137, 2023.
[4] Chaojun Xiao et al. InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory. arXiv:2402.04617, 2024.
[5] Yi Lu et al. LongHeads: Multi-Head Attention is Secretly a Long Context Processor. arXiv:2402.10685, 2024.
[6] Z. Wang et al. Efficient Infinite-Context Transformers with Infini-Attention. arXiv:2404.07143, 2024.
[7] Y. Chen et al. Context Parallelism for Scalable Million-Token Inference. arXiv:2411.01783, 2024.

Thanks for the response, I will maintain my score since it's still not fully addressing long-context problem.

Dear Reviewer RShr,

As this is nearing the end of the author response period, we kindly ask if there is anything additional that you would like us to address. We appreciate your valuable suggestions and feedback.

Sincerely, Authors