Dear Reviewer Bec7,

Thanks for you support! Below, we provide our responses to your questions in detail.

---

Question 1: Need more details of the RoPE theta scaling coefficient and its corresponding context lengths

Thank you for the suggestion. We will update the x-axis labels in Figures 7 and 8 to include the corresponding effective context lengths for each RoPE configuration. This will make it easier for readers to intuitively understand the relationship between model settings and their actual long-context capabilities.

Here are the x-axis labels we will update:

---

Question 2: Why long-context training can improve short-input reasoning capabilities

Thank you for pointing this out. We conducted the below experiment to support that enhancing long-context capability does not harm—and can even benefit—short-input reasoning tasks, which aligns with prior work such as YARN and Position Interpolation (PI).

We evaluate LLaMA3-8B-Instruct models with different long-context capabilities (RoPE ×1, ×4, ×16) on the MMLU-STEM subset, which consists entirely of short-input, STEM-related reasoning questions. All models were fine-tuned on the same Math dataset.

The results show that stronger long-context capability correlates with improved performance even on short-input reasoning tasks. This suggests that long-context training is not merely an overhead for short tasks, but can reinforce reasoning in general.

---

Question 3: More ablation study on cross-domain generalization, model scale sensitivity and context length configuration

Thank you for the thoughtful suggestions. We have conducted additional experiments to address each of the three aspects raised:

1. Cross-domain generalization (long-context ability benefits not only math resoning but also non-math reasoning): We fine-tuned LLaMA3-8B-Instruct models (RoPE ×1, ×4, ×16) on 20k samples from the science and code domains of the open-thoughts/OpenThoughts3-1.2M dataset. The results show that long-context ability also benefits non-math reasoning tasks.

2. Model scale sensitivity (the effect of long-context ability on reasoning ability generalizes across model scales and architectures): We additionally evaluated the 14B-scale Phi-4 model, using RoPE ×1, ×4, and ×16 configurations. After fine-tuning on Math data, we measured performance on MATH500 and AIME. The consistent trends validate that the effect of long-context ability generalizes across model scales and architectures.

3. Robustness of context length configuration (more evaluation benchmarks on long-context ability of models): To test the robustness of long-context configuration, we evaluated LLaMA3-8B-Instruct under six RoPE scaling levels (×1 to ×64) on multiple long-context benchmarks: Needle-in-a-Haystack (NIAH), LongBench, and RULER. We observe a clear upward trend on these benchmarks from RoPE ×1 to ×16, after which the performance begins to decline.

These ablations support the robustness, scalability, and cross-domain generalization of our central claim.

Dear Reviewer EuNs,

Thank you for your support! Below, we address your concerns point by point.

---

Question 1: What about other benchmarks like GPQA or MMLU?

Thank you for the suggestion. We provide additional evaluation results on other benchmarks, including GPQA and MMLU. These results demonstrate that the performance improvement associated with stronger long-context ability is not limited to math reasoning, but also extends to benchmarks such as MMLU and GPQA.

We first evaluate LLaMA3-8B-Instruct models with different long-context capabilities (RoPE ×1, ×4, ×16) on the MMLU-STEM subset after fine-tuning on the Math dataset. The MMLU-STEM subset is used instead of the full MMLU for two reasons: (1) it focuses on STEM-related reasoning, which better aligns with our goal; (2) it avoids excessive evaluation time, as full MMLU includes over 10k examples.

In addition, we collect 20k samples from the science domain of the open-thoughts/OpenThoughts3-1.2M dataset and fine-tune the models, then evaluate them on GPQA. The results are summarized below:

These results demonstrate that the performance improvement associated with stronger long-context ability is not limited to math reasoning, but also extends to benchmarks such as MMLU and GPQA.

---

Question 2: It would be nice to see other measures of long context reasoning capabilities.

Thank you for the valuable feedback. To address this concern, we further evaluate the LLaMA3-8B-Instruct model with six levels of 32k long-context capabilities (RoPE ×1 to ×64) on both Needle-in-a-Haystack (NIAH) and alternative long-context benchmarks, including LongBench and partial RULER tasks，which contain Multi Query, Value Tracking and Question Answering , better reflecting long-context reasoning beyond retrieval.

The results are summarized below:

We observe a clear upward trend from RoPE ×1 to ×16, after which the performance begins to decline. These results indicate that the long-context capabilities associated with different RoPE scales are generally consistent across NIAH, LongBench, and RULER, thereby supporting our claim that stronger long-context ability consistently leads to better reasoning performance.

---

Question 3: Any conclusions from Table 1 is a bit of a stretch since these are all different models

Thank you for the insightful feedback. Table 1 is intended to illustrate that widely-used LLMs with stronger long-context capabilities tend to perform better on reasoning tasks. Since it is infeasible to control fine-tuning settings—such as the SFT dataset, we instead use a diverse set of popular open-source reasoning models that adopt the same base model (Qwen/Qwen2.5-7B-Instruct) but claim different context lengths, serving as representative examples to support it.

To strengthen this illustration, we have added several new models spanning multiple organizations, training pipelines, and context lengths. The updated results are presented below:

As shown above, models with longer context window tend to achieve better average performance on both Math500 and AIME. We emphasize that Table 1 serves as an empirical motivation. Our main evidence is derived from controlled experiments where only the RoPE scaling is varied (see Table 3).

---

Question 4: Legend in figures 7,9,10 are confusing. Are the line indicators shared across all 3 graphs?

Yes, the line indicators are shared across Figures 7, 9, and 10 for consistency and ease of comparison. We will clarify this in the figure captions to avoid confusion.

---

Question 5: Why does rope scaling start to decrease performance after ~x16?

Thank you for the question. While scaling the RoPE theta helps extend the model’s usable context window, this technique has its limits. Beyond a certain point, further increasing the scaling coefficient leads to degraded performance, as shown by the previous work[1].

In our experiments with LLaMA3-8B-Instruct, we found that RoPE ×16 achieves the best long-context capability, as reflected in both reasoning accuracy and NIAH scores. We speculate one of reasons is: when the scaling goes beyond ×16 (e.g., ×32 or ×64), the model’s internal attention dynamics begin to misalign with training-time inductive biases, leading to worse overall performance despite larger nominal context windows.

[1] NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

Dear Reviewer k9kH,

Thanks for your valuable questions! Below, we address your concerns point by point.

---

Question 1: The author should modify the claim and maybe limit it to math reasoning

Thank you for the insightful comment. Based on our additional experiments, our claim (stronger long-context ability consistently leads to better reasoning performance) holds on other reasoning tasks beyond math, including science and code reasoning.

To verify generality, we fine-tuned three Llama3-8B-Instruct models with different long-context capabilities (RoPE scale ×1, ×4, ×16) on 20k samples from the science and code domains of the open-thoughts/OpenThoughts3-1.2M dataset. The models were then evaluated on GPQA (science) and Livecode (code). Evaluation results show that stronger long-context ability consistently leads to better reasoning performance across these domains.

---

Question 2: Confounding variables not isolated. In-depth reasons should be briefly discussed and verified

Thank you for the insightful comment. Controlling confounding variables—so that long-context ability is the only varying factor—has been a central goal of our experimental design.

To this end, we carefully evaluate model performance before and after extending the context window. As shown in Table, LLaMA3-8B-Instruct models with longer context window do not improve performance on reasoning tasks before SFT; in fact, performance often decreases. This indicates that mere window extension does not yield a reasoning boost. However, once the models are fine-tuned using the same datasets and hyperparameters, their reasoning abilities improve significantly in correlation with long-context capacity.

We agree that exploring deeper mechanisms is important. One plausible hypothesis—that longer context ability improves attention concentration across the whole context—is indeed worth investigating. We will include this point in the discussion section and highlight it as a promising direction for future work.

---

Question 3: Limited discussion on base model

Thank you for raising this important point. Based on our additional experiments as follows, our claim (stronger long-context ability consistently leads to better reasoning performance) holds across different model families, including LLaMA, Qwen, and Phi.

In the main paper, we already evaluated LLaMA3-8B-Instruct and Qwen2.5-7B-Instruct, covering two widely used model series. To further verify the generality of our findings, we additionally trained Phi-4 models with varying long-context abilities (RoPE scale ×1, ×4, ×16) and evaluated them on two math reasoning benchmarks: Math500 and AIME.

These results further confirm that the benefit of enhancing long-context ability is not tied to a specific model architecture, but generalizes across different model families. We will clarify this in the revised version.

---

Question 4: Can you provide some examples of both the short and long outputs exceeds 10,000 tokens?

Thank you for the question. To provide further insight, we count how many correct and incorrect generations exceed 10,000 tokens under both short and long training with RoPE theta x 16 on MATH500: (due to limited word number of retbuttal, since the responses are too long, we can not paste responses here but we will include them in the updated version of our paper.)

We observe that models trained under the long-data setting generate fewer wrong ultra-long outputs than models trained under the short-data setting. Our hypothesis is that long-data training helps the model learn how to reason more efficiently, thus avoiding unnecessarily verbose or incorrect completions to generate wrong answers.

---

Question 5: Line 113 and other parts: Lowercase and uppercase should be consistent in the section title.

Thank you for pointing this out. We will revise the section titles to ensure consistent capitalization throughout the paper in the final verison.

Dear Reviewer EVh5,

Thank you for your support! Below, we provide our responses to your questions in detail.

---

Question 1: Table 1 may not be an example to apply comparisons

Thank you for the insightful feedback. Table 1 is intended to illustrate that widely-used LLMs with stronger long-context capabilities tend to perform better on reasoning tasks. Since it is infeasible to control fine-tuning settings—such as the SFT dataset, we instead use a diverse set of popular open-source reasoning models that adopt the same base model (Qwen/Qwen2.5-7B-Instruct) but claim different context lengths, serving as representative examples to support it.

To further support this illustration, we have added several new models—spanning different organizations, training pipelines, and context lengths—into the comparison. The results below show consistent patterns across both 32k and 128k context settings:

As shown above, models with stronger long-context abilities outperform those with shorter context in both math benchmarks. We emphasize that Table 1 serves as an empirical motivation. Our main evidence is derived from controlled experiments where only the RoPE scaling is varied (see Table 3)

---

Question 2: Can we do evaluations on pre SFT models?

Thanks for this insightful suggestion. In our submission, We have evaluated models before SFT. In Table 3, we have already conducted controlled experiments where the only changing factor is RoPE scaling. These results show that base model accuracy before SFT decreases as RoPE scaling increases, while accuracy after SFT increases with context capability—further supporting our claim (stronger long-context ability consistently leads to better reasoning performance).

Here we show the partial results from Table 3. We could see the long-context extension method does not improve model ability before training.

---

Question 3: Can we run RL on these models?

Thank you for raising this insightful question. To explore whether long-context variants of base models can support stronger reasoning in RL, we attempted reinforcement learning (RL) training on different configurations.

We used the Qwen2.5-7B-Instruct model and a merged variant (0.9 × Qwen2.5-7B-Instruct + 0.1 × Qwen2.5-7B-Instruct-1M) as initialization. For the RL algorithm, we adopted Dr.GRPO as our training strategy. Due to resource and time constraints, we conducted RL training on a small dataset of 5k math samples.

After training, the average output length of the model remained below 2k tokens throughout. The final accuracy on MATH500 reached 74.72 and 74.80, respectively, for the two model variants. These results suggest that our current RL setup—with limited data and small-scale tuning—was insufficient to unlock complex strong reasoning and the relation. We consider this an exciting direction for future work and will include this discussion in the final paper.

---

Question 4: Why is the output length decreasing with the long context extension?

Thank you for the question. The observed decrease in output length is actually aligned with ratio of correct answer. With the long context extension, the number of correct answer (which is shorter tipically) increases, thus the average output length decreases, which also been discussed in prior work [1]. To support this with evidence, we show the average lengths of correct and incorrect generations across both short (0–8k) and long (8–16k) training settings for LLaMA3-8B under different RoPE configurations:

• RoPE ×1 (8k context)

• RoPE ×16 (Extended context)

As shown, models with longer context capacity (e.g., RoPE ×16) achieve higher accuracy while generating shorter outputs on average—primarily because a larger portion of the generations are correct, and correct answers tend to be shorter and more concise.

[1]Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs