Training Large Language Models to Reason Efficiently

We thank the reviewer for their thoughtful and constructive feedback. We would like to address their main concerns and clarify a few points raised in the review.

We acknowledge the concern about computational cost. However, we believe it is important to distinguish between training and deployment. The cost of training is amortized over a large number of inference calls, as training is a one-time process. Our method aims to significantly reduce inference-time compute, which can be the dominant cost at scale in real-world deployments.

Comparing against current LRM:

We appreciate the reviewer’s suggestion regarding comparison with methods such as DeepSeek-Zero. However, we believe there may be a misunderstanding here. DeepSeek-Zero focuses on training new reasoning models from base LLMs using verifiable outcomes, whereas our method is designed to post-train existing reasoning models. Our goal is not to improve reasoning capability from scratch, but to make already capable models more efficient at inference time while preserving accuracy. Thus, the objectives and use cases of the two approaches are fundamentally different.

W1: Choice of $\alpha$

We thank the reviewer for highlighting this important point. The sensitivity to the $\alpha$ parameter is indeed a feature, not a flaw. Our method is deliberately designed to offer flexibility—rather than outputting a single model, it yields a family of models with different efficiency-accuracy trade-offs which can be obtained by varying $\alpha$. This allows users to select a model that best fits their application needs, whether they prioritize cost savings or accuracy.

W2: Not enforcing exact token limits

We appreciate the reviewer’s suggestion of enforcing strict token limits. We considered this design choice, but found that such constraints can lead to brittle behavior. Real-world problems vary in complexity, and harder problems naturally require more reasoning steps. Our approach encourages adaptive computation: the model spends fewer tokens on easier problems while allocating more tokens to harder ones, all while maintaining high accuracy. This adaptive behavior is a key strength of our method.

W3: Other benchmarks

Thank you for this suggestion. Following the reviewer's recommendation, we evaluated our method on CommonSenseQA [1] and the Logical Deduction task from BIG-Bench [2], which are out-of-distribution compared to our original math benchmarks. The results are available at: https://imgur.com/a/FEbdRL7. The plots demonstrate that our method generalizes to prompts that are out of distribution such as problems in CommonSenseQA and Logical Deduction. For instance, for $\alpha=0.2$, we get 40% reduction in tokens but only 1.1% drop in accuracy for the 7B model on CommonSenseQA. Similarly, for Logical Deduction with $\alpha=0.2$, we get a 50.7% reduction in number of tokens but only 3.5% drop in accuracy on Logical Deduction.

References

[1] CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge by Talmor et al. [https://aclanthology.org/N19-1421/]

[2] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models by Srivastava et al. [https://arxiv.org/abs/2206.04615]

We thank the reviewer for the positive and encouraging feedback. We're glad the clarity, simplicity, and effectiveness of our method came through, and appreciate your recognition of its generalizability.

Should the reviewer have any further questions, we would be happy to discuss them.

We sincerely thank the reviewer for their thoughtful feedback. We appreciate that the reviewer recognized the practical relevance of our work and the simplicity of our proposed solution.

[1] Evaluating on non-math tasks.

We appreciate the suggestion to evaluate the method on tasks beyond mathematical reasoning. Following this, we conducted experiments on CommonSenseQA [1] (commonsense reasoning) and Logical Deduction from BIG-Bench [2] (logical reasoning). Results are available at https://imgur.com/a/FEbdRL7. These plots indicate that our approach generalizes well to out-of-distribution prompts. For example, on CommonSenseQA, with $\alpha = 0.2$, we observed a 40% reduction in tokens with only a 1.1% drop in relative accuracy using the 7B model. On Logical Deduction, the same $\alpha$ value led to a 50.7% token reduction and just a 3.5% drop in accuracy. These results support the broader applicability of our method beyond the math domain.

[2] On RL instability

Thank you for highlighting this important concern. We fully agree that reproducibility and stability are critical in RL-based methods. We are committed to open-sourcing all our code and training configurations to facilitate replication. In our experience, training has been stable across multiple runs. Training dynamics are visualized at https://imgur.com/a/SxN5Id3, and we did not observe any unexpected divergence or instability.

[3] Theoretical discussion around the reward

We acknowledge that the theoretical underpinnings of length-penalized reward functions are still developing. Our current formulation represents an initial attempt to explore this trade-off space. One open question we raise for future work is whether a Pareto-optimal reward function exists that more effectively balances accuracy and efficiency. We hope this paper serves as a stepping stone for deeper theoretical exploration in this area.

[4] Why length decreases when using $\alpha=0$

This was indeed an intriguing observation for us as well. Recent work by Liu et al. [3] points to a bias in the GRPO loss function: it averages per-token loss across entire sequences, which unintentionally favors shorter correct sequences over longer correct ones, and longer incorrect sequences over shorter incorrect ones. This may explain the unexpected reduction in reasoning length, even when $\alpha=0$. We tested the fix proposed in [3] and observed that the length reduction disappears when applying it. Table 1 below shows normalized accuracy and token usage (relative to a baseline 7B distilled model). The average results highlight that the fix mitigates the unintended length bias.

Table 1: Table showing the effects of fixes proposed in [3]. $\Delta$ NT refers to change in normalized tokens. $\Delta$ NA refers to change in normalized accuracy. All numbers are normalized based on the Baseline scores. All experiments have been conducted on the 7B Distilled model.

References

[1] CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge by Talmor et al. [https://aclanthology.org/N19-1421/]

[2] Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models by Srivastava et al. [https://arxiv.org/abs/2206.04615]

[3] Understanding R1-Zero-Like Training: A Critical Perspective, Liu et al. [https://arxiv.org/pdf/2503.20783]

We thank the reviewer for the thorough and insightful comments, which have greatly helped us improve the clarity and rigor of our manuscript.

Below, we address the reviewer’s major concerns:

[1] Missing baseline:

We appreciate the reviewer highlighting the missing baseline. However, both the 'prompted truncation' and 'TALE' baselines rely on the assumption that the language model can effectively follow explicit instructions, such as "Respond in less than 500 tokens." Our empirical findings suggest that smaller reasoning-focused models (e.g., the DeepSeek distilled models used in our experiments) lack robust instruction-following capabilities. Consequently, as demonstrated in the tables below, there is no meaningful correlation between the instructed token limit and the actual length of generated responses:

Table 1: Number of tokens generated for varying token limits using Distilled-R1-Qwen-1.5B on MATH500:

Table 2: Number of tokens generated for varying token limits using Distilled-R1-Qwen-7B on MATH500:

Prompt used:

"Please think step by step and answer in less than X tokens. Question: {question} Answer:"

Given this limitation, our proposed method offers an advantage by not relying on explicit token-length instructions, ensuring broader applicability and effectiveness for models that lack reliable and general instruction-following abilities. We will add the comparison with these baselines in the next iteration of the manuscript.

We appreciate the reviewer’s suggestion regarding the additional baselines and will include them in our citations. However, we believe that a direct comparison with some of these works may fall outside the scope of this paper. For example, Snell et al. examine scenarios involving parallel sampling from the LLM, whereas Bansal et al. focus on the training aspect.

[2] Clarification regarding unsuccessful experiments:

We apologize for the confusion caused by our previous phrasing. To clarify, our initial exploratory experiments focused on fine-tuning smaller instruct models (Qwen2.5-1.5B and Qwen2.5-3B) using extended reasoning demonstrations from QwQ-32B-Preview. However, these fine-tuned models unexpectedly showed decreased performance compared to their instruct counterparts:

This finding aligns with previously reported observations in the literature [1], suggesting that such fine-tuning may negatively impact smaller instruct-model performance. Due to this challenge, we postponed further experiments until the recent release of highly performant small-scale reasoning models by DeepSeek [2]. The superior capabilities of these new models provided a suitable foundation to test and validate our proposed method effectively.

[3] Use of the term "dynamic":

We apologize for any ambiguity caused by our use of the adjective "dynamic." Originally, our intention was to highlight the adaptive nature of the response-length reduction. To ensure clarity, we will omit the word "dynamic" and instead explicitly state that reductions in response length are more pronounced for easier problems and less so for harder ones.

[4] Minor suggestions:

We thank the reviewer for pointing out these minor but important details. We will carefully implement these corrections and improvements, significantly enhancing the manuscript's clarity and readability.

We appreciate the reviewer’s valuable feedback, which has notably strengthened our manuscript.

References

[1] LIMR: Less is More for RL Scaling by Li et al. [https://arxiv.org/pdf/2502.11886]

[2] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning [https://arxiv.org/abs/2501.12948]