We thank the reviewer for their insightful questions and appreciate the fact that they found our paper simple to read and our method novel.

Below, we address their concerns:

1. Breakdown of results per dataset:

The results for every dataset individually can be found in Section C of the Appendix. We put them there because of a lack of space in the main paper. Figure 3 shows that the reduction in relative response length is dependent on the hardness of the dataset as we mention in line 292. We acknowledge that the presentation of Figure 3 could be improved to highlight the actual message. We will improve upon it in the final version of the paper.

2. What if we start with a base model?

We thank the reviewer for asking the question.

Base models such as Qwen or Llama generally do not exhibit long CoTs before RLVR training. However, as training progresses their response length and accuracy increase. In general, training a base model for reasoning is a very compute-intensive task[1]. However, we investigate the effect of length-controlled training on a small scale replication of long CoT behaviour in the Countdown task as done in TinyZero [2].

We use the Qwen2.5-3B base model and run RLOO with and without length penalty (with $\alpha=0.1$) with a context window limit of 2048 tokens. We make the following observations as training progresses:

a. Convergence is slower when training using the length penalty

Vanilla RLOO reaches peak performance at about 220 RL steps, after which it saturates due to context window limits whereas our method reaches peak performance at about 400 RL steps.

b. Vanilla RLOO reaches higher peak performance

Vanilla RLOO reaches a peak performance of 67.48% on the validation set whereas length-regularization reaches a peak performance of 59.68%

c. During training, when controlled for performance, length penalized training offers a more efficient model

As training progresses and performance improves, vanilla RLOO’s response length grows significantly. However, for the same training accuracy in length-regularized RLOO, the corresponding response length is significantly lower. Notice that we have only tried one combination of hyperparameters.

Results are given in the table below:

We also observe syntactic differences in the responses learnt by both models:

For instance, an example response for the vanilla RLOO model was:

User: Using the numbers [71, 3, 72], create an equation that equals 95. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>. 

Assistant: Let me solve this step by step. <think>72 + 71 = 143, then 143 - 3 = 140, but this doesn't work. Try: 3 + 72 = 75, then 75 + 71 = 146, but this doesn't work. Try: 71 * 3 = 213, then 213 - 72 = 141, but this doesn't work. Try: 71 + 72 = 143, then 143 - 3 = 140, but this doesn't work. Try: 72 * 3 = 216, then 216 - 71 = 145, but this doesn't work. Try: 3 * 72 = 216, then 216 - 71 = 145, but this doesn't work. Try: 3 + 72 = 75, then 75 * 71 = 5425, but this doesn't work. Try: 71 / 3 = 23.(3), then 23.(3) + 72 = 95.73, but this doesn't work. Try: 71 - 72 = -1, then -1 + 3 = 2, but this doesn't work. Try: 72 / 71 = 1.(03), then 1.(03) + 72 = 73.(03), but this doesn't work. Try: 72 - 71 = 1, then 1 + 3 = 4, but this doesn't work. Try: 3 * 71 = 213, then 213 + 72 = 285, but this doesn't work. Try: 71 - 72 = -1, then -1 * 3 = -3, but this doesn't work. Try: 3 * 72 = 216, then 216 + 71 = 287, but this doesn't work. Try: 72 - (71 * 3) = 72 - 213 = -141, but this doesn't work. Try: (72 + 71) * 3 = 143 * 3 = 430, but this doesn't work. Try: 71 + (72 * 3) = 71 + 216 = 287, but this doesn't work. Try: 71 - (72 / 3) = 71 - 24 = 47, but this doesn't work. Try: 72 - (71 / 3) = 72 - 23.(3) = 48.(7), but this doesn't work. Try: 72 + 71 / 3 = 72 + 23.(3) = 95.(3), but this doesn't work. Try: 71 - (72 - 3) = 71 - 75 = -4, but this doesn't work. Try: 71 + 72 - 3 = 143 - 3 = 140, but this doesn't work. Try: 72 / 71 * 3 = 3 * (72 / 71) = 3 * 1.(03) = 3.(09), but this doesn't work. Finally: 3 * (71 + 72) = 3 * 143 = 43.(9), but this doesn't work. Try: 71 - 72 + 72 * 3 = -1 + 216 = 215, but this doesn't work. Ah-ha! 72 - 71 + 72 * 3 = 1 + 216 = 217, but this doesn't work. One more: 3 + 71 * 72 = 3 + 5112 = 5115, but this doesn't work. There goes: (71 + 72) / 3 = 143 / 3 = 47.(6), but this doesn't work. One more: 72 - 71 - 72 / 3 = 1 - 24 = -23, but this doesn't work. There is none: 72 / (71 - 72) = 72 / 1 = 72, but this doesn't work. Which one finally: 72 - (71 + 72 / 3) = 72 - (71 + 24) = 72 - 95 = -23, but this doesn't work. Think deep: 71 - 72 + 72 / 3 = -1 + 24 = 23, but this doesn't work. Finally: <answer>71 - 72 / 3 + 72</answer></thought>

But for the regularized model an example response looked like:

User: Using the numbers [32, 45, 76, 54], create an equation that equals 53. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.

Assistant: Let me solve this step by step. <think> 76-54=22 22+45=67 67-32=35 Nope Try: 54+45=99 99-76=23 23+32=55 Nope Try: 76+54=130 130-45=85 85-32=53 This works! </th>
<answer>76+54-45-32</answer>

Our results in this toy setup demonstrate that even though length-regularization slows down convergence and reduces peak performance, it discovers more token-efficient solutions when controlled for performance. Due to this, we believe that our method, as intended in the paper, would be more effective when used as a post training method compared to doing RLVR from scratch on a base model.

We thank the reviewer again for asking the question, and we are happy to include a section about this experiment in the camera ready if the paper is accepted, which we believe will increase the quality of the manuscript.

References:

[1] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models, arXiv 2025

[2] TinyZero: Jiayi Pan and Junjie Zhang and Xingyao Wang and Lifan Yuan and Hao Peng and Alane Suhr

We thank the reviewer for their comments on the paper’s importance, simplicity and theoretical backing. We address their main concerns below:

1. Baselines

We thank the reviewer's suggestion about running more baseline. As the reviewer noticed, there are several concurrent works, many of which also do not release code. Following the reviewers suggestions, we ran O1-Pruner [1] on the DeepSeek-Distill-Qwen-7B model using the official code available online and we adapted it to our dataset and model and ran it with the hyperparameters in their codebase. We chose $\lambda=2$ since Table 3 in their paper recommends it. Our method outperforms O1-Pruner by generating models which have higher accuracy at a lower token cost (specifically $\alpha=0.05$ and $\alpha=1$). Detailed results are present in the table below:

2. Only works when binary ground truth is available

We acknowledge that this is a limitation and extending this to problems without binary rewards is definitely a question of open research. We will add this to our Limitations Section. One simple way to solve this problem would be to simply modify the reward function $r(x, y)$ for any task where the LLM generates responses to $r’(x, y) = r(x, y)*(1-f(LEN(y))$. This is a generalization of our method as well (where rewards are 0/1). Whether this is the right method for problems without binary rewards, is outside the scope of our current study.

3. Reward hacking

“for some easy problems, generating some irrelevant phrases or not generating any thoughts at all can already achieve correct answers.”

We thank the reviewer for raising this important concern regarding reward hacking. In our setup, the reward function is grounded in correctness (via ground-truth verification) and penalizes excessive length through a controllable factor. As a result, the only way to achieve a high reward is by producing responses that are both correct and concise. Generating irrelevant or off-topic content would consume tokens without improving the correctness signal, and is thus actively discouraged by the reward structure.

Importantly, if a problem is simple enough that the model can provide a correct answer without any chain-of-thought reasoning, this is considered optimal under our objective—both in terms of efficiency and correctness. Nonetheless, we appreciate the reviewer’s suggestion and will clarify this point in the paper.

4. Guiding towards optimal reasoning length

This is an excellent suggestion and there has been recent work which is related to this idea [2]. The key insight they have is that length penalties should only be applied when the pass rate for a prompt during online RL is high. This allows the model to compress its CoT only when it is highly confident.

References:

[1] O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning, arXiv 2025

[2] Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning, arXiv 2025

We thank the reviewer for their thorough review. We address their main concerns below:

1. Theory-practice gap

We thank the reviewer for highlighting this important observation. Indeed, there exists a gap between theory and empirical results, which arises because our theoretical analysis in Section 4.2 is intentionally simplified. It primarily serves as a sanity check, demonstrating the method’s soundness under idealized assumptions without confounding factors like model capacity and training dynamics. We agree that understanding and explicitly addressing this theory-practice gap is valuable. In the revision, we will include a more detailed discussion to clearly outline and analyze how practical factors, such as model capacity, training dynamics, and optimization nuances, contribute to the observed discrepancies between theoretical predictions and empirical outcomes.

2. Learning rate experiment for the 1.5B model

We agree with the reviewer that this ablation would be highly informative. However, despite its simplicity, we were unable to conduct it within the limited timeframe of the rebuttal phase due to resource constraints on our shared infrastructure. Note that each 1.5B model training takes about 80 GPU hours. Doing this for 3 seeds and 5 values of $\alpha$ would be around 1200 GPU hours. We appreciate the suggestion and will make a strong effort to include this experiment in the final version of the manuscript.

3. Steepness of the tradeoff curve and its relation to other hyperparameters

We thank the reviewer for the observation. Factors such as the batch size, training dataset, learning rate, and others generally affect the learning dynamics of a reinforcement learning training setup. We expect that similar consideration to apply in this case: although $\alpha$ is the primary control knob, other hyperparameters may affect the learning dynamics. Unfortunately we are unable to conduct meaningful hyperparameter sweeps due to the short rebuttal time and the lab’s finite compute resources.

4. What is being shortened?

Thanks for pointing this out, we will add a section on how the model learns to shorten its CoT.

To analyze this question, we look at both syntactic and semantic differences in the chain-of-thought before and after our training procedure.

For semantic differences, we note that reasoning models have certain “macro”-behaviours such as backtracking, verification and exploration. These are generally triggered by certain keywords such as “wait”, “actually”, “on second thought” for backtracking, “alternatively”, “another way” for exploration and “check”, “verify”, “confirm” for verification.

We compute the average number of times the 7B model performs such behaviours on the MATH500 test set and report the results in the Table below.

These numbers explain the changes in the model behaviour at a macro-level. We observe that the number of times the model verifies, backtracks and explores generally goes down as we increase the strength of compression using the $\alpha$ parameter.

On a syntactic level, we observe that increasing the strength of regularization using $\alpha$ leads to shorter chain-of-thoughts where the model is generally less verbose. For instance, here is an example CoT for a prompt for the $\alpha=0.2$ and $\alpha=0.4$ models:

Prompt:

Kathy needs money for her trip to Europe. If she has 300 US dollars in the bank but wants to withdraw half of it in British pounds and half of it in euros, how many more euros than pounds will she have? Assume 1 pound = 1.64 USD and 1 euro = 1.32 USD, and round to the nearest whole number.

Response ($\alpha=0.2$)

<think>
First, Kathy has $300 USD. She wants to withdraw half in British pounds and half in euros, so each amount will be $150.

To convert $150 to pounds, I'll divide 150 by the exchange rate of 1.64. This gives approximately 91.46 pounds.

For the euros, I'll divide $150 by the euro exchange rate of 1.32, resulting in approximately 113.64 euros.

Finally, subtracting the pounds from the euros gives about 22.18, which rounds to 22 euros.
</think>

Kathy has \$300 USD. She wants to withdraw half in pounds and half in euros:


$

\text{Amount in pounds} = \frac{150}{1.64} \approx 91.46 \, \text{GBP}

$


$

\text{Amount in euros} = \frac{150}{1.32} \approx 113.64 \, \text{EUR}

$


The difference is:


$

113.64 - 91.46 = 22.18 \approx 22

$


Kathy will have $\boxed{22}$ more euros than pounds.

Response ($\alpha=0.4$):

<think>
First, calculate the amounts in USD to be converted to pounds and euros.

Next, convert each amount to the respective currency.

Finally, find the difference between the euro and pound amounts.
</think>

Kathy withdraws \$150 in pounds and \$150 in euros.

Pounds: 150 / 1.64 ≈ 91.46

Euros: 150 / 1.32 ≈ 113.64

Difference: 113.64 - 91.46 ≈ 22.18

Rounded to the nearest whole number: \boxed{22}

Qualitatively, these responses are the same but the response for $\alpha=0.2$ is more verbose than the response for $\alpha=0.4$ due to unnecessary LaTeX tokens and “filler” tokens (for instance, compare “This gives approximately 91.46 pounds.“ with “ ≈ 91.46”) which don’t contribute semantically and could be removed.

5. Naive online RL baseline

We are unsure what the reviewer means by online baseline. If the reviewer means running the method with $\alpha$ is set to 0 (e.g, policy gradient with no length control), this experiment is part of the result that we present. When $\alpha$ is 0, the reward function is just correctness and is not affected by response length.

We thank the reviewer for their thoughtful review. We’re glad that they found our technique simple and effective. We address their main concerns below:

1. Questions on impact

“There is currently still a tradeoff between the length and performance. This makes the work less influential unless the user has a budget constraint on the number of tokens. Moreover, there is a burden to train different reasoning models with different alpha values.”

We appreciate the reviewer pointing out the existing tradeoff. Certain practical scenarios, such as deploying LLMs on resource-constrained devices (e.g., mobile platforms), impose strict limitations on memory and latency. Therefore, it becomes crucial to explore approaches that enable reasoning models to operate effectively under these conditions.

Similar trade-offs already exist in related methods—for instance, knowledge distillation creates smaller, computationally efficient models at the expense of some accuracy. Our approach specifically targets reasoning models, aiming to reduce inference-time computation by encouraging shorter yet effective reasoning chains. This directly addresses deployment concerns like inference cost reduction and it improves user experience through lower latency.

We appreciate the reviewer highlighting his concern regarding the burden to train different models with different values of $\alpha$. It's worth noting that post-training models to reason efficiently requires relatively limited computational resources, typically around 100 gradient steps, compared to the extensive resources needed to initially train large reasoning models. Specifically, our approach uses at least two orders of magnitude less computational effort compared to the original training phase (for reference, see Section 3.2 in [1]).

Moreover, one could potentially generate a family of reasoning models exhibiting varying levels of token usage and efficiency, for instance, OpenAI’s offerings with different ‘tiers’ of thinking effort such as o4-mini and o4-mini-high.

2. Questions on significance

We acknowledge that our method is simple, but we view this as strength as it makes the method broadly applicable. As we are entering an era where models think longer and longer (e.g, multiple hours for the recent IMO results [2]), being efficient is an increasingly urgent need for a system’s success and adoption. We believe that our method serves as a strong starting point towards these goals.

3. Minor details

Thanks for reading our paper so carefully, we will definitely improve on these points in the final version of the manuscript.

4. Non-mathematical reasoning

We are unsure of what the reviewer means by other settings. Is it for the training dataset or evaluation datasets?

If it is for the evaluation datasets, we wish to point out that we already evaluate non-mathematical benchmarks such as Logical Deduction and CommonSenseQA to assess the method’s generalization and we see similar trends as we see on the mathematical benchmarks. Extended results for these tasks are present in Section C of the Appendix.

If the reviewer meant, how would you train such a model without mathematical reasoning datasets: this is a great point. Extending this training procedure to domains without binary 0/1 rewards is challenging and is a question for further research. We will add this to the Limitations of the paper.

References:

[1] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models, arXiv 2025

[2] Luong and Lockhart, Advanced version of Gemini with Deep Think officially achieves gold-medal standard at the International Mathematical Olympiad

We thank the reviewer for their comments. We appreciate the fact that they thought our paper was well written and that our evaluations were sufficient and the algorithm flexible.

Below, we address the reviewer’s major concerns:

1. Abstract is not concise enough

We thank the reviewer for pointing this out. We agree that the abstract can be made more concise. We will address this in our final version of the manuscript.

2. Reservations about the motivation

“I have some reservations about the motivation. It primarily encourages the model to produce correct answers with short chains-of-thought. While this can effectively improve inference efficiency (particularly speed), it inevitably comes at the cost of accuracy. From my perspective, longer reasoning chains also bring benefits during training, as they promote better interpretability and richer contextual understanding.”

Thank you for this valuable feedback. We agree that longer reasoning chains can indeed enhance interpretability and contextual understanding during training. However, certain practical scenarios, such as deploying LLMs on resource-constrained devices (e.g., mobile platforms), impose strict limitations on memory and latency. Therefore, it becomes crucial to explore approaches that enable reasoning models to operate effectively under these conditions. Similar trade-offs already exist in related methods—for instance, knowledge distillation creates smaller, computationally efficient models at the expense of some accuracy. Our approach specifically targets reasoning models, aiming to reduce inference-time computation by encouraging shorter yet effective reasoning chains. This directly addresses deployment concerns like inference cost reduction and it improves user experience through lower latency. We appreciate the reviewer's suggestion and will expand the motivation section of our paper to thoroughly clarify the balance between efficiency gains and the potential accuracy trade-offs, along with concrete use-case examples.

3. High compute requirements

We appreciate the reviewer's concern and acknowledge that our method, employing reinforcement learning (RL), can demand substantial GPU/TPU resources. We have explicitly mentioned this limitation in the paper [Line 322]. However, alternative methods addressing the same problem, such as supervised fine-tuning (SFT) or Direct Preference Optimization (DPO), do not achieve performance comparable to online RL, as demonstrated in Figure 1. This outcome aligns with the established understanding that on-policy, online RL tailored to the specific objective generally yields superior performance compared to offline methods.

Moreover, it's worth noting that post-training models to reason efficiently requires relatively limited computational resources, typically around 100 gradient steps, compared to the extensive resources needed to initially train large reasoning models. Specifically, our approach uses at least two orders of magnitude less computational effort compared to the original training phase (for reference, see Section 3.2 in [1]).

4. Questions regarding theoretical assumptions

We appreciate the reviewer's valuable observation regarding the strength of our theoretical assumptions and the clarity of certain theoretical results. Indeed, our theoretical framework is intentionally simplified, focusing not on generality but rather on clearly illustrating the fundamental mechanisms and intuition behind our algorithm in an accessible manner. We discuss some of the concerns below. Nonetheless, we recognize the reviewer's feedback as an opportunity to enhance our presentation. Accordingly, we will clarify the theoretical sections in line with the reviewer’s suggestions.

   

Below, we address the reviewers detailed questions:

1. “How does this method address the need to perform multiple full rollouts per update, followed by PPO optimization (as illustrated in Figure 2 and described in Section 4.1), which leads to significant GPU/TPU time and memory demands?"

We thank the reviewer for asking the question. Our method builds on top of standard RL libraries, e.g., OpenRLHF, VeRL, and it involves only a minor change to the reward function. In this way, we can inherit all advanced system-level optimization of existing RL codebases.

2. “the authors assume that $f$ is fixed and constant, yet they fail to account for its implicit dependence on $\theta$“

We thank the reviewer for highlighting this point. We’ve also mentioned this in Lines 216-218 in Section 4.2. The discrepancy arises as we aim to present the core result in the simplest possible setting; as the reviewer noticed, the result strictly holds for any $f$ which is a pure function of the length independent of $\theta$.

3. “assumption 4.1 requires that there exists some capable of exactly realizing any arbitrary distribution.”

Thanks for mentioning this detail. We will add references for this assumption.

The assumption that a transformer model can represent any probability distribution over a finite set of strings can be grounded using 3 simple facts:

a. Every distribution over a finite set of strings $S$ can be converted into a Probabilistic Finite State Automata (PFSA)

Intuitive explanation:

Assume the vocabulary is called $\Sigma$. For every prefix string $p$ in the set $S$, compute conditional probabilities for each token $t \in \Sigma$ by calculating

$Prob(t | p) = \frac{\sum Prob(\text{strings starting with prefix pt})}{\sum Prob(\text{string starting with prefix p})}$

Once we construct the conditional probabilities table for all possible pairs of prefixes and tokens, we can convert this into a Probabilistic Finite State Automata.

b. Every PFSA is a Weighted Finite Automata (WFA): This follows from the definition of a WFA.

c. Transformers can simulate WFAs: This result has been shown in [2] by Rizvi et al.

References:

[1] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models, arXiv 2025

[2] Simulating Weighted Automata over Sequences and Trees with Transformers, AISTATS 2024