ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning

We thank the reviewer for their constructive feedback. The questions raised about controlled comparisons, ablation clarity, model generalization, and quality of shortened outputs are crucial. We have conducted new experiments to address these points directly.

On the Comparison with Baselines

We agree with the reviewer that isolating the key factors for improvement is essential. To create a more controlled comparison and demonstrate the specific impact of our Sample Optimal Length (SOL) reward design, we conducted a new experiment.

We implemented the reward function from a key baseline, Training Efficient, directly within our own training framework. This ensures all other variables are held constant:

• Base Model: DeepSeek-Distill-Qwen-1.5B
• Training Data: The same dataset used for ShorterBetter
• Training Algorithm: GRPO

The Training Efficient reward is formulated as $\mathbb{E}\left[\mathbb{1}_{y = y^*} \cdot (1 - \alpha f(\text{LEN}(y)))\right]$, where $f(\cdot)$ is a sigmoid function over the normalized length. We used $\alpha=1$ for the most aggressive length reduction setting.

Controlled Comparison on Math Benchmarks (1.5B Models)

The results show that under identical training conditions, our SOL-based reward function outperforms the baseline's formulation in both accuracy and length reduction across nearly all benchmarks. This provides strong evidence that our reward design is a key driver of the observed improvements.

Of course, we acknowledge that a full "apple-to-apple" comparison with Training Efficient is unfortunately infeasible within the rebuttal window, because our settings differ in data, RL algorithm, reward design, and compute requirements. Here, we summarize the key distinctions:

In the paper, we compared our trained model with their released checkpoints. Training Efficient shows weaker length reduction, which we believe is due to its conservative reward design: the length signal is normalized and passed through a sigmoid, so long but correct answers are still rewarded, just slightly less. They did not provide the model with a clear signal during RL.

However, ShorterBetter defines an explicit pivot (SOL) and lets it guide the model to quickly identify the desired output length. For example:

• Suppose a prompt yields 8 responses, and 2 are correct.
• One correct answer is length 100, another is length 1000.
• Training Efficient: Both are rewarded positively, with the longer one slightly less.
• ShorterBetter: The 1000-token answer is penalized, because the 100-token answer proves the task can be solved concisely.

This design allows the model to quickly converge toward short, correct answers and achieve higher efficiency in length reduction without sacrificing accuracy.

On Clearer Ablation of the Reward Function

We agree that a clear ablation on the definition of $l^{SOL}$ is essential to prove our central claim. As the reviewer suggested, we tested variants that remove or alter the correctness condition. Our ablation studies confirm that naively targeting the shortest length is not viable and that the specific design of SOL is critical.

1. No Correctness Reward, Shortest Length Target: As shown in Section 5.3, if the reward only minimizes length by targeting the shortest response in the group, the model quickly learns to produce empty outputs, causing accuracy to collapse to zero.
2. "Partial" Shortest Length Target: Using the shortest length as a target only when no correct answers are found also fails. This creates a feedback loop where the model sacrifices correctness for length, leading to training collapse.
3. Correctness Reward + Always Shortest Length Target: In a new ablation, we kept the correctness reward ($\alpha=2$) but always set the length target to the shortest response in a group. This also crashed the training, with logs showing accuracy and length rapidly declining to zero. Even with a reward for correctness, a single short (but incorrect) response heavily penalizes all longer (and potentially correct) responses, creating a fatal bias toward underthinking.

These experiments underscore that the length target design is paramount. Our proposed Sample Optimal Length (SOL), which is robustly anchored to the shortest correct response and defaults to the average length in failure cases, provides the necessary stability to avoid these collapse scenarios.

On Experiments with Other Models

To address the valid concern about model architecture bias, we evaluated ShorterBetter on a model with a different architectural backbone: deepseek-ai/DeepSeek-R1-Distill-Llama-8B. This allows us to test our method's generalization beyond the Qwen series. We trained this 8B Llama-based model for 100 steps using our framework.

The results show that ShorterBetter successfully generalizes to a different model architecture. Here we choose hyperparameter alpha=5 since it achieves a better balance between accuracy and length reduction.

Regarding the "aha-moment" outputs, we view this as a characteristic of certain reasoning trace cases generated during inference, rather than a fundamental property of the DeepSeek distilled model series that would uniquely bias our method's effectiveness. The results on the Llama model support the conclusion that our method is broadly applicable.

On Quality of Shortened Outputs

We thank the reviewer for highlighting the importance of human-involved analysis regarding the quality and interpretability of shortened reasoning traces. To directly address this, we conducted a detailed manual analysis of ShorterBetter-7B’s reasoning traces for all 46 problems from the AMC dataset. Our analysis yields two primary insights:

1. Coarse-level human agreement: As detailed in our response to Reviewer 3dRz, inter-annotator agreement between humans and our LLM judge is high (sentence-level category accuracy: 94.23%). This supports the reliability of our reasoning structure analysis framework.

2. Preservation of coherence and readability: Through careful inspection, we confirmed that the coherence of reasoning traces remains largely intact. Specifically, consider the first problem in the AMC dataset:

$\frac{m}{n}$ is the irreducible fraction value of $3+\frac{1}{3+\frac{1}{3+\frac{1}{3}}}$. What is the value of $m+n$?

Both baseline and ShorterBetter-7B models answered correctly, but ShorterBetter reduced reasoning length from 1320 to 654 tokens. Human inspection revealed some key insights:

• Removal of non-substantive statements: Unnecessary statements (e.g., "I think that's solid" or "each step seems to check out") were effectively eliminated.
• Reduction in redundant verification: The baseline explicitly and verbatim repeated calculations. ShorterBetter instead employed more concise verification methods, utilizing assumptions inherent in the problem.
• More concise pivotal reasoning: ShorterBetter replaced overly verbalized calculations (e.g., "So now, adding $\frac{9}{3}$ + $\frac{1}{3}$ gives me $\frac{10}{3}$") with succinct mathematical expressions, preserving clarity and coherence.

Overall, we observe that ShorterBetter’s RL fine-tuning shrinks reasoning trace length without compromising readability and interpretability. In fact, its reductions are very reasonable and human understandable.

We acknowledge that our current evaluation primarily addresses rigorously structured domains (e.g., mathematics & programming), where criteria such as "politeness" or "helpfulness" are less pertinent. Extending our framework to open-ended settings may introduce new challenges—such as potential reward hacking—where reducing trace length might negatively impact such human-interaction qualities. We plan to investigate these issues in future work.

We thank the reviewer for highlighting the ambiguity in our interpretation of SOL and in the choice of hyperparameters. In response to your questions, our brief replies are as follows:

1. Our method naturally handles both long responses and avoids harmful reasoning shortcuts by adaptively balancing correctness and length.
2. The main trade-off lies in tuning $\alpha$ and rollout size $n$, which balance accuracy and output length.

Detailed analysis and supporting experiments are provided below.

On the concerns on SOL design

1. SOL might be too long

• A long SOL typically arises only on genuinely difficult problems.
• In such cases the shortest correct answer must necessarily be long, so an aggressive length penalty would be inappropriate.
• Our reward function therefore behaves conservatively when SOL is large; it can even penalise outputs that are too short.
• This behaviour is desirable: the model learns to respect problem difficulty and produce longer reasoning chains when required.
• Empirically, Fig. 1 and Table 1 confirm this dynamic balance: after training, outputs on harder benchmarks such as AIME and LiveCodeBench remain noticeably longer than on easier sets, even though overall lengths are still reduced.

2. SOL might be too short, causing reasoning shortcuts

We manually inspected ~50 training examples to test whether the model was merely “guessing” the correct answer with an incorrect process (you can also refer to the last part of our reply to reviewer 5QGT). Such cases were rare:

• All training questions are not multiple-choice, so random guessing is extremely unlikely.
• For datasets like AIME (answer ∈ [1, 100]) a small chance of lucky guesses exists, but its impact on training is negligible.
• Moreover, the α coefficient in the correctness-reward term dampens accuracy oscillations: if an overly short SOL starts to harm reasoning quality, the α-weighted reward for correctness pulls training back on track.
• In later training stages—once lengths have already been markedly reduced—the relative weight of α in the reward becomes larger, effectively stabilising any residual accuracy fluctuations.
• However, we did observe that very large rollout sizes (n) occasionally produce over-aggressive SOLs, leading to accuracy oscillations. Hence, we should choose a moderate n (see detailed analysis below) to avoid this issue.

On the Practical Choice of Hyperparameters

On the choice of $\alpha$ and $\beta$

$\alpha$ and $\beta$ are selected as a pair. Since reinforcement learning frameworks such as GRPO use relative rewards rather than absolute rewards, we fix $\beta = 0.001$ and focus on analyzing the choice of $\alpha$.

The hyperparameter $\alpha$ controls the trade-off between accuracy and efficiency. Its value relative to the length penalty $\beta$ (kept at 0.001) is the key factor. We tested new $\alpha$ values to illustrate its impact.

Performance of ShorterBetter-1.5B with Varying $\alpha$

We present new results for $\alpha=0.1$ and $\alpha=5.0$ to complement the results for $\alpha \in \{0, 1, 2\}$ in the main paper.

For alpha = 0.1, the training log shows that accuracy follows a declining trend over 200 training steps. During this time, the median output length decreases rapidly. For alpha = 5.0, the training log indicates a clear upward trend in accuracy throughout the training process. The median output length also trends downward, decreasing from 5000 to below 2000 tokens.

Our findings provide an intuition for selecting $\alpha$:

• A high $\alpha$ (e.g., 5.0) places a strong emphasis on correctness. As seen in our training dynamics log and the table above, this leads to stable accuracy gains but a weaker length-reduction, resulting in longer outputs compared to the more balanced setting. The accuracy trend is clearly positive during training.
• A low $\alpha$ (e.g., 0.1) makes the length penalty term negatively influential. This creates a risk of degrading the model's reasoning capability, as the reward for correctness may not be sufficient to maintain performance. The training dynamics log shows a downward trend in accuracy. This instability can lead to worse length reduction, as seen in the table, because the model fails to learn effective, concise reasoning paths.
• A balanced $\alpha$ (e.g., 1.0 or 2.0) provides a strong enough correctness signal to preserve or improve accuracy while still exerting significant pressure to reduce output length. This achieves the best overall trade-off demonstrated in our paper.

In practice, the optimal choice depends on the specific goal (e.g., prioritizing efficiency vs. accuracy) and specific model family. Based on our results, we recommend practitioners start with $\alpha$ in the range of [2, 5] and tune as needed.

On the choice of rollout size $n$

Here, we clarify how the rollout size n affects our GRPO algorithm, particularly when n = 4, 8, and 12 on the 1.5 B model. (Results for n = 4 and n = 12 are new and were not included in the original paper.)

Computational constraints

• A larger n increases GPU memory and wall-time requirements, especially in early training when outputs are still long.
• With large n the agent may exhaust memory and trigger OOM errors; therefore we restrict experiments to modest n values.
• In practice we were able to run n = 4, 8, and 12 reliably; higher values were infeasible on our hardware.

Training dynamics

All training-dynamics metrics are tested on questions unseen by the model, thereby faithfully reflecting the generalisation performance and output efficiency.

• Key observation: Increasing n enlarges the candidate pool, so the chosen SOL target becomes more aggressive. This markedly accelerates length reduction but also amplifies accuracy variance.
• Practical recommendation: n = 8 offers the best balance between rapid convergence and accuracy stability and, critically, fits within the resource budget for larger models such as 7 B parameters.

On the choice of rollout size $n$

We appreciate the reviewer for highlighting the ambiguity surrounding our choice of rollout size n. Here, we clarify how the rollout size n affects our GRPO algorithm, particularly when n = 4, 8, and 12 on the 1.5 B model. (Results for n = 4 and n = 12 are new and were not included in the original paper.)

Computational constraints

• A larger n increases GPU memory and wall-time requirements, especially in early training when outputs are still long.
• With large n the agent may exhaust memory and trigger OOM errors; therefore we restrict experiments to modest n values.
• In practice we were able to run n = 4, 8, and 12 reliably; higher values were infeasible on our hardware.

Training dynamics

Due to limited time and computational budget, we were unable to run a full evaluation suite for the rollout size variants (n = 4, 8, 12). Nonetheless, the training-dynamics already capture how each choice of n affects convergence speed, length reduction, and accuracy stability, thereby providing the key evidence needed to understand their impact. All training-dynamics metrics are tested on questions unseen by the model, thereby faithfully reflecting the generalisation performance and output efficiency.

Notes: Due to NeurIPS 2025 formatting constraints, we are unable to embed the training-dynamics plots in this rebuttal.

• Key observation: Increasing n enlarges the candidate pool, so the chosen SOL target becomes more aggressive. This markedly accelerates length reduction but also amplifies accuracy variance.
• Practical recommendation: n = 8 offers the best balance between rapid convergence and accuracy stability and, critically, fits within the resource budget for larger models such as 7 B parameters.

On the validation of automated judge:

We thank the reviewer for highlighting the importance of validating our automated LLM judge. To address this, we conducted a comprehensive human-annotation study on the AMC dataset (46 problems, 2064 sentences, 65,329 tokens) using ShorterBetter-7B’s reasoning traces. Key inter-annotator validation results are as follows:

• Sentence-level accuracy: 94.23% (1945/2064)
• Token-level accuracy: 93.01% (60,765/65,329)

The token-level distribution comparison between the LLM and human judges is summarized below:

Qualitatively, we observe that:

• The inter-annotator agreement is consistently very high across all five categories. Thus, the reliability of our LLM judge’s categorizations is strongly supported.
• Most disagreements occur between the categories "Productive Elaboration & Calculation" and "Pivotal Reasoning"—both regarded positively in our analysis. Consequently, our main conclusion—that ShorterBetter jointly increases the proportion of these meaningful reasoning categories—remains robust despite minor distributional discrepancies.

We selected the AMC dataset as it strikes an ideal balance: it features non-trivial mathematical reasoning while remaining small enough to annotate exhaustively within our rebuttal timeline. For transparency, we plan to release our human-annotation platform and extend validation to additional benchmarks in the near future.

We thank the reviewer for finding our presentation clear and experimental study concrete. We also appreciate the thoughtful feedbacks regarding the novelty and robustness of our method. Below are our point-by-point responses:

On the Novelty of Our Method

The reviewer has pointed out the previous L1 (Aggarwal et al., 2025) and concurrent work ConciseR (Song et al., 2025), and correctly notes that all three methods operate in the space of RL for efficient reasoning. We also believe that comparing with these two methods allows us to clarify and demonstrate the novelty of ShorterBetter. On a high level:

• L1 (Aggarwal et al., 2025) focuses on external controllability, where a user must specify a length budget.
• ConciseR (Song et al., 2025) uses a gated, two-stage process to first improve reasoning and then separately optimize for length.
• ShorterBetter, in contrast, is designed to autonomously discover an optimal reasoning length, based on the model's own capabilities and the problem's difficulty, within a single, unified training process.

Detailed comparison with L1:

The reviewer suggests ShorterBetter can be viewed as a "self-supervised version of L1." While we agree our method is self-supervised, we believe this framing understates the conceptual leap. Notable differences are:

• Objective: L1's objective is adherence to a user-defined budget. The model is rewarded for matching a length ($n_{gold}$) provided in the prompt. ShorterBetter's objective is to find the most efficient path to a correct solution -- which can be viewed as self-supervising the model to approximate a (hidden) optimal length. Crucially, unless the user already knows the "most efficient solution" to a problem a priori, the self-supervised target in our approach will also deviate from $n_{gold}$.
• Assumption: In order for the method to work well, L1 requires the user-defined budget (target) to be informative of the problem at hand, which we argue can be sometimes unrealistic (how do you set a budget for a math question that you cannot solve?). On the other hand, ShorterBetter is built on the more general hypothesis that an Optimal Reasoning Length (OL) exists, which the model learns to find on its own without requiring this prior knowledge.
• Prompt dependence: L1 requires appending an explicit cue—“think for n tokens”—to every training prompt, a design choice that can itself shape the model’s behavior; this interaction is not analyzed in detail in their paper. By contrast, ShorterBetter is completely prompt-agnostic, making it broadly applicable.

Detailed comparison with ConciseR:

ConciseR is an excellent concurrent work, but it addresses a different problem setting with a different methodology. Notable differences are:

• Problem Formulation: We start with a strong but overthinking reasoning model and our goal is to refine its reasoning to be more efficient. ConciseR starts with a weaker base model and must first build up its reasoning capability before optimizing for length. These are fundamentally different starting points and objectives.
• Training Paradigm & Gating: ConciseR uses a sequential two-stage framework. Stage 1 improves reasoning, and Stage 2 enforces conciseness. Crucially, its length reward is conservatively gated; it only activates when all sampled rollouts are correct. In contrast, ShorterBetter uses a single, integrated process. Our $l_{SOL}$ provides a flexible learning signal as long as at least one correct response exists, making our method more sample-efficient, especially on challenging problems where 100% correctness per batch is rare.
• Reward Signal: ConciseR's reward aims for maximum compression (rewarding $L_{Max} - l_i$). ShorterBetter's $l_{SOL}$ reward targets an optimal point, penalizing responses that are too long and too short.

To summarize:

The core novelty of ShorterBetter is the introduction of Sample Optimal Length (SOL) as a dynamic, self-supervised reward signal. This creates an autonomous and integrated framework that guides a model to discover its own optimal reasoning length, rather than simply obeying an external budget (L1) or following a staged compression schedule (ConciseR).

On the Robustness of Our Method

We thank the reviewer for their insightful question on training robustness and hyperparameter sensitivity. We offer this explanation, supplemented by new experiments, to clarify these critical aspects.

On Training Collapse from Naively Minimizing Length

The reviewer correctly notes the training collapse shown in our paper (Fig. 4, Left). This collapse is caused by an unstable reward signal. Our ablation studies confirm that naively targeting the shortest length is not viable:

1. No Correctness Reward, Shortest Length Target: As shown in our paper, if the reward only minimizes length, the model quickly learns to produce empty outputs, causing accuracy to drop to zero.

2. "Partial" Shortest Length Target: Using the shortest length as a target only when no correct answers are found also fails (even with a reward term present). This creates a feedback loop where the model sacrifices correctness for length, leading to a training collapse.

3. Correctness Reward + Always Shortest Length Target: In a new ablation, we kept the correctness reward ($\alpha=2$) but always set the length target to the shortest response in a group. This also crashed the training. Our logs show accuracy and length rapidly declining to zero. Even with a correctness reward, a single short (but incorrect) response heavily penalizes all longer (and potentially correct) responses, creating a fatal bias toward underthinking.

These experiments show that the length target design is critical. Our proposed SOL, anchored to the shortest correct response and defaulting to the average length in failure cases, provides the necessary stability to avoid these collapse scenarios.

On the Practical Choice of Hyperparameters

The hyperparameter $\alpha$ controls the trade-off between accuracy and efficiency. Its value relative to the length penalty $\beta$ (kept at 0.001) is the key factor. We tested new $\alpha$ values to illustrate its impact.

Performance of ShorterBetter-1.5B with Varying $\alpha$

We present new results for $\alpha=0.1$ and $\alpha=5.0$ to complement the results for $\alpha \in \{0, 1, 2\}$ in the main paper.

For alpha = 0.1, the training log shows that accuracy follows a declining trend over 200 training steps. During this time, the median output length decreases rapidly. For alpha = 5.0, the training log indicates a clear upward trend in accuracy throughout the training process. The median output length also trends downward, decreasing from 5000 to below 2000 tokens.

Our findings provide an intuition for selecting $\alpha$:

• A high $\alpha$ (e.g., 5.0) places a strong emphasis on correctness. As seen in our training dynamics log and the table above, this leads to stable accuracy gains but a weaker length-reduction, resulting in longer outputs compared to the more balanced setting. The accuracy trend is clearly positive during training.
• A low $\alpha$ (e.g., 0.1) makes the length penalty term negatively influential. This creates a risk of degrading the model's reasoning capability, as the reward for correctness may not be sufficient to maintain performance. The training dynamics log shows a downward trend in accuracy. This instability can lead to worse length reduction, as seen in the table, because the model fails to learn effective, concise reasoning paths.
• A balanced $\alpha$ (e.g., 1.0 or 2.0) provides a strong enough correctness signal to preserve or improve accuracy while still exerting significant pressure to reduce output length. This achieves the best overall trade-off demonstrated in our paper.

In practice, the optimal choice depends on the specific goal (e.g., prioritizing efficiency vs. accuracy) and specific model family. Based on our results, we recommend practitioners start with $\alpha$ in the range of [2, 5] and tune as needed.

On the "Length After First Appearance" Metric

While we agree that our “Length after First Appearance” metric is correlated with total length,, we believe it provides a unique and valuable perspective (as pointed out by Reviewer 2UEK) on a specific pattern of overthinking: a model's inability to terminate its reasoning process after it has already found a correct solution.

The reviewer correctly notes that redundancy can occur both before the answer is found ("pre-solution") and after ("post-solution"). Our metrics are designed to disentangle these two effects. Total length captures the sum of both inefficiencies. In contrast, our "Length After" metric focuses on the post-solution component. Therefore, we believe that the total length and the "Length After" metrics offer complementary views. We will revise Section 5.2 to make this clear.

Clarification on Sec 6

Finally, our statement at the end of Sec 6 means that we encourage future works in efficient reasoning to also adopt similar reasoning trace analysis, which provides more fine-grained and detailed feedbacks to understand the model behavior than simply looking at overall length reduction.