Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models

[W1]

Reviewer’s Comment: An ablation study to understand the individual contribution of each of these stages would be insightful.

Thank you for raising this valuable point. To better understand the contribution of each stage, we conducted an ablation study that incrementally introduces each component of our framework:

1. DTO (Stage 1): We first add the mechanism that identifies the appropriate point at which reasoning should be finalized.
2. DTO (Stage 1 + 2): Building on top of Stage 1, we incorporate a module that prune intermediate thinking patterns that do not meaningfully contribute to the reasoning objective.

We used DeepSeek-R1-Distill-Qwen-1.5B as the base model for this study, and the results are presented in the table below. As shown, both stages independently yield meaningful improvements. Notably, Stage 1 alone provides substantial gains in reasoning efficiency by enabling early termination once sufficient evidence has been gathered. While the additional gains from Stage 2 are more moderate, we can observe that it still brings meaningful improvements. The full DTO (Stage 1 + 2) consistently delivers the strongest results, outperforming all baselines in terms of efficiency while maintaining or even improving accuracy, as reported in Tables 1 and 2 of the main paper.

We will include the full ablation results and analysis in the revised version.

[W2]

Reviewer’s Comment: Because the evaluation was conducted under an 8192-token limit, there is a possibility that some responses were truncated due to this length constraint.

Thank you for raising this concern regarding the potential truncation of responses. We would like to firstly clarify that all models were evaluated under the same maximum response length of 8192 tokens, ensuring a fair and consistent comparison across methods.

To further examine whether this constraint may have disadvantaged baseline methods, we conducted additional experiments under a higher token limit of 16,384, using DeepSeek-R1-Distill-Qwen-1.5B as the base model. As shown in the tables below, we observe that all methods exhibit a general increase in both accuracy and token usage when granted more space to complete their reasoning. This is expected, as longer context allows models to better develop and finalize their solutions.

However, DTO continues to outperform both the base model and the strongest baseline (FCS + Ref.), even under this more generous constraint. In particular, DTO achieves the highest accuracy while using the fewest tokens, leading to the best efficiency scores overall. For example, on AMC2023, DTO improves accuracy by 8.2 points over the baseline, while reducing token usage by over 50%, resulting in an efficiency gain of 2.43x. Similar patterns are observed on Gaokao and AIME2024.

These findings confirm that DTO’s effectiveness is not merely a byproduct of producing shorter responses under certain token limits, but rather reflects genuine improvements in reasoning quality efficiency, even when models are allowed more room to reason.

[W3]

Reviewer’s Comment: A simple yet potentially strong baseline could involve forcing the model to generate an answer once it reaches a certain token limit.

Thank you for the thoughtful suggestion. We agree that smaller models such as 1.5B or 7B parameters may occasionally struggle to finalize their reasoning and that encouraging termination at a specific token threshold can serve as a useful baseline for evaluating our method.

To assess this idea, we implemented the proposed baseline: for a range of maximum token limits (1024, 2048, 4096, and 6144), we prompted the base model, DeepSeek-R1-Distill-Qwen-1.5B, to produce a final boxed answer when nearing the token limit. To facilitate finalization, we appended the following instruction: "Wait, I suddenly got the final answer to the whole problem.\n\nFinal Answer:".

This prompt-based mechanism was applied to the base model under each token constraint, simulating a forced finalization strategy similar in spirit to the one described in our paper.

As shown in the results, we observe a consistent trend: as the token limit increases, both accuracy and the number of generated tokens increase, indicating that longer sequences enable more complete reasoning. However, we can observe that our method, DTO, achieves higher accuracy while using fewer or comparable tokens, highlighting its strength in both reasoning quality and efficiency.

We will include the full result table in the revised version.

[W4]

Reviewer’s Comment: There is some concern regarding the generalizability of the findings to larger models.

Thank you for raising this important question regarding the generalizability of our method to larger language reasoning models. To address this point, we conducted additional experiments within our available resources to empirically evaluate DTO’s performance in larger-scale settings.

Specifically, we tested our method on a larger model, i.e., DeepSeek-R1-Distill-Qwen-7B, and considered three different settings on this model:

• the base model,
• the strongest baseline (FCS + Ref.), and
• our proposed method (DTO).

These experiments followed the same training and evaluation protocols as outlined in the main paper, except for one change: due to computational constraints, we used 2,500 training instances instead of 5,000. All other experimental settings remained consistent to ensure comparability. Additionally, please note that, due to the same computational constraints, we did not perform hyperparameter tuning and instead used the default configuration from the main paper.

As shown in the extended results table below, DTO continues to achieve superior efficiency metrics even at the 7B scale. It outperforms both the base model and the FCS + Ref. baseline, which is the strongest baseline, across all benchmarks, delivering improvements in both accuracy and token efficiency. These findings suggest that the benefits of DTO generalize beyond 1.5B-sized models and that the method is robust to increased model capacity.

We sincerely appreciate your suggestion, and we believe that these additional results strengthen the case for the applicability of DTO to larger reasoning models. We will incorporate these findings into the revised version of our main paper.

[W1 & Q2]

Reviewer’s Comment: The method seems less efficient in collecting training samples. How is the response segmented into thinking patterns, and how many completions are needed per response to optimize it using DTO?

As described in Section 3 of the main paper, we segment the reasoning trajectory $y$ into distinct thinking patterns by leveraging discourse cues such as “Wait” or “Alternatively”, following prior work [1, 2]. These expressions commonly emerge in large reasoning models' outputs and are known to mark shifts in the model’s reasoning process, including transitions into self-verification, intermediate summarization, or revised hypotheses.

Empirical analysis reveals that a typical reasoning trajectory contains approximately 22 thinking patterns on average. However, not all of these require decoding. For the first stage of DTO - which identifies the point at which the reasoning should be finalized - we evaluate the probability of producing the correct answer from each partial trajectory $\tau_i = \delta_1 \oplus \cdots \oplus \delta_i \oplus \delta_{\text{exit}}$, as defined in Equation (6). We then determine the earliest index $i'$ at which the probability surpasses a predefined confidence threshold. On average, this threshold is reached around the 14th segment, which implies that only the remaining 8 chunks typically require decoding.

Altogether, following the setup in Section 3.3, DTO requires approximately 92 completions per response. Importantly, these completions are lightweight, since the decoding is limited to a small number of tokens - just 16 more than the length of the ground-truth answer- and can be efficiently parallelized using fast inference frameworks [3, 4]. Additionally, while DTO may involve a higher sample collection cost than some baselines, such as FCS + Ref., it is important to note that this procedure is performed offline during training. Once trained, the model yields substantial improvements in inference-time reasoning efficiency, as demonstrated in Tables 1 and 2 of the main paper. Therefore, we believe that the one-time cost of training is justifiable given the long-term benefits in runtime efficiency and scalability.

We will update the manuscript to clarify these aspects more explicitly. Thank you again for your valuable feedback.

[W2 & Q3]

Reviewer’s Comment: Can your proposed method show efficiency in larger LRMs, at least on 7B models?

Thank you for your thoughtful question regarding the scalability of our method to larger language reasoning models. To address this point, we conducted additional experiments within our available resources to empirically evaluate DTO’s performance in larger-scale settings.

Specifically, we tested our method on a larger model, i.e., DeepSeek-R1-Distill-Qwen-7B, and considered three different settings on this model:

• the base model,
• the strongest baseline (FCS + Ref.), and
• our proposed method (DTO).

These experiments followed the same training and evaluation protocols as outlined in the main paper, except for one change: due to computational constraints, we used 2,500 training instances instead of 5,000. All other experimental settings remained consistent to ensure comparability. Additionally, please note that, due to the same computational constraints, we did not perform hyperparameter tuning and instead used the default configuration from the main paper.

As shown in the extended results table below, DTO continues to achieve superior efficiency metrics even at the 7B scale. It outperforms both the base model and the FCS + Ref. baseline, which is the strongest baseline, across all benchmarks, delivering improvements in both accuracy and token efficiency. These findings suggest that the benefits of DTO generalize beyond 1.5B-sized models and that the method is robust to increased model capacity.

We sincerely appreciate your suggestion, and we believe that these additional results strengthen the case for the applicability of DTO to larger reasoning models. We will incorporate these findings into the revised version of our main paper.

[Q1]

Reviewer’s Comment: It is unclear how DTO is applied when the initial response is incorrect.

Thank you for raising this question. Let us clarify the intended interpretation and provide further explanation.

Firstly, the analysis in lines 192-196 is not about optimizing responses that are already correct. Rather, it highlights an important capability of our framework: even when the originally sampled response is incorrect, applying DTO can sometimes recover a correct reasoning trajectory.

This behavior is particularly valuable in challenging settings where most or all sampled responses from the base reasoning model are incorrect. In such cases, prior methods such as FCS + Ref. or DAST are unable to generate preference pairs and must discard the instance entirely, as they rely on having at least one correct response in the sampled pool.

In contrast, DTO enables us to rescue such difficult instances by producing a correct, optimized trajectory from an initially incorrect response. While this recovery is not guaranteed in all cases, when it does succeed, it allows us to include these cases in the pairwise training dataset, potentially offering strong learning signals in hard problem regimes. We believe this capability is one of the strengths of our approach, as reflected in the strong accuracy results reported in Tables 1 and 2 of the main paper, and we will revise the corresponding section to make this point more explicit.

References

[1] Marjanović, Sara Vera, et al. "DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning." arXiv preprint arXiv:2504.07128 (2025).

[2] Lu, Ximing, et al. "Retro-search: Exploring untaken paths for deeper and efficient reasoning." arXiv preprint arXiv:2504.04383 (2025).

[3] Zheng, Lianmin, et al. "Sglang: Efficient execution of structured language model programs." Advances in neural information processing systems 37 (2024): 62557-62583.

[4] Fu, Yichao, et al. "Efficiently Scaling LLM Reasoning with Certaindex." arXiv preprint arXiv:2412.20993 (2024).

Thank you for your response and the additional experiments. I’d like to further clarify Q1:

• For the claim “even when the originally sampled response is incorrect, applying DTO can sometimes recover a correct reasoning trajectory”, I believe this fundamentally relies on the sampling space (not full response resampling, but also process-level sampling as in [1]). Your method uses the gold answer to directly filter for correct responses from this sampling space, so the observed accuracy improvement is an expected outcome rather than a clear benefit of your proposed approach. While this does not diminish your contribution in using DTO to construct preference data, I find the analysis in Section 3.3 insufficiently convincing in isolating the method’s true impact.

[1] MATH-SHEPHERD: VERIFY AND REINFORCE LLMSSTEP-BY-STEP WITHOUT HUMAN ANNOTATIONS. arXiv:2312.08935.

Overall, I think using DTO to construct preference data is a reasonable and promising idea. While the improvements (e.g., saving a few hundred tokens in long responses) are relatively modest, the experiments are solid and the method could inspire future work in this direction. I will maintain my current score.

[W1 & Q2]

Reviewer’s Comment: Please provide the justification for the choice of auxiliary model used in DTO.

We thank the reviewer for this thoughtful and important comment. To clarify, our use of the auxiliary LLM (LLaMA 3.3-70B-Instruct) was intentionally aligned with the setup used in one of our primary baselines, FCS + Ref., which also relies on the same model. By adopting the same auxiliary model, we ensure that the performance gains observed in our method are attributable to the proposed framework itself - rather than differences in model capacity - thereby enabling a fair and controlled comparison.

We agree that the motivation for this design choice should be made more explicit. In the revised version, we will clarify this point to enhance the transparency and fairness of our experimental setup.

[W2]

Reviewer’s Comment: Please explain the rationale behind segmenting reasoning trajectories using discourse cues that signal transitions in thinking, and possible strategies for handling cases where such cues are absent.

Thank you for highlighting this important point. Firstly, we follow prior work [1, 2] in segmenting the reasoning trajectory into multiple thinking patterns using discourse cues such as “Wait” or “Alternatively”, which frequently appear in the outputs of large reasoning models [1, 2]. These cues are widely known to reflect shifts in the model’s reasoning process, including transitions into self-verification, intermediate summarization, or revised hypotheses as described in Section 3.

We acknowledge, however, that this heuristic may not be applicable in all cases, particularly when such cue words are absent. In those situations, a promising extension would be to first analyze the lexical or structural patterns that commonly appear at the beginning of new paragraphs or sentences, and then leverage these patterns to define segment boundaries in a more systematic manner.

While our current approach provides a simple and effective segmentation method grounded in observed discourse patterns, we agree that developing a more general and robust segmentation strategy would be a valuable direction for future work. We will clarify both this limitation and potential extensions in the revised version.

[W3]

Reviewer’s Comment: Compare DTO with a simple in-context learning technique, such as providing few-shot demonstrations or CoTs to Instruct version.

Thank you for the thoughtful comment. While we appreciate the interest in comparing DTO against instruction-tuned variants, we would like to firstly clarify the intended evaluation scope of our work. As described in Section 3.1, our primary goal is to improve the efficiency of the base Large Reasoning Model, e.g., DeepSeek-R1-Distill-Qwen-1.5B, while maintaining or improving its reasoning performance. Accordingly, the efficiency metric defined in Equation (14) is designed to measure gains relative to this shared base model.

In contrast, the Instruct version is not directly derived from our target LRM but is instead based on its backbone model with separate instruction tuning. While we include Instruct ver. for completeness and to offer a broader perspective, it is not the main comparison target for efficiency evaluation, and we therefore do not compute the efficiency metric for them.

To further assess the effectiveness of instruction-tuned models and the feasibility of simple prompting-based enhancements, we conducted additional experiments based on the reviewer’s suggestion. Specifically, using DeepSeek-R1-Distill-Qwen-1.5B as the base model, we evaluated its instruction-tuned version, Qwen2.5-Math-1.5B-Instruct, under two prompting strategies, using 10 examples sampled from the training set: (1) few-shot demonstrations and (2) few-shot CoTs.

The results are presented in the table below. As shown, these prompting strategies do not lead to meaningful improvements in accuracy, and in some cases even degrade performance. This suggests that simply appending demonstrations or CoT examples may be insufficient for eliciting effective reasoning behavior in instruction-tuned models..

We also observe that the Instruct variants tend to perform relatively well on simpler datasets such as GSM8K, where they achieve reasonable accuracy with strong efficiency. However, their performance drops significantly on more challenging benchmarks such as AMC and AIME, suggesting a lack of robustness in handling complex reasoning tasks. These trends imply that while instruction-tuned models can be effective for relatively straightforward tasks, their generalizability may be limited in settings that demand deeper reasoning. Investigating which types of models or prompting strategies best suit different task complexities could be an interesting direction for future work.

We will include these additional results and analyses in the revised version to strengthen the comparison and highlight the advantages of our method.

[Q1]

Reviewer’s Comment: Can DTO handle queries where no golden label is available? I wonder whether DTO is applicable in such settings.

Thank you for this insightful question. As described in the main paper, our current implementation of DTO leverages access to a ground-truth answer to supervise two stages:

1. Identifying the appropriate point at which the reasoning should be finalized, and
2. Pruning intermediate thinking patterns that do not meaningfully contribute to the reasoning objective.

While our method currently operates in a supervised setting, we agree that it is meaningful to consider how our framework could be applied in weakly supervised or unsupervised settings. Importantly, DTO can be naturally extended to scenarios where ground-truth answers are unavailable, by utilizing a reward model as a surrogate for correctness. Below, we outline how each stage of DTO can be adapted accordingly.

1. Identifying the Finalization Point via Reward Comparison

In the absence of a golden answer, we can replace the answer-likelihood signal, which is defined in Equation (6) of the main paper, with a scalar reward function $R(\cdot)$, obtained via a reward model. For each truncated trajectory $\tau_i$ (as defined in Equation (4)), we compute its reward: $R(\tau_i)$. We also compute the reward for the full reasoning trajectory $y$, denoted $R(y)$. We then identify the earliest index $i'$ such that: $|R(\tau_i) - R(y)| \leq \epsilon$,

where $\epsilon$ is a small threshold. This condition ensures that the partial trajectory is nearly as valuable (in terms of reward) as the full trajectory.

We then truncate the reasoning at index $i'$ using the same binary selection function $f(\cdot)$ from Equation (7): $f(\delta_i) = 1$ if $i \leq i'$, $0$ otherwise. This enables DTO to finalize reasoning at an appropriate point based on reward saturation, rather than answer correctness.

2. Pruning Redundant Chunks via Reward Sensitivity

To adapt the pruning stage, we redefine the binary pruning function $g(\delta_i)$ using the reward sensitivity of each chunk: $g(\delta_i) = 1$ if $|R(\Delta^f_x \setminus \{\delta_i\}) - R(\Delta^f_x)| \leq \epsilon$, $0$ otherwise.

That is, we remove a chunk $\delta_i$ only if its removal does not significantly degrade the reward. This allows us to safely discard segments that do not meaningfully contribute to high-reward outputs, while preserving critical content.

Before applying this function, we may optionally employ the same auxiliary model used in the supervised setup to narrow down candidate chunks, identifying those likely to be redundant and reducing the risk of over-pruning.

This reward-based extension maintains DTO’s core motivation while removing its reliance on ground-truth answers. It broadens the method’s applicability to weakly supervised or unsupervised settings, where annotated labels are unavailable.

We appreciate the reviewer’s suggestion and will include a discussion of this extension in the revised version, as we believe it represents a promising direction for future work.

References

[1] Marjanović, Sara Vera, et al. "DeepSeek-R1 Thoughtology: Let's think about LLM Reasoning." arXiv preprint arXiv:2504.07128 (2025).

[2] Lu, Ximing, et al. "Retro-search: Exploring untaken paths for deeper and efficient reasoning." arXiv preprint arXiv:2504.04383 (2025).

[W1 & Q1]

Reviewer’s Comment: Stage 2 of DTO might have a risk of over-pruning.

We appreciate the concern regarding the risk of over-pruning. As described in Section 3, our method, DTO, consists of two stages:

1. Identifying the appropriate point at which the reasoning should be finalized, and
2. Pruning intermediate thinking patterns that do not meaningfully contribute to the reasoning objective.

Stage 2, in particular, is carefully designed to prevent harmful pruning through a conservative, multi-step verification process. To begin, we assess the utility of each intermediate reasoning chunk using an auxiliary LLM. To avoid inadvertently removing valuable reasoning steps, we guide this LLM using explicitly conservative filtering criteria. As detailed in Appendix A.6, the prompt includes the original problem, the complete reasoning trajectory, the correct final answer, and the following instructions:

• Retain any chunk that contributes meaningfully, including narrowing down possibilities, offering partial insight (even if incomplete), performing sub-computations, supporting downstream reasoning, or even reinforcing understanding through redundancy.
• Remove only those chunks that are clearly off-topic, factually incorrect in a harmful way, or provide no meaningful content - with the default bias toward retaining unless strong justification for removal is found.

To further verify the safety of each removal, we perform a lightweight decoding-based validation. For each candidate chunk $\delta_i$ - previously identified by the auxiliary model as a removable thinking pattern - we define the following sequence: $\tilde{\Delta}_x^{f\setminus \delta_i} = \left[\ \delta_j \in \tilde{\Delta}_x^f \middle| \delta_j \ne \delta_i,\ \text{and before ``}\backslash\text{boxed''} \ \right]$

We then quickly decode from this sequence to check whether the model can still produce the correct answer. The pruning of $\delta_i$ is accepted only if the answer remains correct; otherwise, the chunk is retained.

As shown in Tables 1 and 2 of the main paper, this careful procedure enables DTO to consistently outperform baselines in terms of reasoning efficiency, achieving similar or better accuracy while generating fewer tokens.

Additionally, motivated by your comment, we conducted an ablation study to assess the isolated impact of Stage 2, using DeepSeek-R1-Distill-Qwen-1.5B as the base model. Below, we present the results comparing DTO with only Stage 1 to the full version with both stages applied:

These results show that adding Stage 2 on top of Stage 1 has minimal impact on accuracy, while improving efficiency. This confirms that our pruning procedure removes redundant or distracting reasoning without harming the underlying logic.

We appreciate the reviewer’s thoughtful comment again, which prompted us to conduct this additional analysis. We will include these results and clarifications in the revised version to more clearly communicate the safety and effectiveness of our pruning strategy.

Dear Reviewer,

We sincerely appreciate your thoughtful feedback and the time you've devoted to our paper. As the discussion phase draws to a close, we would like to check if our response has sufficiently addressed your comments. If there are any remaining concerns or further suggestions you'd like us to consider, we would be grateful to hear them.

Your insights have been extremely helpful in improving our work, and we're committed to refining the paper as best we can.

Thank you again for your time and contributions.

Best,

Authors