Ada-R1: Hybrid-CoT via Bi-Level Adaptive Reasoning Optimization

We thank the reviewer for these insightful observations. We address each concern below:

weakness 1: While the experiments focus on mathematical reasoning (GSM8K, MATH, etc.), the paper does not address how Ada-R1 performs on other task types

Our primary experiments focus on mathematical reasoning (e.g., GSM8K, MATH, AIME), and we also evaluate OOD dataset Olympiad and Minerva（In paper Table2），also on the LogicQA, GPQA, MMLU. Results show that Ada-R1 continues to exhibit strong efficiency-accuracy trade-offs beyond purely mathematical tasks, supporting its generalizability to other reasoning domains. We plan to further explore other tasks in future work. (prompt is Let’s think step by step and output the final answer within \boxed{}. (e.g., \boxed{A}) with zero shot)

---

weakness 2: The paper’s reliance on linear merging might overlook more sophisticated interactions between the two reasoning styles

We acknowledge that our current merging strategy adopts a simple linear interpolation of Long- and Short-CoT models, which may not capture all possible interactions. However, this simplicity is intentional for two reasons: (1) The goal of Stage I is not to coordinate the two reasoning modes but merely to enable the hybrid model to represent both styles Coordination and preference learning are then handled effectively in Stage II (bi-level preference training) (2) The linear merging provides a transparent and interpretable baseline, making it easier to isolate and analyze the effectiveness of group- and instance-level preferences, as demonstrated in our ablation studies.

---

weakness 3: The paper does not report error bars, confidence intervals, or statistical tests (e.g., t-tests) to validate the significance of performance differences between Ada-R1 and baselines.

In line with common practices in this field (e.g., [1][2][3][4][5][6][7][8]), we report point estimates on standard benchmarks following the evaluation protocols established in prior work. For smaller test sets such as AIME25, we run 4 random sampling rounds and report the average to mitigate variance, we will explicitly clarify this detail in the revised version.

---

weakness 4: The Short-CoT models are fine-tuned from Long-CoT models using a small dataset (2,000 samples), which might not fully capture the diversity of short reasoning patterns.

We trained a short model because Deepseek uses a different special token vocabulary for distillation, making it impossible to directly merge with these short cot models. Furthermore, we tested the existing short cot model (Qwen2.5-7B-Instruct), and found that it performed worse than the model we trained. Based on these considerations, we chose to fine-tune the new short cot model.

---

References

[1] CoThink: Token-Efficient Reasoning via Instruct Models Guiding Reasoning Models

[2] L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning

[3] Scalable Chain of Thoughts via Elastic Reasoning

[4] Thinking Fast and Right: Balancing Accuracy and Reasoning Length with Adaptive Rewards

[5] Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models

[6] ThinkSwitcher: When to Think Hard, When to Think Fast

[7] Let LLMs Break Free from Overthinking via Self-Braking Tuning

[8] REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once

---

question 1

See our response to weakness 1

---

question 2

See our response to weakness 3

---

question 3

The adaptive selection in Ada-R1 does not rely on an external module or classifier to decide between Long-CoT and Short-CoT. Instead, the selection is implicitly learned and performed within the model’s standard autoregressive decoding. Thus, there is no additional latency or computational overhead during inference. Moreover, by significantly reducing the number of generated tokens (e.g., over 50% reduction on average in Table 2), Ada-R1 naturally improves real-time inference performance.

---

question 4

We fully agree with your suggestion and will add this citation to our paper in future revisions

---

limitations

Thank you for this valuable feedback. We agree that the manuscript should explicitly clarify the limitations of our approach, such as the challenges it may encounter when extended beyond mathematical domains. We will add a detailed discussion in the revised version to further enhance the paper’s completeness and transparency.

We thank the reviewer for the time and effort spent reviewing our work. We greatly value the feedback and have done our best to address each point thoroughly in the responses below.

---

weakness 1

Thank you for this valuable suggestion. To demonstrate our model’s out-of-distribution adaptability, we applied the version trained on mathematical reasoning to two classic OOD benchmarks: MMLU and GPQA. Below are the results:

These results show that our model maintains strong performance, and shorter generation lengths—beyond the math domain. We will include the GPQA evaluation in the final manuscript to further substantiate our claims.

---

weakness 2

We appreciate the reviewer’s insightful concern regarding the validity of group-level preferences. To address this, we conducted a focused loss-based analysis to examine whether the distributional shift between the merged model and the pre-merged models is substantial.

Specifically, we compute the average loss of the merged model on Long-CoT and Short-CoT samples respectively, and compare these values with the original Long-CoT and Short-CoT models.

As shown, the loss of the merged model on both Long-CoT and Short-CoT samples remains close to that of the respective original models. This indicates that the merged model retains the reasoning capabilities of both models and is thus well-positioned to leverage group-level preference signals during Stage 2 fine-tuning. These results support the effectiveness of our Bi-Level Preference Training setup, where the merged model benefits from a broader reasoning space while being guided by meaningful preference supervision.

Moreover, the effectiveness of this setup is also reflected in both our main results and ablation studies, where applying group-level preference training to the merged model leads to substantial efficiency gains with limited accuracy degradation.

---

weakness 3

Thanks for you feeback. We would like to clarify following points:

(1) Training separate models is not strictly required. In practice, one can directly utilize existing long-CoT and short-CoT models. In our experiments, we re-trained a short-CoT model only because the released long-CoT model from DeepSeek uses a completely different special token vocabulary (e.g., beginning-of-text tokens, user tokens, etc.) from its base model. To ensure fair comparison and academic rigor, we avoided introducing this confounding factor by training a compatible short-CoT model.

(2) The model merging step is lightweight and efficient. We use the open-source framework Mergekit, and the merge operation completes in under one minute.

(3 & 4) Stage 2 training is conducted as a unified process. While we refer to both group-level and instance-level preferences, the training is carried out jointly using a combined dataset that includes samples from both sources. Our loss function is designed to accommodate both types of supervision simultaneously, and the data sampling is completed in a single pass.

Overall, the pipeline is not very complex in practice and can be implemented efficiently using off-the-shelf tools and pre-existing models.

---

question 1

We appreciate the reviewer’s suggestion. While instance-level preference training does incorporate reasoning efficiency during data construction, our experiments indicate that group-level preference provides complementary benefits that are not fully captured by instance-level training alone.

As shown in the table, Merge + group level achieves significantly better performance than Merge + instance level on MATH (87.8 vs. 81.6) and GSM8k (91.6 vs. 87.95), while maintaining a higher score on AIME25 (30.8 vs. 24.2). This suggests that group-level preferences help promote more robust generalization and better tradeoffs across tasks. Furthermore, bi-level training, which combines both group- and instance-level preferences, consistently performs the best or on par with the best across all benchmarks, further supporting the complementary nature of the two.

In particular, our merged model inherits two distinct reasoning styles from its component models. Instance-level preference training focuses on fine-grained selection between individual outputs, but it does not explicitly guide the model in choosing between these broader reasoning modes. Therefore, removing the group-level stage would lead to notable performance drops. We believe this justifies the inclusion of group-level preference training in our framework.

We thank the reviewer for the time and effort spent reviewing our work. We greatly value the feedback and have done our best to address each point thoroughly in the responses below.

---

weakness 1

We thank the reviewer for pointing out the importance of the Short-CoT baseline's construction. We acknowledge that the approach of fine-tuning a Long-CoT model (DeepSeek-R1-Distill-Qwen-7B) using 2,000 Short-CoT samples from Qwen2.5-Math-7B-Instruct may appear uncommon. However, this design choice was necessitated by the characteristics of existing Long-CoT models such as the DeepSeek distillation series.

Specifically, these Long-CoT models adopt a distinct set of special tokens—different from their base or instruct counterparts—as evidenced by their configuration files on Hugging Face. These special tokens are often critical, serving functions such as marking speaker roles and delineating dialogue boundaries. In the context of model merging, it is generally required that participating models share identical special token vocabularies to ensure compatibility. Therefore, a Short-Cot model obtained by SFT from DeepSeek-R1-Distill-Qwen-7B allowed us to maintain consistency in tokenization and avoid incompatibilities caused by divergent vocabularies.

We agree that the choice and representativeness of the 2,000 Short-CoT samples could influence the performance of the Short-CoT model. However, we emphasize that the final performance of Ada-R1 (as shown in Table 2 and Section 3.2) remains stable and competitive, suggesting that the effect of this approximation on the overall conclusions is minimal.

Admittedly, a more standard alternative would involve distilling a Long-CoT model from a Qwen-base model using consistent token vocabularies, and using Qwen-instruct directly as the Short-CoT model. Nevertheless, due to the computational and time-intensive nature of large-scale distillation, this approach remains infeasible for academic research settings. We hope the reviewer can understand this resource limitation and recognize our efforts to ensure empirical rigor within these constraints.

---

weakness 2

Thank you for this valuable suggestion.Specifically, we define:

Limited Scope: The model is optimized over a narrow region of the CoT space—typically only one reasoning style, mostly Long-CoT (i.e., “thinking” paths). This restricts the model to produce lengthy explanations regardless of whether such reasoning is necessary. As a result, the model struggles to discover shorter, more efficient solutions even when they exist.

Broad Scope: The model is optimized over a wider CoT space that includes both Long-CoT and Short-CoT styles (i.e., both “thinking” and “non-thinking” paths). This enables the model to adaptively choose between concise and detailed reasoning based on the input, leading to higher reasoning efficiency without sacrificing correctness.

In practice, a model trained under a limited scope (e.g., using only Long-CoT data) tends to over-explain even simple problems, resulting in unnecessarily long outputs. In contrast, our method allows the model to flexibly produce shorter answers for easy tasks while still preserving detailed reasoning when required. This adaptivity is a key contributor to the efficiency gains observed in our experiments.

We appreciate the reviewer’s feedback and we will add a more detailed explaination in our final version of paper.

---

weakness 3

For small-scale datasets such as AIME25 (30 samples), we indeed recognize the risk of variance. The results reported in the paper are averaged over 4 independent runs to mitigate randomness. We will explicitly clarify this detail in the revised version to better reflect the robustness and reliability of our findings.

---

weakness 4

We agree that group-level preference training already achieves a strong efficiency–accuracy trade-off, reducing average CoT length by 46.03% with only a 3.31% drop in accuracy. However, adding instance-level preference training further improves this trade-off: our full bi-level method achieves a 50.93% length reduction with only a 1.65% accuracy drop—recovering nearly 2% accuracy while gaining additional compression, as shown in Table 3.

More importantly, instance-level preference serves a complementary optimization role. While group-level training focuses on selecting between long and short reasoning styles based on input difficulty, instance-level training focuses on refining within the chosen group by promoting more concise correct reasoning. These two levels target orthogonal axes —reasoning mode and reasoning verbosity—and optimizing both yields more compact and adaptive reasoning behaviors than either alone.

---

question 1

We choose model merging over mixed training for Stage I due to both efficiency and stability. First, merging is orders of magnitude faster (e.g., only tens of seconds ), making it far more suitable for hyperparameter search and rapid deployment. Second, mixed training requires careful data balance tuning, which is brittle and often leads to suboptimal trade-offs between long and short reasoning styles.

Additionally, we trained an additional model(DeepSeek-R1-Distill-Qwen-7B) as base model on mixed CoT data(2k long cot and 2k short cot) but observed worse performance compared to our merged model, despite significantly longer training time in Stage-I.

Furthermore, prior work like [1] demonstrates that mixed training can degrade existing capabilities due to task interference, whereas merging preserves task-specific strengths without retraining while [2] presents merging as a cost-effective strategy to unify diverse LLM capabilities post-training, avoiding the instability and inefficiency of retraining. What's more, model merge is very fast and only takes tens of seconds.

[1] Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learning

[2] Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities

---

question 2

see our response to weakness 2

---

question 3

Specifically, we set the threshold as 1/(2K), K is the number of candidate responses per question. We will clarify this in the final revision.

---

Final Remark

We once again thank the reviewer for their valuable comments. We hope that our clarifications and revisions have adequately addressed the concerns, and we respectfully ask the reviewer to consider a more favorable score in light of these improvements. If there are any remaining concerns, we would be glad to further clarify them.

Thank you for your feedback.

Regarding the baselines and fine-tuning rationale: All baseline methods are based on the same backbone model, DeepSeek-R1-Distill-Qwen-7B, to ensure fair comparisons. The 2,000 Short-CoT samples used for fine-tuning were derived by running inference with Qwen2.5-Math-7B-Instruct on the MATH dataset. Specifically, we started with 4,000 problems and retained only the 2,000 trajectories where the final answers were correct. For each problem, we generated one single sample using greedy decoding, with a maximum generation length of 4096 tokens.

The reason for using 2k samples for fine-tuning is that we found 2k samples are sufficient for the model to learn the desired short-CoT pattern. We chose Qwen2.5-Math-7B-Instruct as the teacher because the CoTs it generates are short CoTs without any thinking pattern, which aligns with our requirements.

---

Regarding performance: Using the 7B model as an example, our method achieves an average 50% reduction in output length, with only a minor drop in accuracy. In contrast, O1-Pruner achieves only 34% token reduction. In the field of efficient reasoning, as discussed in works such as [1][2][3][4][5][6][7], it is rare to see methods that reduce token usage by over 50% while maintaining performance. Furthermore, Reviewer dmsV also acknowledged that “The results are impressive, showing substantial token cost reductions while maintaining performance.”

In addition, our proposed adaptive hybrid reasoning represents a novel paradigm distinct from reinforcement learning-based approaches and, we believe, constitutes a key contribution of this work.

---

If our responses have adequately addressed your concerns, we would be grateful if you could consider updating your score. If there are any remaining concerns, we would greatly appreciate it if you could share them with us! Thank you once again.

---

References

[1] DAST: Difficulty-Adaptive Slow Thinking for Large Reasoning Models

[2] Training Language Models to Reason Efficiently

[3] HAWKEYE: Efficient Reasoning with Model Collaboration

[4] THINKPRUNE: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning

[5] VeriThinker: Learning to Verify Makes Reasoning Model Efficient

[6] Fast on the Easy, Deep on the Hard: Efficient Reasoning via Powered Length Penalty

[7] TL;DR: Too Long, Do Re-weighting for Efficient LLM Reasoning Compression

We thank you for your follow-up comments. Indeed, we believe there remain some misunderstandings regarding our work.

First, the proposed method introduces a new paradigm that enables the model to autonomously switch between reasoning and non-reasoning modes, thereby reducing more tokens than O1-Pruner. In contrast, O1-Pruner focuses solely on length optimization without implementing reasoning mode switching. As you have noted among the strengths: (1) The core idea of enabling LLMs to adaptively choose reasoning depth (Long-CoT vs. Short-CoT) based on problem complexity addresses a critical efficiency concern in current LRMs. (2) The results show large reductions in reasoning length without substantial accuracy loss. Additionally, reviewer dmsV emphasized that the results are impressive, showing substantial token cost reductions while maintaining performance. These points clearly reflect our model’s adaptive capabilities. Furthermore, our method also proves effective in out-of-domain settings (as demonstrated in the additional experiments requested by reviewer NgJ5), which we regard as one of our important contributions.

Second, we would like to clarify the design of the baselines. Since the model cannot always produce correct answers in a single sample—and some problems may remain unsolved even with multiple samples—we adopted greedy decoding over a larger set of datasets and selected 2k correct samples for training. For all baseline methods, we trained on the same dataset to ensure fairness.

We once again thank you for your feedback and hope that these clarifications further highlight the value and significance of our work.

Best regards,

Authors

We sincerely thank the reviewer for the thoughtful and constructive feedback. We have carefully considered all comments and provide detailed responses below to address the concerns raised.

---

weakness 1

Thank you for your concern about the method process. Although our method adopts two stages, neither is complex. Stage I (model merging) is simple weight-space arithmetic (parameter addition/subtraction) and finishes within tens of seconds. Stage II performs a standard DPO training loop after constructing the bi-level datasets and mixing them, which is also not complex. While our approach incurs a small drop in accuracy, it substantially reduces the number of generated tokens, achieving a favorable accuracy–cost trade‑off. Moreover, it offers a novel and practical perspective for enabling the model to dynamically allocate reasoning mode to each problem for efficient reasoning.

---

weakness 2

Our motivation is explicit in Sec. 3.3 and Table 1. Prior “self‑training for concise reasoning” methods (e.g., DPO‑style shortest‑vs‑longest) and O1‑Pruner optimize within the Long‑CoT distribution, pruning or compressing long traces. This preserves accuracy but forfeits exploration of genuinely Short‑CoT solutions. In contrast, Ada‑R1 first merges Long and Short models to expand the output distribution to both styles, then uses bi‑level preference training to select the style per input and compress within the chosen style, enabling broad‑scope optimization without the large accuracy drops.

---

weakness 3

In our paper, scope explicitly refers to the CoT distribution being optimized: Limited scope: optimization is confined to the Long‑CoT distribution. (only thinking paths) Broad scope: optimization covers both Long‑ and Short‑CoT (i.e., thinking and non‑thinking paths).

Prior approaches like Overthinking or O1-Pruner operate on a limited scope, meaning they focus on optimizing only one type of reasoning path—typically Long-CoT—assuming that longer reasoning is always preferable.

In contrast, our method operates over a broad scope, optimizing over both Long-CoT and Short-CoT distributions. This design reflects the intuition that the optimal reasoning path may vary by input: some queries benefit from detailed reasoning (Long-CoT), while others are better answered succinctly (Short-CoT).

---

question 1

Actually, our evaluation demonstrates that Ada-R1 is capable of generating substantially shorter CoTs (non-thinking CoTs) compared to O1-Pruner and DPO, especially when solving simpler problems. This class of concise CoTs lies outside the original distribution that DPO and O1-Pruner are optimized over, which inherently biases toward longer, detailed (thinking-style) CoTs due to their reliance on Long-CoT initialization or compression.

In Figure 3 (Thinking Ratio Study), Ada-R1 shows a higher proportion of non-thinking CoTs compared to both DPO and the Naive Merge. More importantly, Ada-R1 maintains high accuracy even on these short responses.

Figure 4 (Adaptive Reasoning Study) further validates this behavior: Ada-R1 adaptively increases the use of Long-CoTs (thinking responses) as the problem difficulty grows, while still using short and efficient non-thinking responses for easier problems. This dynamic range of CoT generation is absent in DPO and O1-Pruner, which primarily shorten existing long CoTs (i.e., compressed thinking CoTs) rather than introducing fundamentally shorter reasoning paths (non-thinking).

---

question 2

In fact, MATH is the most widely used dataset that includes different levels of difficulty. We upsample the MATH dataset to ensure the model sees a wide range of problem difficulties during training. This diversity is essential for learning adaptive reasoning—i.e., when to use long vs. short reasoning. Without enough complex problems, the model cannot learn to make this distinction effectively.

---

question 3

Thank you for pointing this out. This method is more generalizable than length-based heuristics for the following reasons: Thinking tokens explicitly reflect the model’s reasoning intent or self-reflection (e.g., hesitation, checking). This makes them a more semantically meaningful indicator of deep reasoning, as opposed to length, which can be inflated by irrelevant or verbose content. Beside, token-based cues are more transferable across different model architectures or training setups. Length distributions may vary significantly, but the presence of reasoning-related tokens like “wait” tends to be consistent among models employing Chain-of-Thought prompting. Moreover, recent studies such as [1][2][3] also determines whether the model is thinking by detecting words such as “wait”. [4] shows that certain tokens like “wait” and “let’s think step by step” are associated with peaks in mutual information, reflecting critical moments such as redirection or self-reflection during reasoning.

---

Final Remark We thank the reviewer for their valuable comments. We hope that our clarifications and revisions have adequately addressed the concerns, and we respectfully ask the reviewer to consider a more favorable score in light of these improvements. If there are any remaining concerns, we would be glad to further clarify them.

---

References

[1] Dynamic Early Exit in Reasoning Models

[2] R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search

[3] Wait, We Don’t Need to “Wait”! Removing Thinking Tokens Improves Reasoning Efficiency

[4] Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoning

Dear Reviewer,

We hope this message finds you well.

As the author-reviewer discussion period is coming to an end, we would like to kindly ask whether we have addressed your concerns. If our responses have adequately addressed your concerns, we would be grateful if you could consider updating your score.

If there are any remaining concerns, we would greatly appreciate it if you could share them with us, so we may have enough time to provide a detailed response. Thank you once again for your time and valuable feedback!

Best regards, Authors