ARM: Adaptive Reasoning Model

We thank the reviewer for the positive and encouraging feedback on the clarity, motivation, and rigor of our work. We welcome the opportunity to address your question below.

[W1] Low-level technical implementation details.

[A1] Thanks for the comment. While the main paper focuses on explaining the methodology clearly, we also provide additional implementation details in the appendix for reproducibility. Appendix B details the dataset processing pipeline. Appendix D details training configurations for both the SFT and RL stages. Appendix G details the implementation of length-penalty baselines. We will provide more technical details in our next revision, and we are happy to provide further clarification to you if needed.

We thank the reviewer for acknowledging the value of our proposed method and the supporting experiments. Below, we address your concerns in detail.

[W1] Performance and efficiency trade-off balance.

[A1] Thank you for the comment. We acknowledge that Pass@1 may show a drop compared to Maj@8, as it does not fully reflect the model’s capacity to arrive at the correct answer. To mitigate this, we additionally adopt Maj@8, which helps reduce bias and variability associated with a single generation. It provides a more robust and reliable estimate of model performance in stochastic generation settings [1].

Moreover, despite the performance drop under Pass@1, ARM achieves a better effectiveness-efficiency trade-off balance, reducing token usage while maintaining strong performance. This demonstrates ARM’s ability to reason adaptively, effectively balancing accuracy and cost.

[W2] Comparison with length penalty baseline.

[A2] Thanks for the constructive feedback. In response, we conduct a comparison between ARM-7B and L1-Qwen-7B-Exact (Due to ThinkPrune only having a 1.5B model and time limits for reproducing it from scratch, we leave it to future study). For fairness, we align the backbone model by using DeepSeek-R1-Distill-Qwen-7B for both models. During L1 inference, we set token budgets to match ARM’s average token usage on each dataset for fair comparison.

As shown in the table below, ARM consistently outperforms L1 with similar token usage, demonstrating the advantage of adaptive reasoning over length constraints. Notably, L1 requires human intervention to set the token budget manually, and an inaccurate assignment of the token budget would bring a performance drop (Please refer to our discussion on token budget estimation for length-constrained methods, presented in Response to Reviewer iAYU, A1, if you are interested). In contrast, ARM autonomously adjusts its reasoning length based on task complexity through format selection, enabling better efficiency-performance trade-offs without manual token tuning.

[W3] Experiment on LLaMA Baseline.

[A3] Thanks for the suggestion. Based on the superior performance of the Qwen model and its inherent adaptability to reinforcement learning [2, 3], we adopt Qwen as the backbone model in our experiments. We further explore models trained on LLaMA-3.2-3B and report the results below. Consistent with our main findings, ARM achieves comparable performance to the GRPO baseline while using fewer tokens across diverse task domains and complexity levels, demonstrating that our methods can generalize to other backbone models.

We also note that the token reduction on LLaMA-based models is less pronounced than on Qwen (e.g., 15.7% reduction on GSM8K vs 55.2% on Qwen-base). Upon closer analysis, we find this is caused by repetitive outputs on occasion produced by the LLaMA-based model—a phenomenon also observed in prior work [3]—which may lead to longer response lengths. This discrepancy may stem from differences in model architecture or pretraining data, and we leave further investigation to future work.

[Q1] How to organize the SFT dataset? Does every question have 4 different types of answers?

[A4] Yes, every question has four different types of answers. Specifically, we collect seed question data from AQuA-Rat. While Direct Answer and Short CoT rationales are directly provided within the dataset, we augment it with Code and Long CoT rationales using GPT-4o and Deepseek-R1, respectively. Moreover, all rationales in the code reasoning format are verified for executability using an external Python interpreter. To ensure high-quality supervision, we filter out instances whose rationales yield incorrect answers, resulting in a curated dataset of 10.8K distinctive questions spanning all four reasoning formats. During SFT, the dataset is expanded to 10.8K * 4 = 43.2K training examples, which are shuffled and randomly fed into training, allowing the model to build a foundational understanding of different reasoning formats.

[Q2] Clarification of evaluation metrics.

[A5] Thanks for the question. Pass@1 refers to the accuracy of the first sampled response. To quantify randomness, we rerun the evaluation multiple times and report the deviation across runs. As shown in the table below, the performance fluctuation is minimal across all benchmarks, typically within ±0.2 to ±0.5, and slightly higher (±1.6) only on AIME due to its small dataset size with only 30 instances and higher difficulty. These results suggest that the observed randomness is slight and within an acceptable range, supporting the reliability of our reported numbers.

References

[1] Zhang Q, Lyu F, Sun Z, et al. A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?[J]. arXiv preprint arXiv:2503.24235, 2025.

[2] Yang A, Yang B, Zhang B, et al. Qwen2. 5 Technical Report[J]. CoRR, 2024.

[3] Wang Z, Zhou F, Li X, et al. Octothinker: Mid-training incentivizes reinforcement learning scaling[J]. arXiv preprint arXiv:2506.20512, 2025.

If the reviewer has any further questions or requires clarification on any point, please feel free to ask. We are committed to making any necessary revisions to further improve our work.

We thank the reviewer for recognizing the novelty of our method, the strong experimental results, and the clarity of our writing. We appreciate the opportunity to address the concerns here.

[W1] Empirical comparison with length-constrained methods.

[A1] Thanks for the suggestion. We provide an in-depth analysis of a previous wisdom method L1 [1] here, which applies a length penalty strategy during training. At inference time, users are required to explicitly specify token budgets in the instructions for the task. We follow the official setting [1] and additionally extend the evaluation to the CSQA benchmark. Results are presented in the table below. We clarify our claims as follows:

Users’ inaccurate token estimation hurts performance, since the length penalty strategy assumes prior knowledge of the task to predefine the appropriate reasoning length. If the user’s estimation is not accurate enough, the performance degradation is significant: 1) Underestimated budgets hurt performance on complex tasks. On harder benchmarks like AIME, performance is severely degraded under low budgets: only 3.3% at 512 tokens, improving gradually to 26.7% at 3600 tokens. Such under-estimation of required reasoning length severely limits performance on challenging benchmarks. 2) Large budgets waste resources on simple tasks. On easier benchmarks like CSQA, enforcing large reasoning budgets leads to token usage increases with minimal, or even detrimental, gains. For example, CSQA accuracy rises marginally from 45.8% at 512 tokens to 46.1% at 3600 tokens. In contrast, ARM learns to allocate longer reasoning only when necessary, avoiding both under- and overestimation.

[W2] Ablation study and generalization of reasoning formats.

[A2] Thanks for the thoughtful question. The four reasoning formats used in ARM are carefully designed to balance accuracy and token efficiency, and have proven effective across a variety of tasks. We also provide more detailed reasons in Section 3.1.

To further investigate whether defining other reasoning formats would help, we add an additional reasoning format: function-calling, implemented via code execution. Specifically, during inference, when the model selects the Code reasoning format, we run the generated code using an interpreter to obtain an answer. If execution fails, the model falls back to simulating the output of the code [2].

As shown in the table below, incorporating this function-calling format yields improvements, suggesting that more fine-grained formats can provide benefits. However, we also note that incorporating additional reasoning formats demands extra resources to implement the pipeline. For example, parallel function calls during training can lead to high memory consumption, and decreasing the number of processes may prolong the training time. Furthermore, formats like function-calling would introduce additional inference-time latency (e.g., due to runtime execution). Therefore, we adopt the current four reasoning formats for their widespread use and practicality, and leave the exploration of more reasoning formats to future work.

We further examine the effect of removing specific reasoning formats on performance and token efficiency. Specifically, during inference, we remove the Direct Answer format on CSQA, Short CoT on GSM8K, and Long CoT on AIME’25. The results are summarized in the table below: On CSQA, removing the Direct Answer format increases token usage by +29.4% with negligible accuracy gain, showing it is crucial for efficiently handling simple tasks. In contrast, on AIME’25, removing Long CoT leads to a significant accuracy drop (-8.4), confirming its importance for complex reasoning. These results validate the necessity of the predefined reasoning formats in enabling adaptive reasoning.

Our existing evaluation spans across in- and out-of-domain commonsense, mathematical, and symbolic reasoning. To further investigate the generalizability of ARM, we further extend the evaluation to two additional benchmarks: 1) GPQA, a complex QA dataset requiring compositional reasoning, and 2) StrategyQA, a multi-hop reasoning benchmark.

As shown in the table below, ARM achieves comparable accuracy to the GRPO baseline while significantly reducing token cost by an average of 50%, and up to 65% in StrategyQA, consistent with our main findings. This demonstrates that ARM’s adaptive reasoning behavior generalizes well to more tasks. Thanks for your suggestion again, and we will update the results in our next revision.

[Q1] Will training data bias cause format collapse?

[A3] Thank you for the thoughtful question. You’re right, bias can indeed result from selecting unsuitable training data. To investigate this, we analyze an ARM trained solely on the AIME dataset (1983-2024), which primarily favors Long CoT solutions due to its competition-level complexity. We present the result in the table below. We evaluate this model on two simpler tasks—CSQA and OBQA—and observe that the model overwhelmingly selects Long CoT (~80%) even when simpler formats would suffice. This confirms that training on a biased dataset can indeed lead to over-reliance on a single reasoning format.

To mitigate this, ARM is trained on a diverse mixture of datasets across a wide range of difficulties. As shown in the table below, the full ARM recipe achieves both higher accuracy and significantly reduced token usage on the tasks, demonstrating that our approach effectively prevents format collapse and encourages adaptive reasoning behavior across domains.

References

[1] Aggarwal P, Welleck S. L1: Controlling how long a reasoning model thinks with reinforcement learning[J]. arXiv preprint arXiv:2503.04697, 2025.

[2] Li C, Liang J, Zeng A, et al. Chain of Code: Reasoning with a Language Model-Augmented Code Emulator[C]//International Conference on Machine Learning. PMLR, 2024: 28259-28277.

Thank you once again for your feedback. We welcome any additional concerns or suggestions you may have.

We thank the reviewer for recognizing our paper’s clarity, strong empirical results, and practical impact. We appreciate the thoughtful and constructive comments and the opportunity to address the concerns raised.

[W1, Q1, Q4] Limited task diversity and generalization concerns.

[A1] Thank you for the question. Our selection of evaluation testbed is based on both domain and complexity; thus, we incorporate the current seven sets to demonstrate the generalization, ranging from easy commonsense to symbolic and complex mathematical reasoning. We also acknowledge that GPQA and StrategyQA are two good choices to demonstrate the performance of ARM, and we extend our evaluation to these two additional benchmarks based on Qwen2.5-7B.

As shown in the table below, ARM achieves comparable accuracy to the GRPO baseline while significantly reducing token cost by an average of 50%, and up to 65% in StrategyQA, consistent with our main findings. This demonstrates that ARM’s adaptive reasoning behavior generalizes well to more tasks. Thanks for your suggestion again, and we will update the results in our next revision.

[W2, Q3] Lack of ablation on format set.

[A2] Thanks for the thoughtful question. The four reasoning formats used in ARM are carefully designed to balance accuracy and token efficiency, and have proven effective across a variety of tasks. We also provide more detailed reasons in Section 3.1.

To further investigate whether defining other reasoning formats would help, we add an additional reasoning format: function-calling, implemented via code execution. Specifically, during inference, when the model selects the Code reasoning format, we run the generated code using an interpreter to obtain an answer. If execution fails, the model falls back to simulating the output of the code [1].

As shown in the table below, incorporating this function-calling format yields improvements, suggesting that more fine-grained formats can provide benefits. However, we also note that incorporating additional reasoning formats demands extra resources to implement the pipeline. For example, parallel function calls during training can lead to high memory consumption, and decreasing the number of processes may prolong the training time. Furthermore, formats like function-calling would introduce additional inference-time latency (e.g., due to runtime execution). Therefore, we adopt the current four reasoning formats for their widespread use and practicality, and leave the exploration of more reasoning formats to future work.

We further examine the effect of removing specific reasoning formats on performance and token efficiency. Specifically, during inference, we remove the Direct Answer format on CSQA, Short CoT on GSM8K, and Long CoT on AIME’25. The results are summarized in the table below: On CSQA, removing the Direct Answer format increases token usage by +29.4% with negligible accuracy gain, showing it is crucial for efficiently handling simple tasks. In contrast, on AIME’25, removing Long CoT leads to a significant accuracy drop (-8.4), confirming its importance for complex reasoning. These results validate the necessity of the predefined reasoning formats in enabling adaptive reasoning.

[W3, Q2, Q5] Training cost and inference overhead.

[A3] Thank you for the valuable comments. We acknowledge that preparing multi-format data introduces some annotation overhead. However, our SFT data pipeline is clear, lightweight, and practical: we leverage existing Direct Answer and Short CoT rationales in the original dataset, and construct the remaining formats (Code and Long CoT) using GPT-4o and DeepSeek-R1 via well-defined prompts (Appendix B). This process is automated, reproducible, and not resource-prohibitive, especially considering the benefit of enabling the model to build a foundational understanding of different reasoning formats.

As for the RL stage, Ada-GRPO is based on GRPO, which is substantially more memory- and compute-efficient than other RL algorithms, like PPO [2]. GRPO is also commonly employed in length-penalty methods such as L1 and ThinkPrune. Our proposed Ada-GRPO further improves efficiency, reducing training time by half relative to vanilla GRPO through adaptive format selection, as discussed in Section 5.2. In terms of wall-clock training time, ARM-7B completes in 37 hours, making it ~2× faster than the GRPO baseline, which takes 64 hours.

Regarding inference cost, although Long CoT is still invoked for easy tasks, ARM significantly reduces the token usage overall. Additionally, the three proposed reasoning modes are designed to support flexibility under different deployment requirements: 1) Adaptive Mode strikes a superior balance. 2) Instruction-Guided Mode offers a clear advantage when the assigned reasoning format is appropriate.3) Consensus-Guided Mode, while more costly, is performance-oriented and applicable in high-reliability settings. The inference token overhead associated with each mode is illustrated in Table 2 in Section 4.3. Below, we further report the inference time overhead under different mode settings, using Adaptive Mode as the baseline (normalized to 1) based on response length.

[W4, Q5] Comparison with length penalty baseline.

[A4] Thank you for the constructive feedback. In response, we conduct a comparison between ARM-7B and L1-Qwen-7B-Exact, which combines GRPO with a length penalty strategy. Due to the limited availability of ThinkPrune—which is only released for a 1.5B model—and time constraints preventing reproduction from scratch, we leave its comparison to future work. For fairness, we align the backbone model by using DeepSeek-R1-Distill-Qwen-7B for both models. During L1 inference, we set token budgets to match ARM’s average token usage on each dataset for fair comparison.

As shown in the table below, ARM consistently outperforms L1 with similar token usage, demonstrating the advantage of adaptive reasoning over length constraints. Notably, L1 requires human intervention to set the token budget manually, and an inaccurate assignment of the token budget would bring a performance drop (Please refer to our discussion on token budget estimation for length-constrained methods, presented in Response to Reviewer iAYU, A1, if you are interested). In contrast, ARM autonomously adjusts its reasoning length based on task complexity through format selection, enabling better efficiency-performance trade-offs without manual token tuning.

[W5, Q5] Some training hyperparameters of Ada-GRPO description.

[A5] Thank you for the thoughtful comment. We extend GRPO into Ada-GRPO by introducing two key components: 1) Format Diversity Scaling Factor to promote format exploration, and 2) Decay Factor to gradually shift focus to performance optimization.

The scaling factor $G/F(o_i)$ is motivated by the group-wise nature of GRPO, where rewards are normalized within the groups. By shaping the reward in this way, Ada-GRPO effectively prevents format collapse into Long CoT, encouraging the model to explore diverse efficient reasoning formats. The scaling factor changes with the decay factor: as steps increase, the scaling factor decays from $G/F(o_i)$ to 1, thereby adjusting its sensitivity. This design helps prevent long-term misalignment due to the over-rewarding of rare formats, ensuring that the model transitions from exploration to exploitation in a controlled manner.

We provide a systematic evaluation of this in Appendix A.2, showing that the decay schedule leads to smoother and more stable accuracy improvements, especially during mid-to-late training. We agree that more sensitivity analysis (e.g., $2*G/F(o_i)$) is valuable, and we plan to expand on this in future work.

References

[1] Li C, Liang J, Zeng A, et al. Chain of code: Reasoning with a language model-augmented code emulator[J]. arXiv preprint arXiv:2312.04474, 2023.

[2] Shao Z, Wang P, Zhu Q, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models[J]. arXiv preprint arXiv:2402.03300, 2024.

Thank you for your thoughtful suggestions and careful review again. Please feel free to reach out if you have any additional concerns. We remain committed to refining and improving our paper.

Thank you for your rebuttal and clarification. I have no further questions.