AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking

We would like to thank you for such a constructive review, and we are glad to address the questions in response below.

Comparison with HAG

Thank you for pointing this out. We apologize for missing the comparison with HAG [1]. We're glad to incorporate new baselines and appreciate the suggestion.

At the same time, we would like to clarify that while HAG self-regulates hyperparameters such as temperature, top-p, top-k, and repetition penalty, our method operates in a different action space, including reasoning steps or methods. Comparison is provided in table below. Please note that current API like GPT-4o, no longer supports top-K and repetition penalty. So, instead, presence_penalty is used in our implementation of HAG.

Design of action space (rationale and analysis)

Thank you for the thoughtful comment. Indeed, as shown in Table 2, optimizing the reasoning method ($a_p$) contributes the most to the performance gain. However, the other actions ($a_s$, $a_t$) also lead to clear improvements over baselines. Detailed plots of distribution of actions can be found in Appendix E. Intuitively, this can be somehow expected, as reasoning method instruction does convey the most information.

Our design of the reasoning instructions is principled. For the action space, we only include discrete temperature values and exclude other hyperparameters, as co-tuning temperature with top-p is disencouraged (see lines 162–163) and other hyperparameter as random seed is not effective as revealed in ablation studies. For design of $a_p$, we select mainstream cognitive strategies such as self-audit and creative reasoning, and introduce variations based on structure-mapping theory from cognitive psychology [1].

Additional Comparison and technical details

We trained adareasoner for 5 epochs as standard. We would fill in more details and avoid any confusion in the experimental parts in our camera-ready version.

[1] Gentner, Dedre. "Structure-mapping: A theoretical framework for analogy." Cognitive science 7.2 (1983): 155-170.

Thank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!

We sincerely thank you for the helpful feedback, detailed responses and recognition of our work. To further clarify on the concerns, we provide responses below.

LLM as Judge reliance

We evaluated only on QA benchmarks with unambiguous targets, and our answer‐generation prompts (Figs 9–10) explicitly instruct the model to "provide a clear result" ensuring a well‐defined ground truth. To handle synonyms and paraphrases, our judge prompt (Fig 8) asks GPT‑4o to evaluate semantic equivalence or other cases rather than exact string matches. We engaged three graduate students, each tasked with annotating three batches from every dataset, with each batch containing 50 samples. The resulting F1 scores are reported in the table below.

Text generator (template editing) experiments

Thank you for these valuable suggestions. While our current action space is discretized, adopting a continuous representation would enable finer‑grained control and integrate naturally with policy network architectures. As a next step, we plan to implement a text‑generator policy operating in this continuous space—freeing us from rigid discrete templates and delivering a more flexible, adaptive reasoning configuration.

Report in single api

Due to API limitations and associated costs, we initially only reported results from a single run. However, as pointed out by other reviewers, we agree that reporting variability is important. We have since included results from multiple runs and new baselines to further reveal the stability of adareasoner. Additionally, we report further experiments using aggregating independently trained single policy heads and expriments on perturbed policy results with deviation result. We also report deviation here calculated by three independent experiments.

• Adareasoner-close-perturb : replaces one action from the original AdaReasoner policy with its closest neighbor in the action space. For reasoning methods, the replacement is selected based on cosine similarity between prompt embeddings (e.g., perturbing step 4 to step 3 or 5).
• Adareasoner-distant-perturb : replaces the action with the third-closest neighbor, allowing us to observe performance changes under larger deviations.
• Adareasoner-emsemble : Three indepdently trained action head with action emsembled together.

Thank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!

We appreciate your detailed feedback and reflective comments. To further clarify and tackle the your concerns, we provide responses below by turn.

Reasoning extensive benchmarks (math and coding)

First of all, we selected Metaphor, MMLU, and similar datasets not because they're easy or lack reasoning challenges, but to demonstrate AdaReasoner's versatility. One of our motivations is to adapt LLM generation across all task types—not just traditionally "serious" domains like coding or math but also those domains that can't be appropriately handled by simple CoT and needs creative thinking or other reasoning methods, etc. Thus a diverse selection of datasets is naturally needed.

Second, we are glad to provide follow-up experiments on math and coding benchmarks. Respectively, we use CodeMMLU [1] as code generation benchmark and HARP(Human Annotated Reasoning Problems) dataset [2] as math understanding benchmark. To make our evaluation more thorough, we also added CaseHOLD [5] datasets of context and causal domain of reasoning intensive questions (extracted from law exams). In our experiments, 300 samples of each dataset are randomly extracted and 50 are used as few-shot examples with 250 as test samples. We independently run the experiment three times and calculate the standard deviation and use GPT-4o in LLM-as-Judge framework.

We have also conducted experiments where no clear and static answer can be provided and evaluated by ROUGE, which can be seen in response to reviewer TrHj.

In-context-learning evaluation

We would like to clarify that In-context CoT is not the only in-context-learning (ICL) method in baselines. We apologize for any confusion caused by our too concise or misleading description of Auto‑CoT in Section 4 line 267. As noted in lines 90–91, Auto‑CoT generates its own reasoning exemplars, effectively functioning as an in‑context learning method and its action space is how to select the demo text. In our implementation, we select by clustering the nearest 10 (defaulted in Auto-CoT) examples for each question among the 100 few-shot examples (25 on each dataset). Then we run Zero‑Shot‑CoT (“Let's think step by step”, temperature=0.1) to produce a rationale and answer, then concatenate all ten question–rationale–answer triplets into a single in‑context prompt, leaving all other settings at their defaults.

In addition, we are also glad to provide additional experiments on other ICL methods, like Curriculum Demonstration Selection[3], which in detail selects top 10 cases from simple to hard simulating curriculum learning ranked by each question's shannon entropy and details provided in table below.

Theoretical K Few-Shot Guarantee

Bound in line 603 of $K \gg |A|\ln|A|$to drive average regret to nearly zero is the worst-case guarantee for an unfactored, adversarial bandit. In real settings, Adareasoner requires much fewer than this bound, and several reasons can be attributed to this cause.

• A shared encoder layer enforces $L$‑smoothness and near‑convexity of the policy objective [4], enabling quick convergence under small $K$.
• A pretrained reward model delivers high fidelity and SNR feedback (“warm start”), accelerating policy updates under few‑shot budgets.

LLM configuration and policy optimality

It is important to clarify that only the action space is universal, not the trained AdaReasoner policy itself. AdaReasoner operates over a shared action space (temperature, reasoning steps, and method), but each LLM requires its own adapter to align the model’s internal representation with the policy. Our ablation study (Table 2)—for example, using GPT-4o with Qwen’s adapter—shows significant performance degradation, confirming that adapters must be trained separately for each model.

Moreover, due to the presence of regret, AdaReasoner's policy is not truly optimal but only an approximation. To examine this, we include two new analyses of perturbed policy. Additionally, we report further experiments using aggregating independently trained single policy heads as the main adapter—with no shared layer also in the table below. We independently run the experiment three times and calculate the standard deviation.

• Adareasoner-close-perturbed : replaces one action from the original AdaReasoner policy with its closest neighbor in the action space. For reasoning methods, the replacement is selected based on cosine similarity between prompt embeddings (e.g., perturbing step 4 to step 3 or 5).
• Adareasoner-distant-perturbed : replaces the action with the third-closest neighbor, allowing us to observe performance changes under larger deviations.
• Adareasoner-emsemble : Three independently trained action heads with action emsembled together.

These analyses provide clear evidence of AdaReasoner's policy suboptimality.

[1] Manh D N, Chau T P, Le Hai N, et al. Codemmlu: A multi-task benchmark for assessing code understanding capabilities of codellms[J]. CoRR, 2024.

[2] Yue A S, Madaan L, Moskovitz T, et al. HARP: A challenging human-annotated math reasoning benchmark[J]. arXiv preprint arXiv:2412.08819, 2024.

[3]Vu, Duc Anh, et al. "Curriculum Demonstration Selection for In-Context Learning." Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing . 2025.

[4]Du, Simon, et al. "Gradient descent finds global minima of deep neural networks." International conference on machine learning. PMLR, 2019.

Thank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!

We sincerely thank you for the thoughtful and constructive feedback, and detailed responses. We hereby provide clarification to the concerns and questions as below.

---

Generalization on open-ended questions

AdaReasoner can address open-ended questions by adjusting the reward signal, even in the absence of ground truth. However, evaluating its outputs using an LLM-as-judge framework remains nontrivial.

Nonetheless, extending AdaReasoner to open-ended tasks is both promising and of practical interest. To explore this direction, we conducted experiments on the RepliQA dataset, evaluating with ROUGE-L recall against expert reference answers (recall is used as only the reference matters). Due to API constraints, we randomly sampled 250 test examples and 50 training examples from the dataset and provide result in table below.

Auto-CoT configuration tuning

We apologize for any confusion caused by our too concise or misleading description of Auto‑CoT in Section 4 line 267. As noted in lines 90–91, Auto‑CoT generates its own reasoning exemplars, effectively functioning as an in‑context learning method and its action space is how to select the demo text. In our implementation, we select by clustering nearest 10 (defaulted in Auto-CoT) examples for each question among the 100 few-shot examples at BERT embedding space. Then we run Zero‑Shot‑CoT (“Let’s think step by step”, temperature=0.1) to produce a rationale and answer, then concatenate all ten question–rationale–answer triplets into a single in‑context prompt, leaving all other settings at their defaults.

Incorporation with advanced reasoning strategies

In our initial AdaReasoner design, we implemented a Best‑of‑N strategy: for each chosen configuration, the LLM generates N responses in parallel, which are then scored and the highest‑scoring output selected. We found the performance gains marginal—since the N responses were often nearly identical under a complex generation instruction or long context—while computing costs rose substantially. Looking ahead, integrating a Tree‑of‑Thought approach offers a promising avenue for further improving AdaReasoner and would be our next target.

Case study and fine-grained analysis

We are glad to clarify more on patterns and already have Appendix E section plots and tables to analyze. As shown in Appendix E, several clear patterns emerge. Table 6 reveals that the MMLU (Math) dataset requires, on average, more reasoning steps than the Metaphor dataset, as one would expect. LogiQA exhibits significantly different distributions of $a_s$ and $a_t$ between correct and incorrect responses. Table 7 highlights interesting trends in the most frequently selected reasoning‑method keywords: self‑checking is broadly preferred, multi‑perspective analysis appears in Metaphor and TruthfulQA, and MMLU emphasizes structured reasoning. Likewise, Figure 7 illustrates the relative efficiency of $a_p$.

Baselines with flexibility and ablation studies

We appreciate the suggestions on hyperparameter configuration. We have conducted experiments on perturbed configurations as provided below. Additionally, we report further experiments using aggregating independently trained single policy heads as the main adapter—with no shared layer also in the table below. We also report deviation here calculated by three independent experiments.

• Adareasoner-close-perturb : replaces one action from the original AdaReasoner policy with its closest neighbor in the action space. For reasoning methods, the replacement is selected based on cosine similarity between prompt embeddings (e.g., perturbing step 4 to step 3 or 5).
• Adareasoner-distant-perturb : replaces the action with the third-closest neighbor, allowing us to observe performance changes under larger deviations.
• Adareasoner-emsemble : Three independently trained action head with action emsembled together.

As for reward with cost-aware signal, we appreciate this thoughtful suggestion. Although we have not explored cost-aware rewards in this version, we agree it is a valuable direction and plan to investigate it in future work.

---

Thank you once again for your detailed feedback—it has been extremely helpful in improving our work. Please don’t hesitate to reach out if you have any further questions. Thank you!