Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning

Thanks for your valuable feedback and appreciation of the quality, clarity, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.

The LLM routing pool lacks some state-of-the-art models

We sincerely thank the reviewer for raising this point. Due to limited GPU resources and budget constraints, our experiments rely on public LLM APIs to access candidate models (Lines 257–263). We carefully selected a representative and diverse set of models. However, some cutting-edge models like GPT-4o are either unavailable via the APIs we use or incur prohibitive costs and latency, making them infeasible for repeated runs in academic settings.

Moreover, Router-R1 is inherently model-agnostic—its routing policy can generalize to any future LLMs, including stronger ones, without requiring algorithmic changes. We hope our work will inspire industrial labs with greater compute resources to further evaluate Router-R1 at scale with larger routing pools.

As for reproducibility, we ran multiple repetitions and observed consistently low variance (typically within ±0.01 EM), which does not affect the conclusions. Reported results are averaged across runs, and we omit variance values for clarity. We will also open-source our code upon publication to support reproducibility.

The results in arXiv version

Thanks for your feedback. To ensure compliance with NeurIPS's double-blind review policy and maintain fairness in the reviewing process, we have communicated the relevant details to the Area Chair. We appreciate your attention and understanding regarding this matter.

New model incorporation

Thank you for the suggestion. While we primarily evaluated Router-R1’s generalization to new models introduced at test time (Table 3), we also experimented with incorporating a new model (LLaMA3-ChatQA-1.5-8B) during training. Below we report the average EM scores across datasets when this model is added to the routing pool at training time (Router-R1-Qwen):

This result aligns closely with the test-time-only setting (Table 3), suggesting that Router-R1 generalizes well to new models and remains robust even when the routing pool is dynamically extended.

Multi-round Calling Frequency

The number of LLM calls required per query depends on multiple factors, notably (1) the complexity of the dataset, and (2) the composition of the routing pool. To understand how task complexity affects routing depth, we conduct a quantitative analysis using the average EM performance across all candidate LLMs on each dataset. This gives a proxy of dataset difficulty (harder datasets typically have lower EM scores).

This aligns with intuition (e.g., multi-hop datasets like Musique are harder), and when correlated with Figure 2a, it shows that more difficult datasets tend to invoke more LLM calls. This supports the motivation for our multi-round routing framework, which flexibly allocates computational resources based on query difficulty.

Moreover, the routing pool itself affects the observed call frequency. Due to API constraints, we use a pool of API-accessible LLMs; stronger routing pools would naturally lead to fewer calls per query. However, this does not undermine the necessity or effectiveness of multi-round routing, especially when combined with cost-aware rewards. Importantly, multi-round routing enables behaviors that one-shot methods cannot learn. For example, with cost reward enabled, Router-R1 often learns to first query smaller models, escalating to larger ones only when needed (Appendix B.2). This adaptive, efficient strategy is not achievable with one-shot routing. Finally, while current experiments focus on QA tasks, we believe the general-purpose design of Router-R1 allows easy extension to more complex reasoning domains in future work.

Runtime efficiency

We acknowledge the importance of runtime efficiency. However, a direct comparison between Router-R1 and single-round baselines in terms of wall-clock runtime would be unfair, as our setup involves heterogeneous inference environments. Candidate LLMs are accessed via APIs (e.g., Together AI), which often apply advanced backend optimizations for fast inference. In contrast, Router-R1's policy model runs locally on a single GPU without such system-level optimizations. Instead, we report inference cost w.r.t. raw cost rewards, which fairly reflects efficiency and is commonly used in prior work. As shown in the following table, Router-R1 achieves a strong cost-performance trade-off under this metric.

Novelty

Our work is motivated by the need for multi-round, adaptive routing in complex reasoning tasks—something one-shot routing fails to handle effectively. This requires a centralized policy LLM to coordinate step-wise model calls, where each candidate LLM functions as a specialized reasoning module rather than a generic tool. Inspired by DeepSeek-R1, we introduce reinforcement learning to train the policy LLM, enabling fine-grained and adaptive routing. To our knowledge, Router-R1 is the first to apply R1-style RL to LLM routing, unifying multi-round reasoning with cost-aware inference (Lines 31–39). This sets it apart from both tool learning paradigms and prior router-style works.

A core innovation is our cost reward design, which allows Router-R1 to learn efficient, hierarchical routing behaviors—favoring small models and escalating only when needed (Appendix B.2). This dynamic trade-off between cost and performance is crucial in real-world, cost-sensitive deployments and cannot be achieved by fixed, one-shot methods.

While Search-R1 also uses RL, it addresses tool use rather than model routing. Unlike tool selection among heterogeneous modules (e.g., calculators)[1,2,3], Router-R1 routes across semantically similar LLMs differing in capability and cost, which is a fundamentally different challenge. We also include Search-R1 as a baseline (Appendix A.2), but emphasize that Router-R1 is a standalone contribution tailored for multi-round LLM routing, not an extension of tool learning.

[1] Toolformer: Language Models Can Teach Themselves to Use Tools. NeurIPS 2023.

[2] CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing. ICLR 2024.

[3] ToolRL: Reward is All Tool Learning Needs. arXiv 2025.

Thank you again for your constructive feedback, and we commit to enriching the content of the paper based on the above in the revised version. We look forward to further engaging in discussions to improve the quality and impact of our work.

Dear Reviewer 7Web,

With only a few hours left in the discussion phase, we would greatly appreciate it if you could share any final thoughts or consider updating your score if our responses have addressed your concerns.

Thank you again for your time and effort.

Thanks for your valuable feedback and appreciation of the clarity, significance, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.

cost vs performance Pareto curves

Thank you for pointing this out, and we fully agree that visualizing the cost–performance trade-off is valuable for understanding routing strategy behavior.

Router-R1 was originally designed to optimize for answer performance, and thus our main results (Table 2) focus on the α = 0 setting, where cost is not explicitly penalized. The cost-aware reward term was included to give Router-R1 the flexibility to balance cost and performance when needed, which enhances its practical value, though it was not the main focus of our comparison experiments due to space constraints.

Nevertheless, we have already conducted an extensive study varying the cost coefficient α, evaluating how Router-R1 performs across different efficiency constraints. While we are unfortunately unable to include figures during the rebuttal phase (per NeurIPS policy), we include below a representative table of EM vs raw cost reward (unnormalized) across α values and baselines:

From this table, we observe:

• At α = 0.0, Router-R1 consistently achieves the highest EM across benchmarks, confirming its strong performance orientation.
• As α increases, cost drops significantly while EM gradually decreases, showing a clear trade-off.
• Compared to other baselines, Router-R1 achieves better performance when cost is lightly constrained (small α), and more efficient routing under tighter budgets (larger α). These trends echo the Pareto-like patterns seen in works like RouterBench, and validate the strength and flexibility of our cost-aware reward design. We will include full cost–performance plots and analysis in the camera-ready version.

T_out is assumed to be known which is a limitation

Thank you for the suggestion. We believe there may be a slight misunderstanding here.

In contrast to approaches like CARROT that rely on explicit cost and performance prediction to guide routing, Router-R1 does not require predicting T_out in advance. Instead, we adopt a reinforcement learning framework where the policy LLM learns to select the most suitable candidate LLMs by observing the actual outcomes of its decisions.

Specifically, during training, the policy LLM selects a candidate LLM, receives its response (whose token count T_out is observed at runtime, not predicted), and incorporates that response into its own context. The reward is computed based on the final answer accuracy and the actual token cost (T_out). This reward then guides policy learning via RL, without requiring any explicit modeling of future token usage.

In other words, Router-R1 delegates cost-performance estimation to the policy LLM itself, which is already a powerful model capable of learning complex trade-offs through experience, rather than relying on heuristic or learned cost predictors.

We greatly appreciate the reviewer’s suggestion. Integrating ideas from CARROT, such as lightweight cost prediction, to further improve training efficiency or cold-start performance could be a promising future direction, and we plan to explore this in follow-up work.

How does Eq 5 mitigate reward hacking?

Thank you for the helpful question.

While Eq. 5 defines the overall reward structure, we would like to clarify that Router-R1 adopts a hierarchical reward mechanism that is not explicitly shown in Eq. 5 for brevity (as noted in Line 167). This design plays a key role in mitigating reward hacking and improving training stability (described in Line 166–172).

Concretely, Router-R1 assigns strict priorities among the three reward components: format > outcome > cost. If the format reward is -1 (e.g., due to invalid or unparseable output), the outcome and cost rewards are ignored (set to 0); If the outcome is wrong (i.e., outcome reward = 0), then cost reward is only considered if the outcome is correct, otherwise the total reward only depends on the format reward.

Consider a situation where the output has the correct format but produces an incorrect answer. Without hierarchical reward, the model could still receive a positive total reward by minimizing the number of output tokens (i.e., reducing total T_out) and maximizing the cost reward. Over time, this may encourage the model to learn a shortcut: always produce very short, low-cost but incorrect answers, as this strategy still yields non-zero reward. This is a classic case of reward hacking. With hierarchical reward, however, once the outcome is incorrect, the cost reward is completely disregarded, and the total reward remains zero regardless of how low the cost is. This effectively eliminates such degenerate behaviors and ensures that correctness is prioritized over cheapness.

In our experiments, we found that this hierarchical reward structure significantly stabilizes training and effectively mitigates reward hacking, leading to better final performance and robustness. We appreciate the reviewer for pointing out this important detail.

Thank you again for your constructive feedback, and we commit to enriching the content of the paper based on the above in the revised version. We look forward to further engaging in discussions to improve the quality and impact of our work.

Thanks for your valuable feedback and appreciation of the quality, clarity, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.

Inference Costs

We thank the reviewer for raising this important point and would like to offer some clarifications and insights regarding the cost evaluation of Router-R1.

(1) Goal of the main experiment: performance-oriented comparison

In the main results table (Table 2), we set the cost reward coefficient $\alpha = 0.0$ to evaluate the performance ceiling of Router-R1, focusing on its multi-round reasoning capabilities. The primary goal here was not cost-efficiency but answer quality under best-effort routing. Due to space limitations, cost-performance tradeoffs were deferred to a separate analysis.

(2) Cost-performance tradeoff analysis

We conduct a comprehensive study on different cost coefficients $\alpha$ in Router-R1, comparing with baselines in terms of exact match (EM) and raw cost rewards (unnormalized). As shown in the following table, with $\alpha = 0.0$, Router-R1 achieves the highest EM, demonstrating its effectiveness in performance-first scenarios. As $\alpha$ increases, the total routing cost drops significantly, while EM degrades moderately—highlighting the controllable trade-off between accuracy and efficiency.

We will include this expanded analysis in the camera-ready version for completeness.

(3) Ensuring fairness in comparison

To ensure fairness when comparing with single-step methods, we introduced two multi-round routing baselines (Prompt LLM*, KNN Router*) that simulate step-by-step decomposition. Furthermore, we note that most prior LLM router works only include the cost of routed LLM calls and similarly omit the router's cost in their analysis[1][2]. To ensure consistency and fair comparison, we follow this practice and focus our cost evaluation on the routed candidate LLMs, which dominate the total computational overhead.

(4) Policy LLM cost is minimal, and Router-R1 enables scalable offloading

We acknowledge the concern about the policy LLM’s potential latency or cost, especially when routing to smaller models. However, in our current setup, the policy LLM is only 3B, significantly smaller than the candidate LLMs. Moreover, as shown in Appendix B (Page 23), the policy LLM performs only lightweight coordination tasks, and the vast majority of output tokens are generated by candidate LLMs. In fact, Router-R1 can reduce latency by delegating parts of the reasoning process to smaller, faster models in the routing pool, instead of relying solely on the policy model. This architecture is particularly beneficial when the policy LLM is larger than some candidate models (a setting increasingly common in practical deployments). Even though our current policy model is lightweight, the Router-R1 framework naturally supports scalable offloading, where a central controller coordinates efficient model usage. This can even lead to lower latency and cost than using a single monolithic model, especially for simple tasks. Interestingly, this resembles ideas in speculative decoding[3], where small models generate drafts and large models verify them, reducing end-to-end inference time and cost.

In summary, Router-R1 adopts a flexible and extensible architecture that balances reasoning quality and computational efficiency. We take care to ensure fair cost comparison, and we appreciate the reviewer’s thoughtful feedback, which we will further address in the final version.

[1] GraphRouter: A Graph-based Router for LLM Selections. ICLR 2025.

[2] RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models. NeurIPS 2024.

[3] Fast Inference from Transformers via Speculative Decoding. ICML 2023.

Missing Baselines

We thank the reviewer for pointing out this baseline. We did evaluate FrugalGPT in our experiments. However, we chose not to include it in the main results table, as its design prioritizes cost reduction, whereas Router-R1 is primarily aimed at maximizing performance under multi-round decomposition, especially in complex reasoning settings. We agree that including FrugalGPT helps contextualize our method across different optimization objectives. Below we provide its results for completeness:

We will include this comparison in the camera-ready version to better position our method within the broader landscape of routing strategies.

Router scalability

Thank you for raising this question. Our method is built upon the VERL framework, which already supports parameter-efficient tuning methods such as LoRA. This means that Router-R1 can be directly scaled to larger policy models using LoRA or similar approaches, without requiring full-parameter training. However, our focus in this work is not on optimizing parameter efficiency, but rather on proposing a multi-round, cost-aware LLM routing strategy. Exploring how Router-R1 behaves under different tuning paradigms (e.g., LoRA or QLoRA) is certainly interesting, and we consider it promising future work.

Sensitivity of Model descriptors

Thank you for the insightful question. To evaluate the sensitivity of model descriptors, we conducted an additional experiment where we compressed the original prompts (shown in Appendix B) to retain only basic attribute information. For example:

LLaMA-3.1-70B-Instruct: A 70B state-of-the-art model for multilingual dialogue and reasoning, excelling in benchmark evaluations.

Compared to the more elaborate descriptors used in our default setup, this version is significantly more concise. The results are shown in the table below (on Router-R1-Qwen). We observe that such modification has minimal impact on overall performance, indicating that Router-R1 is not highly sensitive to the model descriptors.

As discussed in the paper (Line 188–195), model descriptors primarily serve as a cold-start prior, helping the router bootstrap the learning process. As training progresses, Router-R1 learns to distinguish the strengths and weaknesses of candidate LLMs through interaction, rather than relying solely on initial descriptors. We will include this result in the camera-ready version.

Thank you again for your constructive feedback, and we look forward to further engaging in discussions to improve the quality and impact of our work.

Dear Reviewer Gukk,

We understand that the discussion phase may be a busy time, and we sincerely appreciate the thoughtful engagement you've shown so far.

As the deadline approaches, we would be grateful to hear any further thoughts you might have. We hope our previous responses and additional experiments have helped clarify the points you raised. If you feel that your concerns have been adequately addressed, we would kindly ask you to consider updating your review score accordingly.

Your feedback is very important to us, and we truly value the time and effort you've invested in the review process.

Thank you again!

Dear Reviewer Gukk,

With only a few hours left in the discussion phase, we would greatly appreciate it if you could share any final thoughts or consider updating your score if our responses have addressed your concerns.

Thank you again for your time and effort.

Thanks for your valuable feedback and appreciation of the quality, clarity, and contribution to our work. We appreciate your insights and would like to further address the concerns you raised about our work.

Inference Latency

We thank the reviewer for the insightful and constructive comment. We would like to clarify a key point: the multi-round, iterative reasoning of Router-R1 does not necessarily lead to increased inference latency. Instead, it adapts the number of reasoning steps to the difficulty of each task, which can actually reduce latency for simpler questions and avoid unnecessary computation.

(1) Adaptive reasoning reduces latency for simple tasks

As correctly noted by the reviewer, the latency impact depends on the task complexity. We conduct a detailed analysis in Figure 2a, Section 5.4 (Page 9), where we measure the average number of LLM calls across seven QA benchmarks. In comparison with single-hop benchmarks, multi-hop tasks (e.g., Musique, 2WikiMultiHopQA) require more reasoning steps. This task-dependent adaptivity helps minimize latency where possible. (Lines 315–322)

(2) Router-R1 offloads reasoning to lightweight models

Router-R1 can reduce latency by delegating parts of the reasoning process to smaller, faster models in the routing pool, instead of relying solely on the policy model. This is particularly beneficial when the policy LLM is larger than some candidate models (a setting that is increasingly relevant in practical deployments). Although in our current setup the policy LLM is relatively lightweight, the design of Router-R1 naturally supports scalable offloading, where a larger central controller coordinates a pool of efficient LLMs, potentially reducing the end-to-end latency compared to monolithic processing by a single large model.

(3) Case studies demonstrate latency-efficient routing behavior

In Appendix B (Page 22), we present case studies showing that with cost-aware RL training, Router-R1 prefers small models when possible, escalating to larger ones only when necessary. While the main optimization objective is cost, this behavior often correlates with lower latency, since smaller models respond faster in practice.

(4) Configurable cost-performance tradeoff

We further provide a systematic study of different cost coefficients $\alpha$, showing that Router-R1 allows flexible tuning between performance and efficiency. While this primarily reflects monetary cost, lower cost settings often result in fewer or faster model calls, which can also benefit latency. (The “cost” in the table denotes unnormalized raw cost rewards)

(5) System-level latency can be minimized in practice

While API-based access to external models (as used in our experiments) may introduce network overhead, this is not inherent to Router-R1. In practical deployment, models can be hosted locally or optimized via standard acceleration techniques. System-level latency engineering is orthogonal to the algorithmic contributions of this work and is left to future work.

In summary, Router-R1 does not introduce unnecessary latency, but rather provides a task-adaptive mechanism that adjusts inference effort based on question complexity. This design enables a flexible and efficient reasoning strategy that balances latency and accuracy.

Application in Similar Domains

We thank the reviewer for the thoughtful and encouraging suggestion. While our current experiments focus on QA tasks, the modular and hierarchical design of Router-R1 is general and well-suited to complex domains such as SWE, where the router can coordinate submodules like a project manager. We are excited to explore such applications in future work, and we believe Router-R1 holds strong potential for such applications.

The cost of policy LLM

We appreciate the reviewer’s thoughtful suggestion. In our current setup, we use a relatively small 3B policy LLM, whereas the candidate models are significantly larger. Moreover, as shown in Appendix B (Case Study), the vast majority of output tokens come from the candidate LLMs. The policy LLM primarily acts as a controller, assigning subtasks and summarizing information, with a minor computational footprint. For this reason, we did not include it in the cost calculation. That said, we agree this is a valid and practical extension, especially in industrial settings where larger policy models may be used. In such scenarios, it is indeed feasible to integrate the policy LLM cost into the overall cost analysis or even let Router-R1 jointly optimize for the total cost. We view this as a valuable direction for scaling Router-R1 beyond the academic setting and plan to explore it in future work.

Thank you again for your constructive feedback, and we look forward to further engaging in discussions to improve the quality and impact of our work.