Thank you for your thoughtful feedback. We have addressed your concerns in detail below.

Q1: Technical innovation.

A1: We would like to emphasize that our RAGRouter is not merely a combination of existing techniques in contrastive learning and routing, but a novel framework driven by a previously unaddressed problem: RAG-aware dynamic LLM routing.

We are the first to explicitly define the new RAG-aware LLM routing problem. As RAG becomes a mainstream paradigm, different LLMs vary significantly in how they incorporate and utilize retrieved, non-parametric knowledge. However, existing LLM routing methods typically assume static model capabilities and overlook the impact of retrieval context.

To deal with this new problem, we design a new router network architecture that introduces document encoder and RAG capability embedding layer to model how LLMs absorb and respond to retrieved evidence. Further considering the retrieval setting can substantially alter an LLM’s answerability by shifting the knowledge representation, we propose intra- and cross-setting contrastive learning objectives to capture the performance variations with and without RAG settings. Ablation studies have also validated the necessity and effectiveness of each design module.

Q2: Performance gains in scenarios with poor retrieval.

A2: We have reported and analyzed RAGRouter’s performance under noisy retrieved documents in Appendix C (Routing Performance under Noisy Retrieval) of the submission. In particular, we add synthetic noise into the NQ, WebQ, and TriviaQA datasets. Take TriviaQA for example. The evaluation results under four conditions, including Golden Context, Relevant Noise, Irrelevant Noise, and Counterfactual Noise, are summarized as follows.

We can observe that our RAGRouter consistently outperforms traditional LLM routing methods, with performance gains more than 4.58%, highlighting its strong robustness to retrieval noise.

Q3: Comparison with recent routing methods that also aim to capture dynamic shifts in LLM capabilities.

A3: We have conducted a comprehensive literature search and found some concurrent works [1,2] that also explore routing strategies under dynamically shifting LLM capabilities. These studies typically focus on reasoning-oriented scenarios and capture the internal knowledge of LLMs (e.g., different CoT strategies and internal state transitions) to guide routing decisions.

In contrast, our RAGRouter focuses on a totally different setting: routing for LLM with RAG, where capability shifts arise from the interaction between LLMs and external knowledge. This introduces new factors, including varying document quality, model-specific sensitivities to retrieved content, and compatibility between models and documents, which were not found in existing reasoning-centric scenarios. RAGRouter models these new factors to enable effective routing in RAG. It captures how different LLMs respond to varying retrieved contexts through a new network architecture and contrastive learning framework. In contrast, existing LLM routing methods focus on reasoning dynamics and cannot represent or utilize external knowledge, making them not applicable for the new RAG routing setting.

In summary, our RAGRouter can be viewed as a complementary and parallel approach to existing LLM routing methods, offering specific advantages in RAG scenarios, where external knowledge plays a central role in shaping model behavior.

[1] Pan Z, Zhang K, Zhao Y, Han Y. Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection. arXiv preprint arXiv:2505.19435. May 26, 2025.

[2] Zhang W, Qiao S, Luo L, Li Y, Zheng C, Xu Q, Li M, Gui Y, He Y, Qiu J, Hong J. SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model. arXiv preprint arXiv:2507.02822. Jul 3, 2025.

Q4: Training efficiency of RAGRouter.

A4: We clarify that RAGRouter and all other LLM routers are trained on resource-rich cloud infrastructure. Despite this, RAGRouter remains lightweight, with just 136M parameters. It was trained using a single consumer-grade NVIDIA RTX 4090D GPU, achieving an average training time of 11.6 minutes on PopQA and MedMCQA, and 4.5 minutes on NQ, WebQ, and TQA. The peak memory usage during training was only 18.9 GB.

Thanks for your thoughtful feedback on our paper. We have provided responses to all your concerns below.

Q1: Performance on commercial strong LLMs, such as GPT4o.

A1: We have followed your suggestion to add commercial strong LLMs, including GPT-4o and Qwen2.5-Max, to the original candidate LLM pool with parameter size $\ge$ 32B. We then evaluate our RAGRouter on NQ and WebQ, comparing its performance against non–RAG-aware baselines, including KNN Router and MF.

Evaluation results show that although GPT-4o is the top-performing individual model, our RAGRouter delivers an additional +0.63% improvement, while invoking GPT-4o in only 52.3% of samples. RAGRouter further outperforms Qwen2.5-Max (+6.67%), KNN Router (+4.59%), and MF (+1.88%). These results highlight RAGRouter’s effectiveness even in the presence of highly powerful LLMs, leveraging model complementarity to surpass each single LLM, with better latency and cost efficiency by selectively using GPT-4o.

Q2: How well does RAGRouter generalize to other RAG tasks?

A2: Following your suggestion, we have supplemented experiments on WikiASP [1], a summarization task commonly used in RAG-related research. In particular, WikiASP aims to generate aspect-based summaries for Wikipedia entities and naturally supports the use of retrieved evidence. We adopt ROUGE-L as the evaluation metric.

Evaluation results show that RAGRouter outperforms non-RAG-aware routing baselines by more than 5.3%. These findings demonstrate that RAGRouter generalizes well to different RAG tasks.

[1] Hayashi, Hiroaki, et al. "Wikiasp: A dataset for multi-domain aspect-based summarization." Transactions of the Association for Computational Linguistics 9 (2021): 211-225.

Q3: It requires ground-truth labels of "can answer vs. cannot answer".

A3: We clarify that RAGRouter doesn't rely on strict "can answer vs. cannot answer" discrete labels. For continuous performance metrics like ROUGE-L, we can use a score-based probabilistic sampling strategy to generate positive and negative sample pairs for contrastive learning in our RAGRouter. Specifically, for each candidate LLM's score $s\in[0,1]$, we label it as "can answer" (positive sample, label 1) with probability $s$, and "cannot answer" (negative sample, label 0) with probability $1-s$. We have applied this strategy in our newly added experiments on the WikiASP summarization task using ROUGE-L scores, and the results validate its effectiveness. This further highlights the flexibility of RAGRouter in handling tasks with continuous performance metrics, without strictly requiring binary supervision.

Thanks for your feedback! The rebuttal has fully resolved my questions and I decide to keep my score positive.

Thank you for your careful review and valuable feedback. We address all your comments below.

Q1: Clarify the correspondence between the three core RAG-aware factors and the design of different modules.

A1: We clarify the correspondence as follows:

• The factor “the non-parametric knowledge provided by the documents” corresponds to our Document Encoder.

• The factor “the LLM’s ability to process external information” is captured by the RAG Capability Embedding Layer.

• The factor “the query’s role in guiding knowledge retrieval” is represented by the Cross Encoder.

We will incorporate these clarifications into the revised manuscript.

Q2: Include very strong closed-source models.

A2: We have followed your suggestion to add the powerful closed-source LLMs, including GPT-4o and Qwen2.5-Max, to the original candidate LLM pool with parameter size $\ge$ 32B. We then evaluate our RAGRouter on NQ and WebQ, comparing its performance against non–RAG-aware baselines, including KNN Router and MF.

Evaluation results show that although GPT-4o is the top-performing individual model, our RAGRouter delivers an additional +0.63% improvement, while invoking GPT-4o in only 52.3% of samples. RAGRouter further outperforms Qwen2.5-Max (+6.67%), KNN Router (+4.59%), and MF (+1.88%). These results highlight RAGRouter’s effectiveness even in the presence of highly powerful LLMs, leveraging model complementarity to surpass each single LLM, with better latency and cost efficiency by selectively using GPT-4o.

Q3: Experiments on capability vector dimensionality.

A3: We have conducted hyperparameter studies on varying capability vector dimensionality in Appendix E (Figure 9 and Table 9) of the submission, investigating how different embedding dimensions affect performance across five different datasets. A summary of the results is shown below:

As 768-dimensional embeddings yielded the best overall performance, we adopted this setting in the main experiments.

Q4: Will the dataset used to train RAGRouter be open-sourced?

A4: We will release the dataset used to train RAGRouter, along with detailed descriptions of its construction and usage instructions.

Thank you for your thoughtful feedback. We have addressed each of your concerns as follows.

Q1: Ablation study of the RAG Capability Embedding.

A1: We have followed your suggestion to add an ablation study to validate the effectiveness and necessity of explicitly modeling the LLM’s intrinsic RAG capability.

In the ablation setting “w/o RAG Capability Embedding”, we removed the explicit modeling of RAG capabilities and instead assigned each LLM a single embedding that captures its overall behavior, without distinguishing between parametric knowledge and RAG capabilities.

The ablation study results show that without the RAG Capability Embedding, the performance across 5 datasets with local and online retrieval strategies drops by 0.76% in average. Notably, on NQ and WebQ, which involve imperfect retrieval results, the performance drops by 1.25% and 2.5%, respectively. These findings validate the effectiveness of our explicitly introduced and disentangled RAG capability representation, which plays a critical role in enabling effective RAG-aware routing by accounting for the LLM’s ability to process external information in dynamic knowledge update scenarios.

Q2: Failure and correct case analysis.

A2: We have followed your suggestion to add the quantitative analysis of cases where RAGRouter and traditional LLM routing methods fail to select the correct LLM or route correctly on the datasets of PopQA (Local), MedMCQA (Local), and NQ. In particular, we categorize non-trivially unsolvable failures (i.e., excluding cases where all LLMs fail) into 4 types as follows:

• F1: Failure to perceive performance degradation caused by RAG (i.e., cases where LLM's performance decreases after RAG is applied).

• F2: Inherent task difficulty, where the majority of LLMs (e.g., >80%) fail, leading to routing confusion.

• F3: Overconfident selection of stronger LLMs for cases where high-capacity LLMs are chosen but still fail, outside the conditions of F1 and F2.

• F4: Other factors, such as outliers or ambiguous inputs.

We present the type-wise failure rates (as a percentage of all cases) and the overall failure rate (FR) in the table below.

We can observe that compared with traditional LLM routing methods, including KNN Router and MF, our RAGRouter achieves the lowest overall failure rate and the lowest F1-type failure rate. This suggests that RAGRouter more correctly captures performance shifts in LLMs introduced by RAG. We have also provided detailed cases in Appendix F where RAGRouter selects correctly while traditional LLM routing methods fail.

We further examine cases where RAGRouter fails while traditional LLM routing methods succeed. We find that such cases are less than 2% of all the evaluated cases. Notably, more than 50% of these cases fall into the F2 type, where the performance gap between candidate LLMs is inherently small.

Q3: Performance when faced with noisy retrieved documents.

A3: We have reported and analyzed RAGRouter’s performance under noisy retrieved documents in Appendix C (Routing Performance under Noisy Retrieval) of the submission. In particular, we add synthetic noise into the NQ, WebQ, and TriviaQA datasets. Take TriviaQA for example. The evaluation results under four conditions, including Golden Context, Relevant Noise, Irrelevant Noise, and Counterfactual Noise, are summarized as follows.

We can observe that our RAGRouter consistently outperforms traditional LLM routing methods, with performance gains more than 4.58%, highlighting its strong robustness to retrieval noise.

Q4: Additional evaluations on domain transferability or robustness to domain shift.

A4: To evaluate RAGRouter’s generalization across domains, we design two cross-domain settings:

• Setting 1: Trained on MedMCQA (Local) that falls into medical domain, but tested on a new dataset HotpotQA for Wikipedia-based multi-hop QA.

• Setting 2: Trained on NQ that contains real-world search queries, but tested on MedMCQA (Local).

Evaluation results show that RAGRouter still outperforms traditional LLM routing methods by 1.00% and 5.56% in the two cross-domain settings, demonstrating strong domain generalization. This performance advantage can be attributed to RAGRouter’s contrastive learning framework, which effectively captures relative RAG capability differences among candidate LLMs, even when transferred across domains.