Let the LLM Stick to Its Strengths: Learning to Route Economical LLM

We sincerely thank Reviewer MPGf for the valuable feedback. We have carefully considered each point and provide our responses below:

1. Regarding the high cost of constructing the model-data interaction dataset:

First, we commit to open-sourcing the model-data interaction dataset. The cost of its construction is a one-time, offline expense. In application, this data resource can be continuously and seamlessly collected from interactions between the models and users.

Once the dataset is built and EcoRouter is trained on it, the online routing inference process is extremely efficient. (We have added an experiment in our response to Reviewer 3Co2 to substantiate this claim).

Furthermore, this construction cost can be amortized. The process is not GPU-intensive (as it primarily involves inference), and most of the data was generated using NVIDIA 4090 GPUs.

• Handling New Data: Our routing system can generalize directly to new data as it emerges.
• Handling New Models: When a new model is introduced, our method rapidly adapts by performing inference on a lightweight "bridge" coreset to update the capability distribution. This avoids the need for a full re-inference on all historical data and complete retraining, making the cost of small-scale updates manageable.

To enable the router to accurately learn the "preferences" and capability boundaries of different LLMs, a large-scale, high-quality history of interactions is indispensable. This is crucial for achieving precise and economical routing. We will elaborate on these upfront costs in the final version of the paper.

2. Regarding the computational overhead of the router model itself:

The online inference overhead of EcoRouter primarily stems from two components:

1. Feature Extraction: We use a small model like GTE-large (~0.33B) [307] to extract text embeddings. Its parameter count and computational cost are significantly smaller than any LLM in our candidate pool.
2. Router Model Inference: The recommendation framework we employ [327] is essentially a shallow network, making its inference extremely fast with negligible overhead.

We have supplemented our paper with a table detailing the routing latency:

We will add a dedicated section in the final version to analyze the specific latency and other related overheads of the routing model in detail. Thank you for this valuable suggestion.

3. Regarding the simplification of the cost function:

The formula Cost = Price_per_token × Total_tokens is indeed a simplification of real-world API pricing models. We adopted this approach for the following reasons:

• This formula effectively captures the two most critical factors influencing cost: the model's scale (which determines the unit price) and the length of the generated content. Similar proxies are widely used in related work such as FrugalGPT, AutoMix, and Hybrid-LLM.
• A core strength of EcoRouter is its flexibility concerning the cost function. The function is not hard-coded into the model architecture; it is primarily used to rank the training data (see Equation 2 in the paper). To adapt to more complex pricing strategies (e.g., different pricing for input/output tokens, tiered pricing), we only need to replace the cost function, re-rank the training data accordingly, and retrain the recommendation system. The fundamental methodology and framework remain unchanged.
• We have added an experiment where cost requirements are provided as a descriptive input. The results are shown in the table below:

We will discuss this simplification more explicitly in the limitations section of our paper and emphasize that our framework can be easily adapted to more sophisticated cost models.

4. Regarding the comparison with SOTA LLMs:

• EcoRouter is not a new LLM but rather an LLM routing framework or scheduling system. Its performance ceiling is determined by the models within its candidate pool.
• Our framework is designed to leverage and orchestrate these SOTA models, not to compete with them. To demonstrate this, we have conducted a new experiment by adding Qwen2.5-14B to our candidate model pool. This integration is lightweight: we only need to perform inference with the new model on the coreset to obtain its capability distribution, without any need for fine-tuning the router. The results are as follows:

As shown, EcoRouter effectively routes relevant requests (e.g., for GSM8K) to the newly added, more capable Qwen2.5-14B to boost performance. Concurrently, for simpler tasks, it continues to select more economical models, thus maintaining cost-efficiency.

5. Regarding the impact of few-shot sample quantity on performance:

• Inference Modes as Routing Options: In EcoRouter, we treat the tuple (model, inference_mode) as a holistic candidate for routing. The input features for the router can be augmented with additional information, such as the number of few-shot samples or the reasoning strategy to be used (e.g., CoT, ToT [343]).
• Learning the Trade-off: Increasing the number of few-shot samples generally improves accuracy but also increases the context length, leading to higher inference costs. EcoRouter is specifically designed and trained to learn this trade-off.

Once again, we thank you for your valuable time and constructive feedback. We are confident that by incorporating these revisions based on your suggestions, the quality of our paper will be significantly enhanced.

I am very grateful to the author for his efforts in the rebuttal process. The author's rebuttal partly resolved my doubts (I don't see any explanation about Q1 and Q3), and I decided to maintain the previous rating unchanged.

We sincerely thank Reviewer k8wC for their meticulous review of our work. We have added relevant experiments and will present an improved study in the final version of the paper.

1. Regarding the comparison with other model routing methods and a more comprehensive performance and cost analysis:

We have incorporated nine additional comparison methods and seven new benchmarks, including tasks in mathematics, coding, and multi-turn dialogue. This expansion covers all the evaluation metrics considered in RouterBench. For a detailed breakdown, please refer to our response to Reviewer 3Co2.

2. Regarding the missing citations:

Thank you for your careful examination. We have conducted a rigorous proofreading of the entire manuscript and will ensure that all citations are complete and accurate in the final submission.

3. Regarding the definition of the Pareto Frontier and its relevance to the subsequent discussion:

We introduced the concept of the Pareto Frontier to provide a clear framework and ultimate goal for the complex optimization problem of LLM routing. An ideal routing system should identify a Pareto-optimal solution within the two-dimensional space of "cost-accuracy."

In our experimental tables, we establish a theoretical upper bound by creating a "Best-Performing" baseline—a composite model selecting the optimal model for each specific task. This represents a point on the theoretical Pareto Frontier that prioritizes maximum accuracy.

For instance, in Table 2, the average performance of our method (e.g., Ours w/ DIN at 78.20%) closely approaches the Pareto Frontier (79.39%). Crucially, the average response length required to achieve this performance (473) is significantly lower than that of large models like Meta-Llama-3 70B (608).

The table below presents further results demonstrating that by incorporating cost-accuracy control inputs into our routing process, the resulting router can form a superior Pareto Frontier.

In the final version of the paper, we will include a "cost-accuracy" scatter plot to visually demonstrate how EcoRouter effectively pushes the overall system performance towards the theoretical Pareto Frontier.

We sincerely thank Reviewer N3dk for their recognition of our work's contributions, including: innovatively framing the LLM routing problem as a cost-aware recommendation task, introducing the Pareto-front metric to evaluate the cost-benefit trade-off, and explicitly embedding cost into the training process.

1 & 2. Regarding Baseline Models and Evaluation Benchmarks:

We have added nine new comparative methods and seven new benchmarks (including tasks for mathematics, code, and multi-turn dialogue), taking into account all the evaluations mentioned in RouterBench. For details, please refer to our response to Reviewer 3Co2. We will incorporate the relevant experimental results and analysis in the final version of the paper.

3. Regarding the Ablation Study on Feature Groups:

We conducted a study on combinations of different model features, as shown in the table below:

(Note: The checkmark '✓' indicates the feature group was included, and a blank space indicates it was excluded, corresponding to '1' and '0' in the original table respectively.)

4. Regarding the Scale of Training Data:

We are committed to open-sourcing all of our training data. This data was generated through LLM inference, with the majority of interactions with smaller LLMs being conducted on a 4090 GPU.

Furthermore, training the recommendation model is not as time-consuming or expensive as training a large language model. The entire training process for our reported results took approximately 30 hours on a system with 8x NVIDIA A100 GPUs, indicating that the volume of training data is manageable.

We have supplemented our work with an analysis of routing accuracy under reduced training data sizes:

We will include this sensitivity analysis on the data scale in the final version of the paper. Thank you very much for this valuable suggestion.

We sincerely thank Reviewer 3Co2 for their detailed and insightful review of our work.

We are pleased that the reviewer recognized several core strengths of our work, including the importance of the problem, the clarity of our formulation, and the effectiveness of our experiments.

In the following, we provide detailed explanations and clarifications for the points raised. We are committed to incorporating the corresponding revisions into the final version of our paper:

1. Comparison with SOTA Routers

We have added several SOTA comparison methods, including FrugalGPT, GraphRouter, Hybrid-LLM, and RouteLLM. Furthermore, we have included additional experiments on RouToo, RouterDC, and EmbedLLM.

In addition to the existing six benchmarks, we have expanded our evaluation to include Hellaswag, Winogrande, MMLU-Pro, MATH, MBPP, HumanEval, and MT-Bench. This expansion covers a broader range of tasks, including general knowledge, mathematics, code generation, and multi-turn dialogue.

Our method creates a detailed model representation for the routing process. By using a paradigm of (model representation + instruction) -> predicted performance, we consider various aspects of the models, such as their intrinsic information and capability distributions. In a sense, this approach encapsulates the principles of many of the compared methods.

Our replication details are as follows:

1. FrugalGPT (Cascade Strategy)

   • Routing Mechanism: This method employs a cascade calling and post-generation scoring mechanism. It does not predict performance but rather evaluates the generated answer after the fact.
     • Input: (generation result from the current model, original instruction). The output is similar to a reward model, assessing whether the result is satisfactory.
   • Replication Details: We constructed an LLM cascade using Qwen2-7B-Instruct, Qwen2-57B-A14B-Instruct, and Qwen2-72B-Instruct. For the scoring function, we used the DistilBERT model from their official code, retrained on our own data. To ensure evaluation stability, we did not use the "prompt adaptation" or "LLM approximation" (caching) strategies from the original paper.

2. GraphRouter

   • Routing Mechanism: Treats routing as an edge prediction task on a graph.
     • Input: The instruction and a description of the models.
   • Replication Details: Due to limitations in the official implementation regarding multi-GPU distribution for single-graph data, we completed the replication using a subset of the training data.

3. Hybrid-LLM

   • Routing Mechanism: DeBERTa + a classifier.
     • Input: The instruction. The model representation consists of the parameters of each DeBERTa classifier.

4. RouteLLM

   • We primarily replicated its two core strategies (the other two are similar to Hybrid-LLM):
     • Similarity-Weighted Ranking: Input is a new instruction. It is compared against historical data and modeled using the Bradley-Terry model. Output is a full ranking of the models.
     • Matrix Factorization: Input is (instruction embedding, model embedding), where the model embedding is derived from the factorization of a historical performance matrix. Output is a predicted performance score.

5. Methods from RouterBench

   • We replicated its non-cascade routers:
     • k-Nearest Neighbors (KNN) Router: Input is the embedding vector of the request. Output is a performance estimate based on the average performance of the K-nearest neighbors (where historical performance data serves as the model representation).

6. Routoo

   • Routing Mechanism: Uses a decoder-only LLM to process the input, combined with a unique model embedding vector.
     • Input: (instruction, target model ID). The model ID maps to a learnable "model embedding." We did not adopt this user-item style of learnable representation because it would require incremental retraining of the entire routing structure whenever a new model is added.

7. RouterDC

   • Routing Mechanism: Trains the router based on contrastive learning.
     • Input: (instruction, learnable model embedding).

8. EmbedLLM

   • Routing Mechanism: Learns compact representations of models via matrix factorization.
     • Input: The instruction and relevant historical performance data.

2. Cost in Relation to Real API Pricing

Our cost model is based on estimations of model scale, architecture, and generation length. Our initial goal was to establish a general and relatively stable framework for measuring cost that does not directly depend on the frequently changing and complex pricing rules of specific API providers (like OpenAI or Google). This approach ensures our conclusions have broader applicability.

Due to network instability, we have not yet completed supplementary experiments with real APIs. However, in the final version, we will purchase API access for the relevant models (with their real-world pricing) and include a comparison of the actual costs incurred.

3. Lack of Discussion on Routing Latency

The online inference overhead of EcoRouter primarily comes from two components: 1) Feature Extraction: We use a small model like GTE-large (~0.33B) to extract text embeddings. Its parameter count and computational cost are significantly smaller than any LLM in the candidate pool. 2) Recommendation Model Inference: The recommendation framework we employ is essentially a shallow network, making its inference extremely fast with negligible overhead.

We have added the routing latency in the table below:

4. Over-reliance on Scale as a Cost Proxy

We would like to clarify our cost calculation. In our framework, the unit token cost is estimated based on the model's scale. The total cost is then calculated as the product of this "model's unit token cost" and the "length of the generated output (in tokens)".

We will update the paper to make this clearer in the final version, particularly in the tables. We appreciate this valuable suggestion.

5. Potential Train-Test Data Leakage

Our "bridge coreset" and the evaluation test sets are completely disjoint. We will state this more explicitly in §3.1 and §4.1 of the paper.

Furthermore, we've observed that the learning difficulty increases significantly when there is a large distributional shift (measured by the Wasserstein distance) between the historical performance dataset and the target evaluation set.

Once again, we thank you for your valuable feedback. We will continue to work diligently to improve our paper.