Large Language Model Routing with Benchmark Datasets

We thank the reviewer for their questions and comments. Please see our responses below.

The paper only compares with relatively straightforward instance based approach while in related routing LLM sections, it does mention more approaches but not compare with them.

As discussed in the corresponding related work section, all prior methods consider instance-based LLM selection and (1) require a sufficient amount of labeled data from the new task to train a scoring or ranking model and (2) require generating outputs with every candidate LLM. In contrast, we consider a setting where no (or very little) labeled data/references on a new task are available, which we believe is a more practical scenario. The only baseline applicable in such a problem setting is LL (or perplexity), which we compare to in our experiments. We note that LL still suffers from (2), while our method avoids this problem while also outperforming this baseline.

One exception is the experiment in section 5.2, where the train and test data are from the same distribution and thus it is possible to compare to additional baselines mentioned in the related work section, e.g., PairRanker (note that these baselines continue to be inefficient dues to (2)).

In Figure 2, we also added an additional baseline, Few-Shot, which uses the small number of labeled samples from the new task to select an LLM.

The O.O.D problem is well investigated and discussed, however, the authors don't compare with other ways of estimating the accuracy. One popular approach is like G-eval which might overcome the O.O.D issues in some extent.

In appendix B.1 we report results for an alternative method to estimate accuracy, ATC as proposed by Garg et al., and we show that this method for estimating accuracy does not perform as well compared to the method we used. As we report, the MAE for the kernel smoother is 0.118, and the MAE for the ATC is 0.177.

The suggested G-Eval score relies on GPT-4 and thus is a very expensive and closed-source method for accuracy estimation. One of the motivations for our work is to improve the collective capabilities of open-source models, hence using GPT-4 inside the method is not suitable.

There is no clear conclusion that can be drawn from the paper that in practice, what score to use (and with O.O.D score, whether one should use a score).

Based on the experiments in Figure 2, we recommend using kNN as the correctness predictor together with S3 while labeling ~5 samples from the new task for the best tradeoff between the router performance and labeling efforts. The reason to use kNN over MLP is that it is straightforward to have it take advantage of the few labeled samples from the new task, while MLP would require retraining/fine-tuning with up-weighted new-task labeled samples. When it is possible to obtain more labeled samples from the new task (~25-50), we recommend S1 instead of S3 as the kNN classifier becomes better calibrated, thus reducing the need to model correctness with S3. This recommendation is also supported by the results in Figure 2 for higher $\alpha$ values and is briefly discussed in the last paragraph of Section 5.1.

That said, rather than a concrete practical recommendation, the main contribution of our work is the new problem formulation of LLM routing using benchmark datasets that has the potential to improve the collective capabilities of open-source LLMs. The use of benchmark datasets is important as it allows researchers with limited computational resources to contribute to LLM research by reusing publicly available LLM (per-sample) evaluation data (e.g., HELM, Open LLM Leaderboard, evaluation data released as part of our code) instead of running costly evaluations themselves. Thus, our main focus is on bringing forth various aspects of the problem along with methodological approaches to address them.

We thank the reviewer for their questions and comments. Please see our responses below.

I believe that the application scope of this work may be somewhat limited. The main approach relies heavily on the Large Language Model (LLM) learning from past similar tasks and using that knowledge to measure performance on new tasks. However, I would like to highlight that acquiring the necessary "knowledge" for a slightly larger LLM can be a costly process, requiring evaluation on a large number of benchmarks.

While our approach relies on data from evaluating LLMs across tasks, it is crucial to highlight that we propose to reuse benchmark evaluations. The current practice of releasing new LLMs is to accompany them with an extensive evaluation of these LLMs across many tasks to demonstrate their credibility. In other words, there is essentially no additional cost to apply our method provided new LLM releases are accompanied by per-sample benchmark evaluation results. Such data is already readily available for some benchmarks, e.g., HELM provides per-sample evaluation results that can be downloaded from their website and Open LLM Leaderboard provides per-sample evaluations for over 1000 LLMs on their benchmark datasets, which can be downloaded from HuggingFace. Thus, model routing using benchmark evaluations is cost-efficient and practical given current practices, while it also provides a new LLM-related problem to study for researchers with limited computational resources. Please also see the discussion in Section 2 in the "Benchmarking" paragraph.

When dealing with a new dataset, an alternative approach could involve evaluating a selection of promising models on a smaller amount of data to identify the best-performing model. This evaluation process can be time-efficient. Thus, the efficiency of different models could also be included.

Such an approach would be inefficient as it would require (1) obtaining labels/references for some amount of data from the new task and (2) evaluating every candidate LLM on the data from the new task (i.e., outside of the benchmark evaluation data). Results presented in Table 1 demonstrate that our method is capable of selecting an LLM without any labeled data from the new task and without running inference with every candidate model on any of the data from the new task. Our method also outperforms perplexity (LL)-based LLM selection, which bypasses (1), but still suffers from (2).

Another concern I have is related to the adequacy of the baseline comparisons. It seems that a simple baseline approach could involve evaluating all relevant LLMs ( < 100) on a very small dataset and selecting the best-performing model.

Thank you for the suggestion! While in Table 1 we consider a setting where no labels/references are available for the new task, Figure 2 presents results with our method when a few labeled samples are available. We have added a baseline (Few-Shot) that uses performance only on those labeled samples to select an LLM. We see that our method outperforms such a baseline by a large margin.

---

Please let us know if you have any further questions or concerns. If we have addressed your concerns, we would appreciate it if you could consider increasing the score.

We thank the reviewer for their questions and comments. Please see our responses below.

I'm somewhat confused about whether it's crucial to use "imperfect" one if we have a "perfect" correctness predictor. In other words, why do we opt for an imperfect correctness predictor, such as a non-parametric classifier, instead of a parametric one?

If we have a "perfect" correctness predictor, the problem is fairly trivial and all proposed scores will perform as the oracle. Unfortunately, the perfect correctness predictor is generally unattainable, analogous to perfect accuracy being unattainable in most supervised learning tasks, especially with distribution shift. The "imperfection" is not related to the choice of correctness predictor (nonparametric vs parametric) but to the challenging nature of the binary classification problem corresponding to correctness prediction. In Appendix B.1 we present results using a neural network as the correctness predictor, i.e., a parametric classifier.

From my perspective, this work bears similarities to the Mixture of Experts (MoE) model, where experts in MoE are replaced with LLMs. So, what is the distinguishes between this work and MoE, where LLMs serve as experts? Would the non-parametric method remain efficient if we use it for the traditional MoE?

The traditional application of MoE is to decide which expert(s) to use based on the input. In this paper, we mainly consider the problem of selecting a single model for a task, i.e., a collection of inputs. It is perhaps possible to use some variation of MoE to improve correctness predictors ($g_m$s), thus potentially improving the effectiveness of model routing with our scores, however, it is unclear how MoE alone can be used in our problem setting.

One exception is the experiment in section 5.2, where the model is selected per sample and some of the baselines, e.g., PairRanker (which requires inference with all experts unlike our method), can be viewed as variations of MoE.

We have added a brief discussion regarding MoE in the conclusion.

This paper doesn't seem to clarify the difference between this method and certain fine-tuning techniques, nor does it address whether the proposed method outperforms the current fine-tuning methods. If we fine-tune the selected LLM, would it certainly perform better than an LLM that hasn't been selected? Or should we use the selected LLM directly after the router has chosen it?

We consider the problem of selecting an LLM for a new task where no (or very few) labels/references are available, thus fine-tuning is not a viable option. In our experiments, the LLM selected by the router is directly applied to the samples from the new task.

When the new task comes with a sufficient amount of labels/references, choosing a model most suitable for fine-tuning is an interesting problem, but is beyond the scope of this work. In the future work part of the conclusion, we briefly discuss the potential impact of routing models that were already fine-tuned for various domains, i.e., "expert" LLMs, where we anticipate our method to continue being effective.

The results from the candidate LLMs (Table 5) clearly indicate that larger models outperform their smaller counterparts. This might suggest that the optimal strategy is simply to choose the largest model available. However, I believe this perspective may not be entirely accurate. Therefore, I propose showing more detail to challenge and potentially debunk this assumption.

Note that Table 5 presents the average performance of models over all datasets, where indeed larger models outperform smaller ones. However, our results throughout the paper demonstrate that when choosing different models for different benchmark datasets we are able to improve the average performance, compared with a single large model. Specifically, Table 1 demonstrates that our method outperforms llama-2-70b (referred to as Best Model on Average, or BMA, the largest model considered which has the best average performance as shown in Table 5) while reducing inference cost by occasionally choosing smaller models (see "# Params" column). In addition, in Figure 2, we show that the results of our approach can be improved further when using in-distribution data, obtained, for example, by evaluating each LLM on a small subset of the new task, in addition to the benchmark data used for training our predictor.

---

Please let us know if you have any further questions or concerns. If we have addressed your concerns, we would appreciate it if you could consider increasing the score.

We thank the reviewer for their questions and comments. Please see our responses below.

The concept of routing is a prevalent strategy in conventional Mixture-of-Experts (MoE) solutions. More comprehensive discussion and experimental comparisons are encouraged.

The traditional application of MoE is to decide which expert(s) to use based on the input. In this paper, we mainly consider the problem of selecting a single model for a task, i.e., a collection of inputs. It is perhaps possible to use some variation of MoE to improve correctness predictors ($g_m$s), thus potentially improving the effectiveness of model routing with our scores, however, it is unclear how MoE alone can be used as a baseline.

One exception is the experiment in section 5.2, where the model is selected per sample and some of the baselines, e.g., PairRanker (which requires inference with all experts unlike our method), can be viewed as variations of MoE.

Following your suggestion, we have added a comment regarding MoE in the conclusion.

Results of "S3 true p" need further practical analysis.

We present this score to demonstrate the upper bound on the proposed model routing score that takes into account the potential inaccuracies of the correctness predictors. As discussed at the top of page 7, we find it encouraging that even with correctness predictors that on average have accuracy as low as 0.59 on a binary classification task, it is conceptually possible to route LLMs efficiently.

$g_m(x)$ is defined to evaluate the correctness of model $m$ on input $x$ and gold label $y$. The lack of $y$ in $g_m(x)$ causes confusion.

$g_m(x)$ is a classifier which estimates the probability that an LLM $m$ will generate the correct answer for an input $x$. The probability is determined entirely from $x$ so including $y$ in $g_m(x)$ would be incorrect.

Eq. 1 is a little bit confusing. It estimates the loss of $g_m(x)$ and $y(x,m)$ given $g_m$. $g_m$ seems to be both independent variable and dependent variable.

For our approach, we are training the probabilistic estimator $g_m(x)$ to predict the gold label $y(x, m)$. Eq. 1 states that we are selecting the $g_m(x)$ that minimizes the binary cross-entropy loss between the prediction and the gold label. In other words, we are simply training a standard binary classifier in eq 1. We have clarified this in the updated draft.

The problem of OOD in Eq.3 lacks necessary discussion. Eq.3 does not contain notation of $P(y|x)$, it estimates $g_m$. Although the target of $g_m$ is to estimate the correctness of model $m$ that can be potential affected by OOD, there still lacks necessary clarification about what $P(y|x)$ represents in Eq.3.

Eq. 3 uses $g_m$, which is an estimator of $P(y(x,m)=1|x)$ as stated after eq. 1. We are simply stating that a (probabilistic) binary classifier may fail to generalize out-of-distribution, particularly in terms of its calibration quality. We have clarified this in the updated draft.

The relation of solution of Eq.4 and OOD is not clear.

Eq. 4 presents one of the model routing scores based on predictions with $g_m$s. As a binary classifier, $g_m$ is likely to have lower accuracy when evaluated on OOD data, thus the score based on these predictions may be less effective when selecting an LLM for a new task. We have clarified this in the updated draft.

There is only one optimization problem, that is Eq.1, to find the best predictor function. Given the learned $g_m$, the rest of the method is to use $g_m$ to choose the language model. How to design a better predictor also needs more discussion.

We reiterate that eq. 1 is simply saying that we are training a binary classifier. How to train binary classifiers has been extensively studied in ML literature over the past several decades. In this paper, we considered simple choices such as kNN and MLP and instead focused on the questions that are specific to our problem setting and have not been studied previously, e.g., how to improve model routing with imperfect classifiers.