Smoothie: Label Free Language Model Routing

We thank the reviewer for their thoughtful comments and for engaging with our work. We are grateful they appreciated (1) the generality of Smoothie’s algorithm beyond supervised methods, and (2) the breadth of experimental results.

We refer the reviewer to our general response for more information on the performance statistics for individual models in the experiments (W2), as well as a discussion of the limitations. We apologize for the lack of presentation clarity in the current version of the experimental section. We hope that revisions in response to this review and others will improve clarity.

Our response here addresses concerns regarding (1) the types of tasks Smoothie can be used for (W1), and (2) whether Smoothie requires more computation than supervised methods (Q2).

Performance on non-semantic tasks

The reviewer noted that because Smoothie relies on embeddings of model generations to learn quality scores for each model, Smoothie is inherently limited by the types of relationships those embeddings can capture. We agree with this observation, and have mentioned it in the limitations in the global response. However, we find that current embedding models do allow Smoothie to perform well on tasks like mathematical reasoning, as shown by our evaluation on GSM8K in the global response.

Does Smoothie require more computation than supervised methods?

Supervised methods require engineers to expend computation in two ways. First, engineers must produce generations for the ensemble over a training set. Second, engineers must train a router–typically using SGD–to map queries to models in the ensemble. At test time, supervised routers only expend computation performing inference for the selected model, i.e., one model out of the ensemble.

Smoothie does not require the creation of a large training dataset, and operates solely on test-time generations without annotations. Fitting Smoothie’s weights is significantly cheaper than training a router via SGD, with our method taking only seconds in practice due to their closed form in Algorithm 1. However, Smoothie requires a generation from each model in the ensemble for the test sample, which supervised methods do not.

Fortunately, the need for computing all model generations per test sample can be removed with a small algorithmic tweak, making Smoothie even more efficient and its runtime independent of $n$. Suppose we have a held-out set of $n_{train}$ train samples with precomputed generations from the models in the ensemble. For each test sample, we retrieve the most similar train samples, learn the Smoothie weights for the sample using the corresponding train sample generations, and return the model with the highest Smoothie weight (i.e., in line 5 in Algorithm 1, KNN is now over a held-out training dataset). The benefit here is that Smoothie selects the model for a test sample without needing model generations for that sample. Only $n_{train} \times m$ generations are needed, regardless of how large the test dataset $n$ is.

Smoothie is still effective with this modification, which we refer to as Smoothie-Train. For instance on the 3B and 7B ensemble (using a train set of size 250, compared to test sets of size 1000), Smoothie-Train identifies the best performing model on 8/14 single-task datasets, outperforming a random selection baseline by up to 7.1 points rouge2 and 12.5 points accuracy. On the multi-task datasets across both the 3B and 7B ensemble, Smoothie-Train outperforms random sampling by an average of 3.7pts accuracy on the accuracy tasks, and by an average of 1.6pts rouge2 on the rouge2 tasks.

Dear reviewer gxgv,

Thank you so much for raising your score. We agree that the performance on GSM8K is interesting and will update our final draft with these results!

We thank the reviewer for their feedback and are happy that they found the topic interesting. In our global response, we discuss the limitations of Smoothie, which the reviewer pointed out. Below, we address the reviewer’s comments on writing clarifications as well as the Minimum Bayes method.

Writing clarifications: We will update the draft to address the writing errors mentioned in the weaknesses section. For selecting j and k in the Smoothie algorithm, they can be randomly selected, although to reduce variance we average over all $C(m-1, 2)$ pairs of (j, k) to get $i$’s Smoothie weight. We will clarify this in our paper.

Minimum Bayes Risk (MBR): One way of minimizing Bayes risk is to select an output that has the highest similarity to all other outputs (such as [1], mentioned in [2]). That is, the model to route to for sample $x$ is $\arg \max_i \sum_j sim(g_i(x), g_j(x))$, where $sim$ is cosine similarity, for instance.

We demonstrate theoretically and empirically that Smoothie-Local with k=1 (no nearest neighbor smoothing) is conceptually similar to MBR. Note that Smoothie routes to the model with the lowest value of equation 3 in the paper. If we ignore the subtraction of $\delta_{jk}(x)$, Smoothie selects the model $\arg \min_i \sum_j \delta_{ij}(x)$; for k=1, this is the generation with the lowest embedding distance from all other generations for x. Since L2 distance is inversely correlated with cosine similarity, Smoothie-Local (k=1) is hence similar to MBR. Most importantly, we found that MBR and Smoothie-Local (k=1) matched performance on both multi-task datasets and both ensembles (a 0.0003 point average difference).

Therefore, Smoothie is a more general version of the MBR rule above that can almost exactly recover MBR’s behavior when k=1. Moreover, we find that Smoothie performs better than MBR for 4 out of 7 single-task datasets when using a larger k. We will update our draft to compare to MBR.

[1] https://aclanthology.org/D18-1449/

[2] https://arxiv.org/pdf/2310.01387

We thank the reviewer for their detailed feedback and are glad that they found the problem interesting and practical. Below, we address the reviewer’s concerns around related work and baselines, differences with Shin et. al., and writing clarifications.

Related work: Thank you for your suggestion on moving more related works up into the body. We will add relevant works on routing and ensembling to the body in our updated draft.

We compare and evaluate the AEC method presented in the paper mentioned in W2. In this paper, each generation has two embeddings, a global embedding (Universal Sentence Encoder) and a local one (GLEU). They solve an optimization problem that estimates each generation’s error by constructing a loss that enforces that the local and global embedding similarity between the true generation (produced by a weighted average) and the candidate generation should be the same. In contrast, Smoothie uses one embedding space, relies on a multivariate Gaussian structure among embeddings, and does not require gradient descent to learn the weight parameters.

To evaluate AEC, we implemented it ourselves since we were unable to find a codebase online. We used 100 epochs and a learning rate of 0.001 for all datasets. We find that Smoothie outperforms AEC on multi-task datasets by 0.9 points on average, and on single-task datasets by 2.3 points on average across 3B and 7B ensembles. We will update our draft with the discussion and empirical results on AEC.

Finally, in the spirit of the reviewer’s requests for additional baselines, we also report a comparison to PairRM. We refer to the global response for a description of this baseline’s performance.

Comparison with Shin et. al.: Both Smoothie and Shin et. al. use a multivariate Gaussian model. However, in Smoothie we apply it to model routing with SBERT embeddings on natural language datasets, whereas Shin only conducts synthetic experiments in hyperbolic spaces and metric spaces induced by synthetic graphs. Moreover, Smoothie uses nearest neighbor kernel smoothing to allow for sample-dependent weights—critical for routing—while Shin only calculates one global set of weights over the dataset. We will add this comparison to the related work in the body of our paper.

Writing clarifications: We thank you for pointing these writing errors out and apologize for them. We will address them in our updated draft.

We thank the reviewer for their thoughtful comments and for engaging with our work. We are glad to hear they appreciated our approach and the comprehensiveness of our experiments. We refer the reviewer to our general response for more information on (1) Smoothie’s routing behavior, and (2) Smoothie’s limitations. Our individual response to reviewer RLiV focuses primarily on the concerns regarding the efficiency of routers.

Why run a pool of small models as opposed to one large model?

Reviewer RLiV asked about the tradeoff between spending resources on a single large model, as opposed to multiple small models. We believe there are several reasons why multiple smaller models might be preferred to a large model.

First, an ensemble of small models may exceed the performance of a large model while incurring the same computational cost. For instance, we observe that applying Smoothie to an ensemble of three 1-2B parameter models (Qwen, Gemma, and Phi-3) outperforms each of the following 7B models by up to 20pts accuracy on SQuAD: Llama-2, Storm, Snorkel, Vicuna, Nous-Capybara, and Mistral.

Second, engineers increasingly access models through APIs, and API calls for larger models can often be 4-5x the cost of API calls to smaller models. On Together for instance, Llama-3 8B costs 10c per million tokens and Llama-3 70B costs 54c per million tokens. Running an ensemble of five 8B models thus costs less than a single 70B model.

Can Smoothie be made more efficient?

Reviewer RLiV also asked whether it was possible to incorporate more discussion of Smoothie’s efficiency. To estimate the Smoothie weights for routing, we use a simple closed-form procedure that does not require any SGD or training, as described in Algorithm 1. As a result, Smoothie weights on the entire dataset can be computed in seconds—for the 7B ensemble, SmoothieLocal on the multi-task datasets takes 2.14 seconds per 1000 samples, and SmoothieGlobal on the single-task datasets takes under 0.03 seconds per 1000 samples. Moreover, Smoothie does not require any ground-truth annotations; however, all m model generations per test sample are needed as input to the algorithm. That is, we need $n \times m$ generations for a test dataset of n samples.

Fortunately, the need for computing all model generations per test sample can be removed with a small algorithm tweak, making Smoothie even more efficient and its runtime independent of $n$. Suppose we have a held-out set of $n_{train}$ train samples with precomputed generations from the models in the ensemble. For each test sample, we retrieve the most similar train samples, learn the Smoothie weights for the sample using the corresponding train sample generations, and return the model with the highest Smoothie weight (i.e., in line 5 in Algorithm 1, KNN is now over a held-out training dataset). The benefit here is that Smoothie selects the model for a test sample without needing model generations for that sample. Only $n_{train} \times m$ generations are needed, regardless of how large the test dataset $n$ is.

Smoothie is still effective with this modification, which we refer to as Smoothie-Train. For instance on the 3B and 7B ensemble (using a train set of size 250, compared to test sets of size 1000), Smoothie-Train identifies the best performing model on 8/14 single-task datasets, outperforming a random selection baseline by up to 7.1 points rouge2 and 12.5 points accuracy. On the multi-task datasets across both the 3B and 7B ensemble, Smoothie-Train outperforms random sampling by an average of 3.7pts accuracy on the accuracy tasks, and by an average of 1.6pts rouge2 on the rouge2 tasks.

We will update the paper to include this discussion.