Optimising Calls to Large Language Models with Uncertainty-Based Two-Tier Selection

Speculative decoding is not directly relevant to operational costs-saving approaches as it involves calling the target (large) model for each query in the verification step, which results in the same performance as the random baseline.

We will incorporate additional results and a discussion on long-form generation in the final version of the paper.

Ramirez et al. 2023 proposed using Margin Sampling in the context of a locally finetuned model. We show that Margin Sampling is a good router across off-the-shelf LLMs and outperforms recent works that use neural routers. Moreover, we extend results for short-form generation and a multi-task setup.

LLMs of different families

Since Margin Sampling only uses the probability distribution of the small model, we hypothesise that it should still be a reliable estimator for routing when using LLMs of different families. To validate this, we conduct additional experiments. The following results are in the same conditions as Table 1, and we show the average across tasks.

How crucial is the threshold?

The usage of a dynamic threshold is an artefact of the online nature of the setup. In preliminary experiments, we did not observe a substantial difference between the dynamic threshold and doing an offline evaluation (i.e., assuming we have the whole dataset, therefore we can order the values and find the ‘best’ threshold values). We present here these results, which are averaged across tasks. The following should be compared with the average column in Table 1. We observe that these gold threshold values only lead to slim improvements, and conclude that dynamic threshold does not result in major performance inefficiencies.

What is the standard size of dataset used to train the routing module?

FrugalGPT do not report how many examples are used for training [1]. In [2], they assume less than 1,000 data points are available per task.

[1] https://arxiv.org/abs/2305.05176

[2] https://arxiv.org/abs/2308.06077

The authors only conducted experiments on short-generation tasks citing the evaluation complexity of long-generation tasks, which leaves a common use-case of LLMs unevaluated.

Short generation is a common use case of LLMs in research and practical settings, and we show that Margin Sampling is effective. We do not expect that it would work for longer text generation, where a more complex and expensive technique may be required. We will incorporate additional results and a discussion on long-form generation in the final version of the paper.

The paper also lacks adequate analysis and assessment of the dynamic thresholding strategy [...] How would the order of samples affect the effectiveness of dynamic thresholding?

The usage of a dynamic threshold is an artefact of the online nature of the setup. In preliminary experiments, we did not observe a substantial difference between the dynamic threshold and doing an offline evaluation (i.e., assuming we have the whole dataset, therefore we can order the values and find the ‘best’ threshold values). We present here these results, which are averaged across tasks. The following should be compared with the average column in Table 1. We observe that these gold threshold values only lead to slim improvements, and conclude that dynamic threshold does not result in major performance inefficiencies.

Regarding how the order of samples affects dynamic thresholding, this is accounted in our experiments (by running on three seeds).

Could you provide the variance or standard deviation for the reported results?

The values of the standard deviation of our experiments in general are much lower than the precision of the results that we report. The reason for this is that we report Area Under the Curve (AUC) across 100 budgets. You can find the values of the standard deviation for Table 1 at https://shorturl.at/W8aty.

This paper is a perfect fit for an industry track, but it might not be of very significant interest to the research community [...]

We want to note that several of the works we compare against have been published in conferences such as ICLR and WSDM; there has been a growing scientific interest in the problem.

The strategies can be viewed as descriptions of practices that worked for a set of pairs and might not be sufficient for a research paper.

We disagree with the statement that the strategies can be viewed as descriptions of practices that worked for a set of pairs. We compared Margin Sampling against Šakota et al. (WSDM 2024), Ding et al. (ICLR 2024) and Chen et al. (ES-FoMo at ICML 2024). In addition, we compare against Yue et al (ICLR 2024) in Table 8. For the training of the auxiliary models, we follow the original papers as much as possible and perform a hyperparameter search where values are omitted.

Both small and large models contain billions of parameters. What would constitute the threshold for categorising a model as small or large?

Regarding what constitutes a ‘small’ or ‘large’ model: model size has been dramatically increasing in recent years, deeming absolute thresholds as susceptible to becoming outdated. Instead, we opt to refer to ‘small’ or ‘large’ in purely relative terms, as implied in 3.1 Problem Definition. The practitioner would have access to two models of different costs, and in this paper we use ‘small’ as a synonym for ‘cheaper’. Since we don’t want to assume any size, we study a variety of parameter sizes for models (7b - Mistral, 13b - Llama, and GPT-3, rumoured to be 175b) and do ablations with multiple relative cost settings (Section 5.3.2).

In the multi-task setup, we observe that the same conclusions hold when the supervised methods are trained with 5,000 datapoints (Table 7 in the Appendix).

In addition, we have run experiments on OpenbookQA and Wikifact, where the supervised methods have been trained with 5,000 datapoints. The trend, as expected, is that supervised methods improve the performance, but Margin Sampling still has a relevant performance.

Mistral 7B - Mixtral 8x7B

Llama-2 13B - Llama-2 70B

GPT-3 - GPT-4

What is the maximum number of examples that can be used for training the rankers?

There is no limit to the number of examples that can be used for training the rankers. However, our setup implicitly assumes a low-resource setting, i.e. if a practitioner has enough data, then they may directly prefer using a task-specific model like a classifier rather than using an expensive off-the-shelf LLM. In this spirit, the original papers that introduced the supervised models trained them on limited data [1-3]. For instance, [1] assumes less than 1,000 data points are available per task.

While we have not explicitly studied the effects of context length, it must be noted that some tasks are generally longer. For instance, bAbI has around 250 input tokens per datapoint. The cost of Margin Sampling is independent of context length, so it would not be more expensive for longer contexts.

[1] https://arxiv.org/abs/2308.06077

[2] https://openreview.net/forum?id=02f3mUtqnM

[3] https://arxiv.org/abs/2305.05176