FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

Thank you for your helpful feedback and support for the paper!

Thank you for your helpful feedback and support for the paper! We answer your questions below.

[Justification for the chosen datasets]: We chose the datasets because they represented an important subset of real-world applications. For example, the HEADLINES and OVERRULING datasets represent the task of text classification and tagging, which is listed as an important application of GPT-4 by OPENAI (see, e.g., https://platform.openai.com/examples/default-tweet-classifier and https://platform.openai.com/examples/default-review-classifier). The SCIQ dataset measures the scientific question answering ability, which is key for QA applications such as the Khan academy pilot program (see https://openai.com/gpt-4). We have also added a discussion on this in the revised draft (see Section D in the appendix).

[Evaluations are short outputs]: Tasks with short outputs often come with ground-truth labels that enable objective evaluations. Many LLM benchmarks and evaluations rely on short outputs or multiple choice questions. Quality evaluation for long generations, on the other hand, often involves LLM as a judger, with potential biases and subjectiveness. Thus, we focus on the former to ensure the evaluation is objective and matches human judgments. We have updated the revised draft to make this clear (see Section D in the Appendix).

[Are any datasets in the training set of some models]: Data contamination is indeed important. Without access to the training datasets and documents of black-box commercial LLM services, however, it is challenging to definitively detect if a data point has been seen during training.

[What’s the reason of a 2-d output for training the judge as a regression problem]: This is because there is no numerical label for quality, and thus we cannot use the standard regression training paradigm directly. Instead, we train a model to predict correctness, which is binary and can be curated from the ground-truth labels. Then we use the model’s confidence as the judge’s quality estimation. We have added a detailed discussion in the revision (Section D in the appendix).

[Compare with sampling outputs from the same LLM multiple times, and ensembling their answers]: We have conducted an additional experiment for the model ensembles. In particular, we query ChatGPT 3 times with different temperatures (0, 0.1, 0.2) and then take the mode of the generation to be the final output. As shown in Table 5 in the appendix, sampling multiple times leads to marginal performance gains on the HEADLINES dataset. Yet, the gain is relatively smaller than FrugalGPT, whose accuracy can be 87% at the same cost. We have added this in the appendix (Section B) too.

[Why the leftmost dot of FrugalGPT is still better than the cheap models in Figure 1(d)]: This is because the leftmost dot of FrugalGPT actually used a bit more budget than the cheapest model. In fact, it routed 2% of queries to ChatGPT. This leads to an accuracy gain of 0.7% (=78.7%-78%).

[It would be interesting to study and evaluate how FrugalGPT performs when there is distribution shift on the input text]: We conducted new experiments for distribution shift on the input text. Specifically, we asked ChatGPT to rephrase each question without changing its meaning. On the HEADLINES dataset, we observe that FrugalGPT is quite robust to such shifts. For example, GPT-4 achieves 84% overall performance on the shifted dataset, and FrugalGPT can reach the same accuracy with only 33% of the cost. We have added this analysis in the revised draft (see Section B.3 in the Appendix).

Thank you again for your time! We hope you would consider increasing your score if we have answered your questions. Please let us know if you have additional questions and we are happy to follow up. Thanks!

Thank you very much for your time and feedback.

[The benefit of FrugalGPT over simple methods such as fine-tuning]: Fine-tuning requires more data. For example, 127k data points are needed to fine-tune a ROBERTa model on COQA [R1], while only a few hundred samples are needed for FrugalGPT. In addition, FrugalGPT is complementary to fine-tuning an individual LLM since we can wrap FrugalGPT on top of the fine-tuned LLM and other LLMs to reduce cost.

[R1], I. Staliūnaitė et al, Compositional and Lexical Semantics in RoBERTa, BERT and DistilBERT: A Case Study on CoQA, 2020.

[Experiments are mainly conducted on QA and text classification tasks]: These tasks are selected partially because they are relatively easy to evaluate, which is why these datasets are commonly used to evaluate LLM research. In addition, several of our datasets are quite challenging. For example, on the COQA and SCIQ datasets, even GPT-4 is only able to achieve 40%-70% accuracy.

[FrugalGPT requires a training set to train the judge and the permutation function, and need extra information to compute MPI]: FrugalGPT is designed for applications involving a large number of LLM queries. Consider, for example, Yabble (https://openai.com/customer-stories/yabble), which analyzes customer posts for sentiment, or, Jasper (https://www.jasper.ai/the-prompt/gpt4-future-of-gen-ai-business), which offers chatbots for varying domains. Here, as in many other industry applications of LLM, many tens of thousands of new queries per day are expected. In these cases, the cost of tuning FrugalGPT on a few thousand examples is well worth the reduction in cost when applying it to millions over time. We also conducted an additional study on the effects of FrugalGPT training set size in the HEADLINES dataset. Specifically, we varied the data used to train FrugalGPT from 500 to 5000, and measured the accuracy of the FrugalGPT LLM cascade on the test set with 5000 samples. As shown in the following table, FrugalGPT’s performance was consistently better than GPT-4 (85.5%). We have added this in the revised paper (see Section B.5 in the Appendix).

[From my understanding, calling API is most common when there are only few training examples, or the task is complex]: Many enterprises are interested in using LLMs on a large number of relatively repetitive tasks. For example, financial companies such as Stripe are using LLMs to perform large scale fraud detection (https://www.techcircle.in/2023/03/16/11-companies-using-gpt-4-in-consumer-products), and Yabble analyzes customer posts for sentiment (https://openai.com/customer-stories/yabble), which is similar to our evaluation datasets AGNEWS. Insurance companies are also using LLMs to process millions of claims. Also, FrugalGPT requires fewer labeled data points than fine tuning a model for the task would (since it only needs to train a router that can still use the big model).

[Is FrugalGPT a task-specific or task-agnostic method]: The FrugalGPT paradigm is task-agnostic, but different tasks do require additional training for better performance. For example, our experiments show the cost savings due to FrugalGPT in diverse tasks including price prediction, reading comprehension and news classification.

[What's the cost of estimating MPI]: MPI was estimated based on each API’s performance on the training partitions. FrugalGPT was designed for applications requiring numerous and continuous queries to LLMs. In such applications, the cost of estimating MPI is often small (less than 5%) compared to processing queries during deployment.

[What data is used to train the DistilBERT judge]: The judge was trained on the training partition of each evaluation dataset. The judge training is data efficient: a few hundred examples are enough to obtain a high quality judge.

***[Is it possible to use the DistilBERT judge directly for inference for classification tasks]: Yes. One can use the judge to process a query with each possible label and pick the label with the maximum score returned by the judge. However, this performance is much worse than FrugalGPT. For example, on the HEADLINES dataset, directly using the judge for inference gives an accuracy of 60% performance compared to the FrugalGPT’s accuracy of 87%. This is because judging is easier than prediction. For example, for a given query, the judge may assign low scores to all possible labels. In this case, FrugalGPT would defer the query to the next LLM, which may give a high-quality answer. Since all assigned scores are low, directly using the judge for labeling can give poor predictions.

Thank you for your helpful feedback and support of the paper! We answer your questions as follows.

[How much does the search space pruning help in practice, versus just ordering the APIs in increasing order of cost?]: Pruning does help identify the most effective set of LLMs to query. As an ablation study, we compare the performance of FrugalGPT and that of using three APIs in increasing order of cost: GPT-J, GPT-Neo, and GPT-4. We chose these three APIs because they roughly lie on the surface of the convex hull of all APIs in Figure 1. As shown below, pruning (FrugalGPT) brings additional 1-2% performance improvements compared to using a fixed ordering of APIs in increasing cost.

[How much changing the threshold T affected results? Is the method very sensitive to T?]: The threshold T does affect the performance, as it controls when a query is routed to an API. Fortunately, practitioners do not need to tune it themselves: the threshold is learned automatically during FrugalGPT’s training process (the optimization problem on the bottom of page 4).

[Writing a discussion on the constraints of the design space]: Thank you for the helpful suggestion! We have added a short discussion in the revised draft (see Section D in the appendix).

[If APIs give probability/logit outputs, is a scorer still necessary?]: The scorer is a flexible module to measure the quality of a generation; it does not necessarily need to be the particular form we used. If the APIs give probability/logit outputs that truly capture the quality of their generations, then leveraging these signals can certainly simplify the scorer design. However, in practice, the API’s probability and logit are often not calibrated, which is why having a scorer helps.

[Do you believe that additionally optimizing for this can result in similar improvements to latency and an improvement in cost (ie, either replacing cost budget with latency budget or combining the two)?]: Yes. Replacing the cost budget cost by the latency budget should work. For latency-critical applications, one can set up a small latency budget and always call the API with the fastest response rate.