Cost-efficient Knowledge-based Question Answering with Large Language Models

We would like to sincerely express our gratitude for your encouraging support and strong recognition, especially for your emphasis of our contribution to the community in the weakness again. We will carefully revise the final version following your suggestions.

Response to weaknesses

• W1: Clarity. Thanks for pointing this out, we will further clarify the setting in the revised version. In Section 2.2 Performance Evaluation, we introduced two metrics 'calls' and 'API fees', where calls (times) are used for evaluating our performance based on open-source LLMs and API fees (dollars$) are for GPT series. In auto-ML research, small models are considered to be free since they are far cheaper than big models like LLMs. Following your suggestions, we have also considered and quantified the cost of local KGMs and local LLMs through cloud service fee in dollars. Comparisons have been made over three domain-specific datasets uniformly with the API fees of GPT series, where in this case, our evaluation is more convincing now. cloud service fee: calculates the token-level cost based on basic requirements of GPU resources in one cloud server, instantiated by AWS g4dn.xlarge and p3.8xlarge with 0.526 USD and 12.24 USD per hour.

we have included two more metrics Inference Latency and Cost Advantage to comprehensively evaluate our performance. This also showcases that our evaluation can be general and followers can easily adapt to their own domain with specific evaluation metrics. We show the new results on three benchmark datasets as follows. Inference Latency (s): the time used for making predictions in seconds. Cost Advantage (%): used in HybridLLM-ICLR'24, as the percentage of questions answered by small models, while this metric is widely adopted for automatic ML.

• W2: Significance. We appreciate your expertise and believe this could help us further increase the impact within the community. To address your concerns, we would like to show our generalizability on two open-ended QA datasets, in addition to three original domain-specific multi-choice benchmarks.

Response to questions and suggestions

• Q1: Cost of KGMs. Yes, as explained previously, small models are considered to be free in auto-ML since they are far cheaper than big models like LLMs. Following your suggestions, we have also considered and quantified the cost of local KGMs and local LLMs through cloud service fee in dollars$.
• Q2: Figure 3 clarification. Yes, three subfigures are drawn for CSQA, OBQA and MedQA respectively.
• Q3: Figure 4 clarification. Yes, axis-Y represents for the intervals of selection times. This case study observes the selection as k goes, which showcases the ability of our framework to balance exploration and exploitation (color changes from shallow->deep or deep->shallow). The datasets are for CSQA, OBQA and MedQA respectively.
• S1&S2: Typos. Thanks for your carefulness, we will fix the typos in the final version.

We gratefully thank you for your constructive comments which we believe will absolutely improve the quality of our paper. We also wish to invite you to check our new results with a more comprehensive consideration of evaluation metrics, cost of KGMs and generalizability.

Response to weaknesses

• W1: Evaluation metrics. Thanks for your constructive comments, we are inspired and have included two more metrics Inference Latency and Cost Advantage to address your concerns. Indeed, in our submission, we have already included two metrics 'calls' and 'API fees' , where calls are used for evaluating open-source LLMs and API fees are for GPT series. We show the new results on three benchmark datasets as follows. Inference Latency (s): the time used for making predictions in seconds. Cost Advantage (%): as the percentage of questions answered by small models, while this metric is widely adopted for automatic ML and used in HybridLLM-ICLR'24.

• W2: Cost of KGMs Thanks very much for the inspiration, following your suggestions, we have also considered and quantified the cost of local KGMs and local LLMs through cloud service fee in dollars. Comparisons have been made over three domain-specific datasets uniformly with the API fees of GPT series, where in this case, our evaluation is more convincing now. For details, please check the previous weakness for new results. cloud service fee: calculates the token-level cost based on basic requirements of GPU resources in one cloud server, instantiated by AWS g4dn.xlarge and p3.8xlarge with USD 0.526 and USD12.24 per hour.
• W3: Generalizability Thanks for your constructive comments. We believe considering generalizability will definitely enhance the soundness of our paper and enlarge the impact to the community. To address your concerns, we would like to show our generalizability on two open-ended QA datasets, in addition to three original domain-specific multi-choice benchmarks. In this pipeline of setting, the models are not limited to providing answers with choices or under any pre-defined formats.

We would like to gratefully thank you for your strong support! We also cherish your expertise for plenty of professional suggestions. It is encouraging to be acknowledged by experts from the community. We believe your insightful comments will greatly help us to enrich the experiments and further demonstrate our contributions.

Following your suggestions, all the newly demonstrated algorithms and experiments will be added to the final revised version.

Response to weaknesses

• W1: Algorithms and pseudo-codes. Thanks for your constructive suggestions. We have provided four pseudo-codes in the PDF of the general rebuttal, which demonstrates the training and inference stage of our main framework, as well as the cluster-based Thompson Sampling and contextual MAB. Please kindly check the details in the PDF.
• W2: Base model combinations. We appreciate your insightful comments on a meaningful ablation study. We have done some preliminary trials during our experiments. Some intuitive results are shown hereunder over CSQA dataset. We observe that, both the quality and the number lay an indispensible influence on the final decision. As a fast-adaptive framework, we can readily include more advanced and cheaper base models to move the Pareto frontier for KGQA considering both higher accuracy and lower costs.

• W3: Missing references. Thanks for your carefulness, we have properly added citations for both LLMs.
• W4: Heuristic Baselines. Thanks for your guidance, we have implemented the heuristic methods on all three domain-specific datasets. For epsilon-greedy, we have set the threshold as '0.7'.

Response to questions

• Q1: Migration Study from other domains. Thank you for providing these two references. We will properly cite them in the related work section. For FrugalGPT, we set the threshold for switching KGMs to LLMs as '0.85' after necessary hyperparameter tuning.

• Q2: Base model combinations. Thanks for your valuable question. Yes, we have provided our observations and preliminary results in the aforementioned rebuttal.

Thanks for your recognition of our contributions to a very important and practical problem. We value your insightful comments and will carefully revise the final version following your suggestions.

Response to weaknesses

• W1: Implementation details. Thanks for raising these points. In the original main result, we use HamQA, GPT 3.5 and GPT-4 as KGMs and LLMs.

• Framework training. We provided two pseudo-codes in the PDF of general response to demonstrate the process. The optimization is under a Bayesian online learning framework where our model keeps learning with new queries before using up the budget and continuously updates the parameters based on historical observations. This enables us to be generalizable to any unknown scenarios.
• KGM pretraining. We pretrain the base KGMs based on the training dataset. We carefully checked and ensured the data distributions among train, dev and test are identical. The pre-trained KGMs are directly leveraged for inference without further training.
• Cost functions. We do not necessarily utilize a cost function since our model is optimized under Bayesian online learning. More generalizably, after defining the parameter prior, we calculated the posterior probability according to the feedback from LLMs. Instead of minimizing the cost function, we aim to optimize the expectation function and update the parameters including: 1. posterior distribution Beta($\alpha^{k−1}_{c}$, $\beta^{k−1}_{c}$) for clustered-Thompson Sampling 2. $\mu^{k−1}_{a}$ and $\eta^{k−1}_{a}$ for contextual MAB.

• W2-1: Mechanism of Thompson Sampling. Thank you for the comment. We have added two pseudo-codes in the PDF of the general response to demonstrate: 1. clustered Thompson Sampling 2. contextual MAB.

• W2-2: Evaluation metrics. Thanks for your constructive comments, we are inspired to have two more metrics Inference Latency and Cost Advantage. We also quantified the costs of KGMs and local LLMs with cloud service fee to address your concerns. In our submission, we have already included two metrics 'calls' and 'API fees', where calls are used for open-source LLMs and API fees are for GPT series. This also showcases that our evaluation can be general and easily adapted to various metrics. We show the new results on three benchmark datasets as follows. Inference Latency (s): the time span between question input and prediction output in seconds. Cost Advantage (%): used in HybridLLM-ICLR'24, as the percentage of questions answered by small models. cloud service fee: calculates the token-level cost based on basic requirements of GPU resources in cloud servers, instantiated by AWS g4dn.xlarge and p3.8xlarge with USD 0.526 and USD12.24 per hour.

• W3: Performance Improvements. Thank you for initiating the discussion. First, cost savings between 10% and 20% is already satisfying industrial applications. For example, a typical intelligent customer service in large e-commercial company will handle around 3M tokens a day, after applying our framework, the cost saving can be remarkably around 6600 USD$ a year. Second, in the table for W2, we showcase the performance of using cheaper LLMs (Llama3 7b) with GPT3.5 and GPT-4. When we adopted Llama3 to replace KGMs on MedQA, the performance was boosted with over 41.92% savings and 3.26% ACC improvements.

• W4: Importance of KGMs. We are grateful and excited to claim that this limitation is no longer constraining our paper after being guided by all reviewers. First, this limitation can be easily solved since we are a fast-adaptive and pluggable framework. We can readily involve advanced KGMs given prosperous community to achieve higher performance. Second, this limitation can also be solved by using cheaper local LLMs than KGMs. Evaluated by cloud service fees, the results have been provided in the previous rebuttal, and we can achieve 41.92% cost savings while improving the accuracy around 3.26% on MedQA.