Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness

Thanks for your constructive comments, please refer to the Response to all reviewers for the two frequently asked questions. We also provide the answer to your other question:

Question: Several descriptions are ambiguous. For instance, what does "counts" in figure 5 refer to? Does mean the number of samples for each trial's dataset or is it the number of trails? Which dataset was Table 4 based on?

Answer:" counts" in figure 5 refers to the number of trails. Table 4 is base on the TriviaQA and Summarize-from-feedback two datasets.

Question: Some statements might be inaccurate. For instance, the statement about "AUROC represents the likelihood..." seems off. The correct logic should perhaps be about a "certainty score" rather than an "uncertainty score".

Answer: AUROC represents the likelihood that a classifier will rank a randomly chosen positive instance higher than a randomly chosen negative one, assuming 'positive' ranks higher than 'negative'. Usually, we name it as "confidence score" instead of "certainty score".

Question: Some descriptions are hard to understand: e.g., the sentence "Note here we only modify the prompt used to sample varied responses for computing the observed consistency, not the prompt originally given to produce the original reference response" in Sec 3.1 "Producing Diverse Output". How do you modify the prompt?

Answer: we only modify the prompt, e.g., adding CoT prompt to generate the observed consistency. But for the reference answer, we do not modify any prompt, same with the user provided.

Question: For the section titled "Sec 6.1 dataset", how exactly was the "we also manually validated the accuracy of LLM-generated responses" conducted?

Answer: "how exactly was the "we also manually validated the accuracy of LLM-generated responses"" means we do some human check of the LLM-generated answer. Since the correct answers provided by the datasets do not entail all valid solutions.

Question: In the last paragraph of "Sec 3.2": If you ask LLM, "Are you really sure the proposed answer is correct? Choose again", will this prompt introduce some bias? For instance, GPT-4 often exhibits sycophantic behavior[2]. If you continually question it on an issue, even if its response is correct, it might change it to an incorrect one. How do you address this issue?

Answer: Asking a Large Language Model (LLM) like GPT-4 to reconsider its answer with a prompt like "Are you really sure the proposed answer is correct? Choose again" can indeed introduce certain biases or lead to unintended consequences in the model's responses. Ensuring neutral phrasing and seeking elaboration (for example we also prompt LLM to output the explanation) can help maintain the consistency and reliability of the model's outputs.

Thanks for your constructive comments, please refer to the Response to all reviewers for the two frequently asked questions. We also provide the answer to your other question:

Question: Letting ChatGPT explain itself seems quite straightforward. Since ChatGPT is trained with RLHF, it rarely outputs uncertain answers in ChatGPT's opinion. I feel like training a surrogate model to make the answer judgment is more intuitive.

Answer: It's true that ChatGPT's training through RLHF does typically lead to responses that seem certain in the model's 'opinion.' However, it's important to remember that this certainty does not always equate to factual accuracy, and the model can occasionally produce incorrect or 'hallucinated' answers. The idea of training a surrogate model for answer judgment is indeed an intuitive approach. It could provide an additional layer of evaluation, potentially enhancing the reliability of the responses. However, the development of such a model would involve complex challenges, particularly in creating a well-curated and extensive training dataset that accurately represents a wide range of possible responses and their corresponding judgments.

Question: Regarding to various LLM.

Answer: We acknowledge that our testing was limited to OpenAI's products. This is primarily because OpenAI's product are currently the most extensively used. We attempted to obtain access to other closed-source APIs like Bard from Google, but found that Bard's API is not as widely available to public, only open to limited number of users. Regarding open-source model like LLama, they typically return the probability for each token, allowing for different methods to quantify uncertainty, as mentioned by reviewer [1]. Our approach is designed for more general settings without token-probabilities available (closed APIs, chains/agents and other text->text mappings that are not merely one LLM input->output). Note that the experiments with GPT3 show our method significantly outperforms relying on the token-probabilities to quantify LLM uncertainty.

[1]. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023.

Question: Other than AUROC, it would be also good to show the accuracy of predictions, which can give a basic sense of the confidence score.

Answer: We have already shown the accuracy of predictions in the table 2

Question: Hyperparameters chosen in our method

Answer: Sorry for the typo, $\omega_2$ is $\beta$. The chosen of $\alpha$ and $\beta$ is from the grid search in [0, 1], the chosen of temperature for $T = 1.0$ is simply that is the largest allowed temperature value in the API.

Question: Could you give a formal definition of confidence selection and random selection in Sec. 6.3 with more details?

Answer: If we have 10 prediction, Confidence selection means selecting prediction where the confidence score (return by our method) exceeds a predetermined threshold. Random Selection, on the other hand, is a process where choices are made randomly, without any specific criteria or pattern

Thanks for your constructive comments, please refer to the Response to all reviewers for the two frequently asked questions. We also provide the answer to your other question:

Question: CoT is necessary to see in all relevant results and even for uncertainty quantification for a fair comparison.

Answer: We include Chain of Thought (CoT) prompting for answer generation. For each question, we use CoT prompting to generate 10 candidate answers. Each of these answers is accompanied by a confidence score, allowing us to select the response with the highest confidence. Here are the results.

Our method is better than CoT prompting. It's important to clarify that the dataset utilized in our study differs from the one in https://github.com/FranxYao/chain-of-thought-hub. For instance, while their GSM8k dataset contains 8,000 samples, we employed a version of GSM8k with only 1,318 samples as referenced in the CoT paper (https://arxiv.org/abs/2201.11903). This distinction in dataset size and composition is a key factor why our results are not directly comparable to those found in https://github.com/FranxYao/chain-of-thought-hub.

Question: Hyperparameters chosen in our method

Answer: Sorry for the typo, $\omega_2$ is $\beta$. The chosen of $\alpha$ and $\beta$ is from the grid search in [0, 1], the chosen of temperature for $T = 1.0$ is simply that is the largest allowed temperature value in the API.

On the academic front, the central issue in instruction tuning lies in being data-centric, focusing on how to effectively utilize high-quality data to fine-tune the large language model. We utilize our method to assign confidence scores for the Alpaca dataset (https://huggingface.co/datasets/tatsu-lab/alpaca). This enables us to assign a confidence rating to each output for a given input, as demonstrated by the following example:

[Input] None

[Instruction] Suggest ways to reduce plastic waste in everyday life.

[Response] One way to reduce plastic waste in everyday life is to make the switch to reusable items. This includes bringing a reusable water bottle instead of buying bottled water, investing in a reusable coffee cup, bringing your own bags when shopping, and using reusable containers for storing and transporting food. You can also reduce the amount of plastic waste you create by avoiding or limiting the use of take-out or single-use containers, utensils, and straws. Additionally, try to find sustainable alternatives to plastic items when shopping and avoid purchasing items wrapped in plastic whenever possible. Finally, make sure you recycle any plastic items you do have.

[Confidence for Response]: 0.9393

Here is another example:

[Input] Banana

[Instruction] Classify the item as either animal or vegetable.

[Response] Animal: No, it’s a vegetable.

[confidence for Response] 0.1099

Our method distinguishes between high-quality and low-quality examples in the original datasets. By filtering out the low confidence examples and using only the high confidence ones, we fine-tune the Llama model. Here are the results of the fine-tuned LLama model:

We selectively fine-tune using only high-confidence data, removing any low-quality information. As demonstrated, this approach yields performance on par with or even superior to that achieved with the complete original alpaca dataset, as evaluated by the alpaca-eval metric.

To conclude, we highlight several applications of our method for both industry and academic sides beyond its use in question-answer generation tasks.

Question: Relationship to related work and novelty is missing.

Answer: We want to point out here it is very challenge to benchmark some LLM baselines. This difficulty often arises due to the extensive use of meticulously crafted prompt engineering in some papers. Certainly, we will draw comparisons with Chain of Thought (CoT) prompting as we consider to be highly generalized methods that can be applied to a wide range of scenarios.

Our goal is not necessarily to outperform some baselines, as prompt engineering can significantly impact results. Instead, our goal is to develop a general-purpose pipeline across as many tasks as possible, as the first question answering. We strive to minimize task-specific, ad-hoc processes in favor of creating simple, yet effective methods.

But from the perspective of paper writing, we will discuss more about the related work in the revised version like reviewers suggested. For example, [1] prompts LLMs to generate both an answer and a level of confidence, but it is required to train additional models via supervised learning, unlike our method which does not employ any additional training. [2] introduces semantic entropy, incorporating linguistic invariances created by shared meanings. However their approach requires access to token-level probabilities from the LLM, which is often not accessible with today’s black-box APIs.

[1] Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022a. [2] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664, 2023.