Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

We appreciate reviewer bmdf's constructive feedback. We are glad the reviewer finds the topic important, and the evaluation systematic and extensive. Here we answer all the questions and hope they can address the concerns.

C1: LLMs are known to be "Fluent but non-factual" or "Convincing though wrong". Given that this study focuses only on evaluation I would have expected more novel findings.

Firstly, we concur with your observation that LLMs can be "Fluent but non-factual" or "Convincing though wrong". However, we would like to clarify that uncertainty and factuality, despite their similarity in concept, measure the model's two different aspects–a model can exhibit both non-factual outputs and provide well-calibrated uncertainty estimates. While the aspect of factuality (e.g., hallucination) has been extensively studied, the uncertainty in LLMs remains underexplored. This leads to the critical, yet unresolved question: can LLMs accurately express uncertainty? Our work is indeed motivated from here, and we aim to delve further into: 1) Can we utilize better prompt strategies to elicit confidence accurately? 2) What alternative methods could better estimate uncertainty? How is their performance? 3) Can we recommend some stable combination algorithms for practitioners that can generalize?

To answer these questions, we have first proposed an evaluation framework to analyze which prompting, sampling, and aggregation strategies can effectively mitigate overconfidence. Following the reviewer's suggestions, we have included a discussion in Sec 6 of our paper (highlighted in blue text) that offers recommendations for practitioners to elicit the model’s confidence, which we believe are valuable contributions from our empirical analysis. We kindly refer the reviewer to the paper for more information.

While some of our findings may not seem surprising, we believe our research, as the first research work to extensively study this problem, can provide a foundation for further study and call for more attention to the uncertainty of the model's prediction in addition to factuality.

C2: Larger LLMs are better (on all benchmarks) than smaller ones is also general knowledge.

Thank you for the valuable feedback. We agree with you that larger LLMs perform better on a wide variety of tasks. However, for the task of calibration, it has been found that larger models are not always well-calibrated: for example, GPT4 fine-tuned on RLHF indeed hurts the calibration performance[1]. Furthermore, in a black-box setting, particularly when the model is not trained to output verbalized confidence, it still requires experiments to confirm whether this common knowledge can be generalized.

[1] OpenAI, R. (2023). GPT-4 technical report. arXiv, 2303-08774.

C3: It would be interesting (and maybe even necessary) to study how much the uncertainty estimates depends on the prompt, i.e., if you state in the prompt "Appear confident" are estimates more overconfident?

Thank you for the interesting idea! Inspired by the reviewer's suggestion, we have explored how prompts like "You are a confident GPT" and "You are a cautious GPT" affect confidence elicitation performance and found small differences in accuracy and uncertainty metrics, as shown below. In addition, we also visualize the plots of the output confidence distribution and find that they are quite similar. This suggests that this type of variation in prompt has a limited impact on the performance compared to the prompting strategies studied in Sec 3.2. The visualization plots, implementation details and more analysis can be found in Appendix Sec B.1.

Additionally, we would like to share an interesting experiment we previously conducted for the misleading sampling strategy in the original submission (see Appendix Sec B.5). We append hint texts with varying degrees of certainty (Table 8) to the prompts, such as strong claim: "I am 100% confident that the answer should be xx", weak claim: "I vaguely remember the answer is" or the external source: "The Wikipedia says the answer should be xx”. Then we measure the uncertainty by computing how likely the model will change their original prediction. Interestingly, our experiments showed that prompts with weaker confidence descriptions (weak claims) often yielded better results than those with strong claims or external course hints. Details are provided in our paper's appendix Sec B.5.

We sincerely thank reviewer uQRx for the constructive suggestion on the white-box comparison experiment. We are glad that the reviewer finds the paper well-written, the finding meaningful, and the evaluation thorough. Here we first clarify the questions and update later with experiment results to address the concerns.

W1:Compare with white-box based calibration approaches such as token probability-based calibration approach on open-source models and observe how large the gap can be on ECE and failure prediction?

Thanks for this constructive suggestion! We are conducting experiments comparing black-box and white-box calibration approaches now and will update the results soon.

Q2: Equation 3. Shouldn't we maximize with respect to P since we are doing an MLE estimation?

Thank you for pointing out the discrepancy in Equation 3. You're correct; it should maximize with respect to P. This has been corrected in the revised manuscript.

Q: Can the authors propose the best way for practitioners while also noting the limitations?

Thanks for the constructive suggestions. We have added a discussion and conclusion on the best way for practitioners into Section 6 of the paper. Specifically, we recommend the practitioners use “Top-K prompt + Self-Random sampling (M=5) + Avg-Conf/Pair-Rank aggregation” as the best method so far due to their stable performances.

The recommendation is made by balancing efficiency, simplicity, and effectiveness through multiple benchmark datasets:

• Top-K outperforms all other methods on GPT-3.5 and is comparable to the top-performing method Self-Probing on GPT4. Compared to Self-Probing which requires two inference phases, the Top-K prompt is chosen for the balance between effectiveness and efficiency.
• As shown in Sec 5.3, ensemble methods (e.g., $M=5$) are consistently more effective than verbalized confidence ($M=1$) in eliciting a model's confidence. Regarding the sampling strategies, Self-Random is selected for being more straightforward and commonly used, since the performance difference of different sampling strategies is minimal.
• For aggregation, strategies based on both answers and verbalized confidences (e.g., Avg-Conf and Pair-Rank) outperform aggregation based on answers only (e.g., consistency). Then we recommend Pair-Rank and Avg-Conf for different downstream tasks according to their relatively good performance on different metrics. For example, for tasks that prioritize the exact confidence values, like calculating expected risk, Pair-Rank is recommended, while Avg-Conf is better suited for tasks related to failure prediction, e.g., factual error detection.

We sincerely thank reviewer Bvbz for the constructive suggestion on the discussion of why some strategies work, and why some do not. We are glad that the reviewer finds the question important, the study comprehensive, the paper with clarity, and the findings impactful in practice. Here we answer all the questions and hope they can address the concerns.

Q1: Discussions on why some strategies work, and why some do not work.

Thank you for this constructive suggestion. We have added a dedicated discussion to Appendix C and we highlight the discussion of some effective strategies and advice here:

1. Consistency among multiple responses is more effective compared to verbalized confidence ($M=1$), with particularly notable improvements on the arithmetic task (Sec 5.3).

   • This is because sampling more queries allows us to directly approximate the model's internal distribution, $P\_{model}(\mathbf{x}\_t|\mathbf{x}\_{1:t-1})$, which is trained to mirror the ground truth data distribution.
   • Issues making this method ineffective can be: 1) the model's poor calibration [1], i.e., $P\_{model}(\mathbf{x}\_t|\mathbf{x}\_{1:t-1})$ does not align well with $P\_{data}(\mathbf{x}\_t|\mathbf{x}\_{1:t-1})$; or 2) the computational constraints limiting the number of sampled queries, leading to inaccurate estimates.

2. Aggregation based on answers and verbalized confidences (e.g., Avg-Conf and Pair-Rank) outperforms aggregation based on answers only (e.g., consistency) (Sec 5.4).

   • This advantage is particularly notable when LLM queries are costly and the number of queries we can sample is constrained.
   • This is due to the coarse granularity of the consistency-based aggregation's output—limited to 6 possible values (0, ⅕, ⅖, ⅗, ⅘, 1) when M=5. This can lead to poor calibration performance. The verbalized confidence, despite being less precise, still captures the model's uncertainty tendency and allows for finer-grained output values, and hence can be combined to enhance calibration performance (see Table 4).

3. For verbalized confidence, we note that humans are able to verbalize their uncertainty, e.g., giving insight as to whether our answers and reasonings are correct or not. So it is reasonable to expect LLMs to have also learned this ability, or to learn it at some point in the future.

   • The current suboptimal performance of verbalized confidence points to an important research gap, and this might be explained by the inherent inaccuracy of the training data, particularly human expressions of uncertainty. For example, as studied by [2], humans sometimes tend to exaggerate their a priori probability for an event that has occurred.

4. Compared to Vanilla prompt, Top-K, CoT, and Multi-Step can significantly reduce ECE in ChatGPT.

   • We argue that the improvement is largely due to these prompt strategies enhancing the model's accuracy, which narrows the gap between average confidence and actual accuracy, rather than a significant boost in their ability to differentiate between correct and incorrect samples. This is also supported by the modest gains in AUROC and AUPRC, compared to the significant improvement in ECE.

[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." ICLR (2023).

[2] Garthwaite, Paul H., Joseph B. Kadane, and Anthony O'Hagan. "Statistical methods for eliciting probability distributions." Journal of the American statistical Association (2005)

Q1: Is there a reason to believe that the model's certainty should be correlated with model correctness? Or to understand a priori (e.g., based on training data) when it might be more or less likely to be so?

This is indeed an excellent question! In fact, the correlation depends on the definition of uncertainty, e.g., data/model uncertainty[1]. Here to answer the reviewer’s question, we clarify the following subquestions:

1. What is our definition of model certainty?
2. Does this defined model certainty reflect model correctness?

In line with established research [1,2,3,4,6], we define the 'ground-truth' certainty of a model's prediction $\mathbf{x}\_t$ as the corresponding probability from the data distribution $P_{data}(\mathbf{x}\_t|\mathbf{x}\_{1:t-1})$. This probability is highly correlated with model correctness, that is, a lower probability implies that the prediction is less likely to be correct. Therefore, we believe that the model’s certainty should be correlated with model correctness, which is also practically useful, e.g., help humans identify potential factual errors of LLMs.

3. Can our proposed uncertainty scores capture the model correctness?

We suspect the reviewer's question might also arise from doubts about whether our proposed consistency or verbalized confidence can capture model correctness, e.g., the models can be confident but wrong. We would like to clarify that our methods are designed with the potential to capture correctness, although issues such as computational constraints can affect their efficacy:

• For consistency, sampling more queries theoretically allows us to approximate the model's internal distribution, thus capturing correctness. The issues making this method ineffective can be the model's poor calibration [4] or the computational constraints. For example, the limited number of sampled queries might lead to inaccurate estimates.

• For verbalized confidence, we note that humans are able to verbalize their uncertainty, e.g., giving insight as to whether our answers and reasonings are correct or not. So it is reasonable to expect LLMs to have also learned this ability, or to learn it at some point in the future. LLM’s current inaccuracy at this task points to an important research gap and might be explained by the training data being affected by the observed overconfident tendency of human uncertainty expression[5].

• [1] Malinin, Andrey, and Mark Gales. "Predictive uncertainty estimation via prior networks." NeurIPS (2018).
• [2] Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. "On calibration of modern neural networks." ICML 2017.
• [3] Liu, Jeremiah, et al. "Simple and principled uncertainty estimation with deterministic deep learning via distance awareness." NeurIPS (2020)
• [4] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." ICLR (2023).
• [5] Garthwaite, Paul H., Joseph B. Kadane, and Anthony O'Hagan. "Statistical methods for eliciting probability distributions." Journal of the American statistical Association (2005)
• [6] Jiang, Zhengbao, et al. "How can we know when language models know? on the calibration of language models for question answering." ACL (2021).

Q2: How was the specific prompt text chosen for each of the prompt strategies? was it optimized or experimented with?

Thanks for the question. Indeed, the prompts (see Appendix Sec D for all the used prompts) are selected through iterative refinement to avoid potential ambiguity and bias, and we are happy to share some insights from this refinement process.

Initially, we started with a straightforward prompt like "Provide your confidence in this answer." However, we quickly discovered that the model's responses varied widely, ranging from textual descriptors like "confidence: high," to statements such as "I am very confident," and even disclaimers like "as AI language models, I cannot provide confidence."

In addition, we observed that providing examples in the prompts can insert bias into the distribution of outputs. For instance, saying "ONLY the option letter, e.g., C" would inadvertently increase the frequency of 'C' as a selected answer in weaker models like Vicuna.

Consequently, we refined our prompts further by removing such examples to reduce bias. These improvements were made step by step, informed by careful observation and analysis of the output results. While we cannot guarantee that the current prompts are the optimal versions, they have been designed to minimize the interference of irrelevant factors as much as possible.

We thank the reviewer Gbf5 for recognizing the importance of our work, the elegance of our approach, and the appreciation for our discussion. Here we answer all the questions and hope they can address the concerns.

Weakness: The high-level observations are interesting but difficult to operationalize or build upon.

Thanks for the helpful feedback. To guide practitioners on how to utilize our findings, we have revised the paper to include a dedicated discussion in Section 6.

TL;DR: we highlight simple yet effective strategies and recommend the practitioners use “Top-K prompt + Self-Random sampling + Avg-Conf/Pair-Rank aggregation” as the best method so far due to their stable performance.

The recommendation is made by balancing efficiency, simplicity, and effectiveness through multiple benchmark datasets:

• Top-K outperforms all other methods on GPT-3.5 and is comparable to the top-performing method Self-Probing on GPT4. Compared to Self-Probing which requires two inference phases, the Top-K prompt is chosen for the balance between effectiveness and efficiency.
• As shown in Sec 5.3, ensemble methods (e.g., $M=5$) are consistently more effective than verbalized confidence ($M=1$) in eliciting a model's confidence. Regarding the sampling strategies, Self-Random is selected for being more straightforward and commonly used, since the performance difference of different sampling strategies is minimal.
• For aggregation, strategies based on both answers and verbalized confidences (e.g., Avg-Conf and Pair-Rank) outperform aggregation based on answers only (e.g., consistency). Then we recommend Pair-Rank and Avg-Conf for different downstream tasks according to their relatively good performance on different metrics. For example, for tasks that prioritize the exact confidence values, like calculating expected risk, Pair-Rank is recommended, while Avg-Conf is better suited for tasks related to failure prediction, e.g., factual error detection.

In the revised manuscript, we have incorporated a comparative analysis between black-box methods and their white-box counterparts, e.g., sequence probability. For more results and details, please see the response to reviewer uQRx at (https://openreview.net/forum?id=gjeQKFxFpZ&noteId=h8O7QAxL4y) and Appendix B.1 in the paper.