Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

• The evaluation does not consider actual black/grey-box commercial LLMs such as ChatGPT, while this is (implicitly) what the authors propose their method for. If a black-box target LLM is substantially better than the tool LLM, the black-box based uncertainty estimates may outperform the features that rely on the tool-LLM activation. Results will be more persuasive if the experiment is applied to the most popular target LLMs.
• It may be expected that the uncertainty estimates of this method get substantially worse when the task is different from the supervised learning data, but the authors do not sufficiently explore this. An experiment where the uncertainty model is trained on Question-Answering but evaluated on Multiple-Choice might show when this model fails.
• Authors are effectively comparing a zero-shot uncertainty estimation method against a supervised-learning uncertainty estimation method. This is arguably not a fair comparison. Results would be more complete if authors additionally evaluate a “normal” supervised NLP model that does both the predictions and uncertainty estimates.
• The argument that there is extra information (w.r.t. to the output layer) in the hidden activations in LLMs that is otherwise not the case in normal ML models is interesting, but is missing a thorough investigation. Additionally, I think it has relatively limited relevance to the main portion of the work. I’d consider (if it aligns with the authors interest) to reserve the investigation into the hidden layers of an LLM for a separate dedicated paper, so this paper can maintain focused on one point. The evidence is currently fairly week that this information would exist. It may be that a model is not good at picking it out of the output layer, but that does not mean the information does not exist. The authors also don’t test a comparison in a normal NN. Overall, I think the discussion of hidden activations should not be considered as a core contribution or finding of the current paper.
• Throughout the paper there are no standard deviations reported for the results. Particularly when sometimes the differences between conditions are fairly small it is unclear whether differences are coincidental or significant.

Additional Feedback (minor comments)

• It is unclear whether the AUROC scores in Tables 1 and 2 are for predicting the answer, or for predicting whether an answer is (in)correct. I assume these AUROC scores are for the uncertainty estimation, but it may be good to make this clearer.
• Table 3 should be made clearer by adding in bold the best performances. Additionally, to show whether whitebox features indeed suffer from more performance decrease than black-box features it would be good to add averages at the bottom of the table.
• On Line 38 it appears that the references (Gal & Ghahramani 20216, Lakshminarayanan et al. 2017, Guo et al. 2017, Minderer et al. 2021) claim that “the need for uncertainty estimation [is greater for] generative AI”, but this is not what those papers say. I understand the intention the authors have here (citing papers on UQ for normal ML), but with the current phrasing/placement it gives the wrong impression. Consider rephrasing.
• The (informal) definition of uncertainty estimation on Line 44 defines it as predicting quality, but then says that “quality” is confidence and uncertainty, which is circular reasoning. The “truthfulness” can be a good measure, but that cannot be the only measure of quality. I’d propose to reconsider the argument made here and rephrase.
• There’s a small language mistake in Line 90. “restrict its application to black-box LLMs” means “it can only be used on black-box LLMs”, but I think you mean the opposite.
• In Section 1.1 or Appendix A it might be appropriate to discuss verbalized uncertainty (such as the A4U method) to give a broad overview of existing work.
• Line 196 “The next is to …” -> “The next step is to …”
• There’s a typo in Line 213.
• In Tables 1 and 2 there’s a benchmark called “A4C”, but in the Benchmarks section it is defined as A4U.
• Appendix B.4 seems to be incomplete. The proof is not provided.

• Insufficient uncertainty taxonomy: The paper does not clearly distinguish between different types of uncertainty (epistemic vs. aleatoric), which is crucial for understanding what kind of uncertainty is being estimated and how it should be interpreted.

• Limited theoretical analysis of tool LLM reliability: In the black-box supervised (Bb-S) setting, the method relies on a secondary LLM to estimate uncertainty. The paper lacks a theoretical analysis of when and why this approach works and potential failure modes if the tool LLM itself is uncertain or hallucinating.

• Scope of evaluation: The experiments focus on smaller open-source models (7B-13 B parameters). Discussion on the method's scalability to larger models or different architectures (e.g., GPT-3.5, GPT-4 or even larger LLaMA models) is limited, and this has to be clearly highlighted in the limitations and future work.

• Potential for recursive uncertainty: The paper does not address the potential issue of recursive uncertainty when the tool LLM used for estimation introduces its own uncertainties or errors, especially in the black-box setting.

• Limitations discussion: The limitations section is brief. A more thorough discussion of potential failure modes, edge cases, and the impact of labeled data quality would provide a balanced view of the method's applicability.

1. The proposed model captures total uncertainty defined in eq (1) but does not distinguish between aleatoric (data-based) and epistemic (model-based) uncertainty. Due to this reason, the proposed method is not sensitive to OOD data as shown in section 5.3, which should be detected using epistemic uncertainty.

2. From my understanding, the discussion provided in 4.2 cannot explain how uncertainty quantification (UQ) differs between traditional machine learning models and LLMs. Proposition 4.1 only states that the populational minimizers $f^*$ has the conditional independence structure (or sufficient statistics), but all models are learned from empirical, potentially violating this independence in practice. Actually, there are some existing work using the hidden activations to improve uncertainty estimation, see

Shen, Maohao, Yuheng Bu, Prasanna Sattigeri, Soumya Ghosh, Subhro Das, and Gregory Wornell. "Post-hoc uncertainty learning using a dirichlet meta-model." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 8, pp. 9772-9781. 2023.

Therefore, the difference in the loss function or the presence of conditional independence cannot be the fundamental distinction between uncertainty estimation in traditional ML models and LLMs.

3. The baselines chosen for comparison may be too simplistic or lack relevance for the current problem scope. Please consider comparing with the following two papers in the rebuttal.

Hou, Bairu, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. "Decomposing uncertainty for large language models through input clarification ensembling." ICML 2024.

Shen, Maohao, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory Wornell, and Soumya Ghosh. "Thermometer: Towards Universal Calibration for Large Language Models." ICML 2024.

• The definition of uncertainty that this work assumes and discusses their method with is debatable. The uncertainty producing module, the $g$ function, is completely modular and external to the model it can be used to evaluate. Therefore, the notion of uncertainty is independent of the model, and only dependent on responses (which can come from any model). However, predictive uncertainty is understood as the "uncertainty in making a prediction". As a thought experiment, suppose a language model is given an input $*What's the capital of France*$ and deterministically outputs $*Berlin*$. Since the model output is deterministic, one would perceive this model as being "incorrect with 0 uncertainty". However, the method in this paper would likely mark this response as "incorrect" and thus label this response as being "highly uncertain". It is also debatable which signal is more useful or actionable - "how uncertain the model is in producing a prediction" vs. "the predicted correctness of a response". I believe these conflicting definitions/notions need to be reconciled.
• While the paper presents the method and motivation in a slightly different manner, at its core, I don't see how the presented method is very different from Azaria & Mitchell [1]. Their method is essentially identical to the "white-box" method this paper presents. This paper extends this core idea to the black and grey box setting, but the idea of training an external model with extracted features to predict some label of correctness is identical. [1] frame their problem in the context of hallucination, whereas this paper's framing is w.r.t. uncertainty, but the idea doesn't differ very much.
• Why wasn't [1] included in the baseline methods for empirical evaluations? Suppose you were to include it - how would it differ from the proposed method in terms of implementation?

[1]: Azaria & Mitchell, 2023: "The Internal State of an LLM Knows When It’s Lying"