Large Language Models Must Be Taught to Know What They Don’t Know

Thank you for your thoughtful comments and supportive review!

Here is our response to the weaknesses you listed pointwise:

Response 1

Yes, our method requires access to the model parameters. Luckily for our method, there are incredibly strong open-source LLMs available. The most recent and relevant case is Meta releasing LLaMA 3.1 405B, which has performance on par with the best models from OpenAI and Anthropic. We’re confident that as the abilities of closed-source LLMs progress there will also be parallel progress in open-source LLMs, and therefore any method that requires fine-tuning the model should be possible to run in a fair comparison with state-of-the-art models.

You proposed a comparison with sampling when estimating uncertainties for a different model. We assume that you mean that you want to see a comparison between, for example, the uncertainty estimates of LLaMA-2 applied to Mistral generations and uncertainty estimates from sampling Mistral by itself. We can provide those numbers below:

All models are trained/evaluated on answers generated from Mistral 7B and we show the model used to construct the uncertainty estimate in parentheses (where applicable). If you were instead proposing a different comparison, please let us know. From the table, it’s easy to see that even constructing an uncertainty estimate from a different language model (e.g. LLaMA-2 7B applied to Mistral 7B) significantly outperforms a sampling method applied to the same model.

Response 2

We included bars for Probe and Zero-Shot Classifier in the transfer plots so that they could be used as reference points when evaluating the transfer ECEs and AUROCs. While you’re correct that there are notable differences between the in-distribution and transfer performance of the runed models, these large percent differences are not so significant in the scheme of the broader evaluation. While the ECE numbers increase in transfer and AUROC numbers decrease, the final transfer values are significantly better than the baseline methods applied \it{in-distribution}. Therefore in many cases it might be as practical to apply LoRA fine-tuned models in transfer as applying a worse method to data from the actual downstream task.

Response 3

We’re happy to provide some clarifications. As you point out, one possibility is that the uncertainty estimates are using information present in the question to predict how hard a question will be, independent of the actual answer that was generated. The baseline with ground truth questions and incorrect answers provides a sense of how good such a heuristic could be in practice. As we can see, it does actually perform reasonably well. But it performs significantly worse than when the model is provided with both the ground truth question and a real generated answer which demonstrates that the relationship between the answer and question matters. If the uncertainty estimates were purely using the topic of the question or answer, we would expect both methods to perform roughly the same, because the relationship between the question and answer wouldn’t affect the topic they are associated with. However, we observed that the performance is different across all models.

Question Answers

Here are answers for your list of questions:

• “MMLU (MC)” and “MMLU (OE)” in the x-axis label on the Figure 3 (left) represent the multiple-choice (MC) and open-ended (OE) settings of the massively multitask language understanding (MMLU) dataset. The position of the “MMLU” x-axis labels is suboptimal and we apologize for any confusion. The labels would be better placed above the plots as titles instead of x-axis labels.
• Conveniently we already ran this comparison for the initial submission, and a more detailed explanation can be found in Appendix B.1. While we tried both counting and likelihood accumulation for the sampling methods, we did not find a significant difference in the performance of these variants on our evaluation, though there may well be a difference on other tasks.
• We sample the model to construct correct and incorrect answers. As a result, the learned estimator is specific to the model used to generate the answers and not to the dataset of question-answer pairs. While the question-answer pairs are fixed for a given dataset, different models can have very different rates of correct and incorrect answers, which necessitates model-specific training.
• Thank you for sharing these additional related works. [2] is already referenced in our submission as [59], with citations in both Section 2 and Appendix B.2. We will also add LitCab to our overview and cite it.

We appreciate your feedback. We respond to your comments below, providing numerous clarifications and new results inspired by your comments.

Technical Novelty

We would like to clarify that our goal in the paper was to show that more complex methods, including those with technical complexity, are in fact not necessary when a small number of labeled examples are available. Given the rising popularity of approaches that use zero-shot prompting or sampling, we think this point is clearly important and runs contrary to prior work, making it sufficiently novel. Rather than introduce yet another method with technical complexity, we wished to step back and scientifically understand key facets of calibration for language models. And indeed our experiments provide many novel insights that should not be dismissed. We found that regularization and use of LoRA were critical for good performance. These observations go beyond prior work. We ultimately see use of existing popular tools like LoRA as a strength, facilitating broad adoption.

Extended Experimental Results

Inspired by your comments, we provide results on the MMLU-Pro benchmark, which is a more challenging multi-task language understanding benchmark. This evaluation was created after the release of all models we consider therefore there is no risk of contamination. Tables containing the results are available in our general rebuttal comment. Overall, the new results mirror our original results on MMLU and provide evidence that our estimates do generalize. MMLU is also considered significantly harder than MMLU and frontier models typically have lower accuracy on MMLU-Pro by 20% or more.

Trends in MMLU Performance

While there is variance in each metric, there are clear trends in the relative performance. To make this clear we can look at the win rate of LoRA + Prompt over Zero-Shot Query as an illustrative example. The table below shows these win rates for AUROC and ECE. In AUROC, there is a clear advantage to using LoRA + Prompt for all models. ECE is less consistent, but the majority of variance is the result of the model, not the subset of MMLU:

We found that Mistral 7B in particular was highly anomalous among the models we considered. When we consider llama-3 models, which are already included in Figure 1, most models do not display good zero-shot calibration:

We also include results for Qwen-2 7B-Instruct, a strong model released by Alibaba, to make it even more clear that Mistral 7B is anomalous in its ECE results:

These results have the corresponding win rates:

While some models are surprisingly calibrated zero-shot, these models often have bad AUROCs. This is possible because probabilities can be calibrated without being useful for making predictions. For example, a model can completely ignore the input and still have perfect calibration if the statistics of its predictions match the statistics of the data, even if the predictions are completely random. Ideally we want uncertainty estimates that are both good at discrimination and are calibrated, because such a model can reliably filter incorrect answers from correct answers while also having probabilities that are easy to interpret (because they correspond to real rates of correctness or incorrectness).

Confidence Distribution on MMLU

There are two points worth clarifying. First, our main aim in showing the histogram of confidence scores was to show that unanswerable questions obtain lower confidence under our model. Lower confidence doesn’t necessarily imply better calibration, as correctness cannot be defined for unanswerable questions. Second, MMLU is different from SelfAware because correctness is well-defined for all questions, and therefore we can use AUROC and ECE to evaluate the quality of our uncertainty. Although we do not show the confidence distribution in a plot analogous to Figure 3 (right), the confidence distribution of correct and incorrect answers is already captured in the ECE and AUROC metrics.

Summary of Contributions

Given the widespread adoption of LLMs the significance of this paper’s subject is self-evident. We provide several useful observations that are directly applicable to improving the usability of LLMs for question-answering. These observations are deliberately simple but are novel and contradict statements made in prior work. For instance, we show that fine-tuning with even a small number of labeled examples can dramatically outperform prompting and sampling methods in the majority of cases. We also include an extensive analysis of our method’s robustness to distribution shifts and even study how our uncertainties affect user behavior in a study of human-AI collaboration.

We hope that you will consider raising your score in light of our updated experiments and clarifications.

Thank you for your review! We respond to your comments below.

Connection between Base Model and Uncertainty Estimate

As you note, the model that generates uncertainties is not exactly identical to the model that generates answers. However, the assertion that these models are entirely independent or that our method is a departure from prior work is fundamentally incorrect.

The setup we employ is the standard in every supervised learning method in our related work section. Lin et. al. and Kadavath et. al., for example, are both foundational papers in LLM uncertainty estimation that employ a separate uncertainty model in their finetuning experiments.. Azaria and Mitchell also use a probe, which results in two separate models, one for generating uncertainties and one for generating answers. In fact, only the zero-shot classifier or sampling methods don’t adopt this two-model approach, and as we show in the experiments they tend to have weaker performance.

Even if our setup was not standard in prior work, it would still be inaccurate to characterize the two models as independent. To make this clear, we can review the details of the three methods we explore:

• Probe: The uncertainty model and base model share features, and the MLP has approximately 50K parameters, effectively sharing 99.999% of their parameters.
• LoRA: For approximately 3.5M LoRA weights in a 7B parameter model, and 50K parameters in the MLP classifier, the models share 99.95% of their parameters.
• LoRA + Prompt: This approach only uses LoRA and no new head. Therefore the models share 99.95% of their parameters. For LoRA + Prompt, we also regularize the KL divergence between the base model and uncertainty estimate (Section 5, Table 1), and this regularization makes not just the weights but the outputs of the two models tied together.

Making the two models exactly identical is an exciting avenue for future work. There is no conceptual reason that our method shouldn’t extend to that setting, but in practice it’s challenging to fine-tune state-of-the-art LLMs while avoiding catastrophic forgetting in answer generation will most likely require mixing large batches of graded data with standard pre-training data. Most academic groups are simply ill-equipped to run this experiment, and it doesn’t seem fair to penalize us for having limited compute, especially when our setting aligns with prior work.

Length of Open-Ended Answers

To determine whether open-ended MMLU is sufficiently different from multiple-choice MMLU, we can look at some basic statistics of the generations sampled from each model.

While the generations do not consist of whole paragraphs, as we describe in Section 4 (“Do We Get Good Uncertainties Out of the Box?”), the behavior of probabilities over 10 tokens is fundamentally different from the behavior of probabilities over single tokens because of multiple possible phrasings. We make this distinction clear in Figure 1 left by showing that probabilities over single tokens are excellent discriminators while perplexities over many tokens are not.

Additionally, we run two experiments that should help clarify that our results do in fact extend to long sequences. First, Figures 11-13 in Appendix E show that the uncertainties predicted by our models do not have a consistent relationship with the length of the sequence. Many of the sequences displayed are between 100 and 200 tokens long, which is comparable to a long response in a dialogue with a chat model.

In addition to these results, we also ran an additional experiment to augment the perplexity results in Figure 1. We explore how perplexity relates to quality scores on a dataset with very long graded generations. MT-Bench contains approximately 100 long-form questions with answers in dialogue form. We use precomputed answers graded with GPT-4 for LLaMA-2 7B Chat, LLaMA-2 13B Chat, and LLaMA-2 70B Chat and compute the perplexity associated with each answer.

From the low Spearman’s rhos between the perplexity assigned by the model and the score given by GPT-4, we can see that perplexity is not a reliable predictor of quality, and furthermore, perplexity does not become a more reliable predictor of quality as model sizes and capabilities increase, unlike previous work.

Humanities vs. Social Sciences

While “humanities” and “social sciences” are often synonymous, they do have distinction definitions as super-categories of MMLU. While literature is in the super category “humanities” in MMLU, legal briefs and economics are defined as “social sciences.” It’s not obvious that reading literature should confer an understanding of economics, or vice versa.

Calibration of Unanswerable Questions

Our purpose in displaying the confidence score is to show that the confidence for unanswerable questions is significantly lower than the confidence scores for answerable questions. The ECE of the answerable questions is 0.185 and the AUROC is 0.78. These represent 36% and 15% improvements over the baseline respectively.

Closing Remark

We’ve done our best to address your comments in detail and provide additional experiments where possible. We hope that you will consider raising your score in light of these clarifications and new evidence.

Attributing generalization

Our current experiments are insufficient to claim that the question topic has no influence on the model. What we can conclude, however, is that the uncertainty estimate cannot be using only the question topic, because, if it were, we would not expect the model trained on correct answers to have significantly better performance. Therefore, the calibrated model must be learning something meaningful about correct answers, not simply a short-cut association between the question and correctness.

Interpreting User Study

Indeed, this conclusion is a result of observing the shift in the distribution of correct v/s incorrect answers.

When the confidence scores are generated at random, the user cannot reliably judge whether to seek AI assistance (i.e. use the answer provided by LLM). (c) shows that when the confidence estimates cover a wide range, AI assistance is not very useful as a miscalibrated confidence estimate may just misguide the user to use (or not use) the assistance. Now, comparing (a) and (b) shows that in cases when users did not rely on LLM’s answer, the confidence estimated by the LLM is spread relatively uniformly over the 35-45% range in (a) as compared to more clustered mass around 40% in (b). The confidence scores then provide a more meaningful signal to the user for when to choose AI assistance. Similarly, looking at the distribution of confidence when the users did seek AI assistance, comparing (a) and (b) shows that a higher fraction of the users were able to rely on the LLM.

Most importantly, these results indicate that humans can respond to AI assistance differently depending on uncertainty estimates

Viability of Few-Shot ICL

For clarity, we note that all our evaluations with MMLU were conducted with the standard 5-shot prompting.

One key difference to the calibration considered in the paper you referenced (with GPT-3) is our use of open-ended answers instead of multiple-choice answers. In Figure 1, the probabilities assigned to multiple-choice options are often useful predictors of correctness. However, because open-ended answers comprise multiple tokens, using model probabilities in open-ended settings is more fraught.

Kadavath et al. show that many models similar to GPT-3 can have reasonable calibration when prompted in the correct way. Unlike using the probability of answer choices, these prompting methods are applicable to open-ended generations. Notably, we show that fine-tuning performs significantly better in both multiple-choice and open-ended settings.

You allude to the possibility of including demonstrations of the uncertainty prompts (e.g. “Is the proposed answer correct or not?”) within context. For the verbal elicitation baseline, we use a similar approach to bias the model towards correctly formatted outputs. We include a prompt (Appendix B.2) that gives an example of outputting a calibrated probability. As we’ve highlighted before, this approach can perform surprisingly well for a black-box method but dramatically underperforms fine-tuning methods.

Challenges of Online Learning

You’re correct–the training data is stationary. We rely on a one-time labeling to learn the correctness classifier.

The key challenge in simultaneously pre-training and learning the classifier is that as the model learns, the correctness labels may change. This non-stationarity in labels will destabilize learning, as the label can be a moving target. However, we believe it is feasible to incorporate a correctness classifier more stably similar to how LLMs are aligned with multiple rounds of data generation (e.g. LLaMA 3.1), and is a promising research direction to pursue.

We’re glad that you find the paper informative and appreciate your feedback. Thank you for sharing the additional reference, which we will incorporate into the related work section. We respond to your questions below, and make several clarifications. We also provide additional results in the general post which you might find helpful. We value your support and would appreciate it if you would consider increasing your score in light of our response and the timeliness and comprehensiveness of this work.

Differences from Kadavath et. al.

Kadavath et al. is closely related to our work, but a few key details of our evaluations and conclusions differ. From the top, our key emphasis is on answers with more than one token (open-ended versus multiple-choice). We explore how open-ended answers can have fundamentally different behavior from multiple choice answers because of ambiguity in phrasing. Unlike Kadavath et al., we show that prompting strategies (e.g. zero-shot classifier, verbal elicitation) are often relatively weak when compared to fine-tuning methods. These methods often improve very slowly, if at all, with the strength of the underlying model. By contrast, fine-tuning methods are effective even for relatively small models and their performance improves with the power of the underlying model.

Kadavath et al. also includes experiments with fine-tuning a model on graded examples, but they are different from our own in several ways. Kadavath et al. train a correctness classifier which takes only questions as input, without answers, estimating whether a question might be answerable but not whether a given answer is correct. As we explore in Section 6.1, a model conditioned on only questions is sensitive to learning shortcut features of the question, effectively clustering question topics. By contrast, we show that our model is learning the underlying relationship between the question and answer. Beyond differences in the setup of our fine-tuning experiments Kadavath et al. suggests that language models are strongest when evaluating their own generations and subsequently posit that uncertainty estimation is linked to self-knowledge. We find that capable models can readily learn good uncertainties for predictions of other models without any knowledge of their internals (see Section 6.1).

Clarifications to Figure 1

Each dot represents a single model, the models included are LLaMA-2 (7B, 7B-Chat, 13B, 13B-Chat, 70B, 70B-Chat), LLaMA-3 (7B, 7B-Instruct, 70B, 70B-Instruct), and Mistral (7B, 7B-Instruct). For “Verbal” only the Chat/Instruct models are included because base models generated nonsensical probabilities from when using only prompting. The coordinates of each dot show the test metric averaged over all subsets of MMLU.

Base / Instruct Models

In terms of ECE and AUROC, Figure 7 of the appendix provides a detailed breakdown for all base and instruct models. Unlike in the paper you referenced, base models typically have worse accuracy, ECE, and AUROC when using prompting methods (e.g. zero-shot classifier). One potential reason for the difference is the mismatch between multiple-choice and open-ended settings. Post-training can impair the per-token calibration of a model without hurting its calibration at the level of correctness/incorrectness (captured by prompting methods or trained estimators). In our experiments, post-training can actually be helpful because it improves the model’s ability to adhere to formatting or instruction constraints. Instruction-tuned models tend to have better open-ended accuracies overall (because their answers are direct and concise) and perform better with zero-shot prompting methods where the output format can be complicated or abstract (as in the case of outputting probabilities as numbers).

Notably, however, our fine-tuning procedures create useful estimates for both base and post-trained model variants, with improved ECE and AUROC. Models with calibration deteriorated by RLHF can be re-calibrated with post-training on graded examples.

Details on estimating P(correct)

“Probe” is a classification head on top of the penultimate layer (before token prediction) of the LLM, where the input is the representation of the last token in the context. This head, a small 3-layer neural network, similar to the prescription of Azaria and Mitchell (reference [3] from the submission), serves as the model for binary classification (correct v/s incorrect).

Our “LoRA + Prompt” method is essentially a trained version of the zero-shot classifier in Line 159, where instead of attaching a head, where we use the same prompt as in the zero-shot classifier but train through the features of the model using low-rank adapters (LoRA). This approach allows the correctness predictor to extract slightly different information from the intermediate layers of the LLM. The loss is computed over the two tokens corresponding to the options (i) and (ii).

Training Set Details

Our training set is a uniform mixture of various open datasets listed in Appendix C.2. For each model, we generate outputs for the samples in the mixture labeled for correctness (as graded by GPT-3.5). Because we use generations from the model and the models are not perfect, we get both correct and incorrect labels in our training set. We will make these details explicit in Section 5.

Uncertainty Model vs Verifier

In principle, a trained calibration model is very similar to a trained verifier. However, a trained verifier in the linked paper is being used to select a single answer from a set of generated samples by estimating which is most likely to be the correct answer. By contrast, our goal is to estimate uncertainty for the answer that has been chosen. We therefore care about properties like calibration (ECE) which indicate that the model’s confidence is not just useful for filtering but also interpretable as a probability.

We appreciate your thoughtful and supportive feedback!

Additional Benchmarks

Inspired by your comments, we provide results on the MMLU-Pro benchmark, which is a more challenging multi-task language understanding benchmark. This dataset is much newer than any of the models we test and bears no risk of contamination. Overall, we see that supervised methods provide much stronger performance in terms of ECE and AUROC over the zero-shot classifier.

LLaMA-2 7B-Chat

LLaMA-2 13B-Chat

Mistral 7B-Instruct

Training Details

For all probing experiments, we use small 3-layer MLP with hidden sizes of [256, 128, 64] and ReLU non-linearities, same as used by Azaria and Mitchell (2023) (reference [3] in the submission).

We find regularization parameter 1.0 to be sufficient for generalization across all models.

For LoRA + Prompt, we use the following prompt:

Is the proposed answer correct?
Choices:
  (i) no
  (ii) yes
Answer:

We will make these details explicit in our camera-ready version of Section 5.

AUROCs Near 0.5

You are correct that some methods, such as Probe and sBERT, have close to random performance. We don’t believe these results contradict our paper’s main claims. Firstly, we argue throughout the paper that fine-tuning through the model with LoRA leads to much better results than methods that use frozen features. In many cases frozen features get near 0.5 AUROC, while LoRA + Prompt gets near 0.7, which is significantly better than random. Likewise, the “Incorrect” label refers to a baseline and its relatively poor performance demonstrates that our model performs well because it learns a meaningful connection between questions and correct answers. Secondly, while several baseline methods have AUROCs near 0.5 in aggregate, and this demonstrates their inferior performance compared to other methods (such as LoRA + Prompt), there are many subsets of MMLU for which these models perform significantly better than random (though still worse than better methods like LoRA + Prompt). We provide a detailed breakdown over all MMLU tasks in Appendix D.

Related Work

Thank you for bringing Dennis et. al. to our attention. We see their work as complementary to ours. Both our paper and their paper show that LLMs can be fine-tuned to improve uncertainty estimation but the origin of the labels is different. In their paper, a sampling method is distilled into a single uncertainty estimator, while in our paper, we distill an expert grader into an uncertainty estimator. In practice we find that sampling methods had poor discrimination (low AUROC), so it’s unlikely that training on the outputs of the sampling procedure would yield a useful estimator in our case. In cases where sampling performs on par with an external grader, however, distilling from a sampling method might be preferable because of the cost of expert labels. We will add this discussion to our related work section in the paper.

Clarifications on Figure 1

There are 12 points in the plots in Figure 1 (left). These correspond to the following 12 models: LLaMA-2 7B, LLaMA-2 7B Chat, LLaMA-2 13B, LLaMA-2 13B Chat, LLaMA-2 70B, LLaMA-2 70B Chat, LLaMA-3 8B, LLaMA-3 8B Instruct, LLaMA-3 70B, LLaMA-3 70B Instruct, Mistral 7B, Mistral 7B Instruct. The plots in Figure 1 (right) should contain the exact same models. In the case of “Zero-Shot Classifier”, the LLaMA-2 70B numbers were dropped accidentally. We include the AUROC and ECE numbers below:

The 6 points displayed for “Verbal” correspond to only the Chat/Instruct model versions. Only the Chat/Instruct version were used because base model versions were all incapable of generating verbalized probabilities with prompt engineering on its own. We agree that including more models would be ideal. We are limited by our access to compute and make our best effort to run many of the state-of-art models available at the time of submission. We also included a 95% bootstrapped confidence interval for our linear fit to give a sense of the statistical power of the trend line. We don’t think there’s any reason to believe that there will be a sudden inflection point in the curves for “Zero-Shot Classifier” or “Verbal”, and other papers covering uncertainty estimates for models as powerful as GPT-4 have also found lack of calibration/discrimination that persists for even the most capable models [1].

You also asked whether the scale of perplexity numbers affect the final AUROCs. AUROC is scale invariant because the true positive rate (TPR) and false positive rate (FPR) for a predictor $f$ and cutoff $k$ are equivalent to the TPR and FPR of a predictor $g = c \cdot f$ with cutoff $c \cdot k$. AUROC is an integral of TPR over the FPR in the interval [0,1], and we could rewrite AUROC($g$) as AUROC($f$) using a change of variables. AUROC therefore only captures how well each model ranks the incorrect and correct answers. ECE by contrast, is not scale invariant and is only appropriate for normalized values, which is why we use AUROC in this comparison.

References

[1] Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ArXiv, abs/2306.13063, 2023.