Measuring LLM Confidence through Stable Explanations

Dear authors, if you happen to have new data on the baselines, dataset sizes, or the order of the explanations, I'd be more than happy to discuss the ratings, as I've indicated in the review. I do believe that your general topic, combining uncertainty and explanations, is promising.

We sincerely thank the reviewer for the constructive comments and thoughtful suggestions. Please see the overall comment for concerns regarding increasing number of samples, the explanation conditioning experiment, and open-ended generation. We aim to address your specific concerns as follows:

Many common baselines are missing: Thank you for the additional baseline suggestions.

• We have added the P(true) confidence metric from Kadavath et al. 2022 to the main results (see overall comment).
• Semantic Entropy (Kuhn et al., 2023) reduces to the naive entropy in the case of multiple choice questions, as answer clusters are already well defined (i.e. no variation syntactically). While we therefore use answer token probability directly as our main confidence metric baseline, we have added additional experiments (Appendix B.7 of the revised manuscript) demonstrating that using the entropy metric corresponds to the token probability almost exactly when it comes to the selective uncertainty task. This is unsurprising as the model typically assigns most probability to a single token, meaning the overall entropy is strongly dependent on this specific confidence.
• For a similar reason we chose not to include uncertainty quantification arising from different calculations made from the answer similarity matrix (Lin et. al 2024). In the case of multiple choice answers this graph Laplacian becomes block diagonal, and the NLI eccentricity confidence estimate becomes similar to counting the frequency of responses (except now using an l2 norm). However, we agree that these semantic similarity baselines should be included for future work on open-ended generation (see further discussion in overall comment). We also appreciate the codebase suggestion and certainly will use this to help standardize future experiments!

Using a higher number of samples: We have doubled the number of samples for each experiment, resulting in similar relative improvements in selective uncertainty (see overall comment).

Release of Code: We will release our code repo in the camera-ready submission.

Notational Clarification: Thank you for the feedback on notation, we have updated definitions for the sampled explanation set $E$, and $\tau$-length token sequence $\mathcal{A}^\tau$. We clarify that the softmax is taken over the answer subset $\mathcal{S}$, and this set is determined by the multiple choice question itself (not necessarily fixed between questions).

Sampling multiple explanations is expensive: While we agree that sampling multiple explanations can be expensive, this is on par with methods like CoT consistency and Semantic Entropy (Wang et al. 2023,Kuhn et al. 2023). Additionally, one could imagine using a smaller version of the model for efficient explanation generation while using the larger model to verify logical consistency and compute conditional answer distributions (similar in spirit to speculative decoding )(Leviathan et al 2023).

Choice of Temperature: We follow the same temperature setting as suggested in the CoT consistency paper (Wang et al 2023, Figure 4), which also shows robustness to temperature variation in the range of $T\in (0.5,0.95)$.

Exploring Additional Models, Out-of-Distribution Performance: Please see overall comment

[1] Kadavath, Saurav, et al. "Language models (mostly) know what they know." arXiv preprint arXiv:2207.05221 (2022).

[2] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." arXiv preprint arXiv:2302.09664 (2023).

[3] Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. "Generating with confidence: Uncertainty quantification for black-box large language models." arXiv preprint arXiv:2305.19187 (2023).

[4] Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022).

[5] Leviathan, Yaniv, Matan Kalman, and Yossi Matias. "Fast inference from transformers via speculative decoding." International Conference on Machine Learning. PMLR, 2023.

We thank the reviewer for their acknowledgement of strong empirical results and helpful feedback. We hope to address some concerns below, and have also provided additional experimental results/figures in the overall comment.

Using the same model for generation+evaluation: Yes the same model is used for generation and evaluation, and you are certainly correct that if the model was trained on incorrect facts it may mistakenly place high entailment probability on an explanation. However, we would contend that the model’s uncertainty in this case should still be low, since our method is intended to measure the uncertainty of a specific model with respect to the logical relations it has learned during training.

Open Ended Generation: This method has a natural extension to open-ended generation, please see the overall comment for our discussion.

We thank the reviewer for the constructive comments and feedback. Please see the overall comment for discussion around the stability terminology, and updated+additional experiments. We address specific concerns as follows:

Computational Complexity + Semantic Clustering: In the multiple choice setting, we make two additional calls (single generated token each) to the model for each explanation generated, one to evaluate entailment probability, and a second to determine the answer logit vector. As a result our method is still O(n) where n is the number of explanations sampled. This computational efficiency is also a reason why we did not initially use pairwise similarity between explanations. However, we agree that this would be an interesting line of future work.

Additional Experiments: Please see the overall comment for additional experimental baselines, as well as an updated figure 1a which now examines explanations generated before the answer.

Ensemble-based calibration methods: To our knowledge CoT-consistency is the primary ‘ensemble based’ method for use with LLMs without requiring access to internal weights or activations.

Additional Motivation for Using Explanations: As for one concrete reason why explanation-based methods are better suited for these uncertainty tasks: consider that classical methods are restricted in evaluating uncertainty for a fixed set of hypothesis/answer classes. In contrast, our method can evaluate answers not seen during training. While using the LLM to predict something like P(True) [1] for this new answer will clearly not produce a calibrated likelihood that the answer is correct, we contend that the likelihood of answer+explanation pairs is still meaningful. This is because we make the assumption that while the train and test distributions may be different, both worlds follow the same underlying logic.

For example, examining the default softmax of answer token probabilities for the following question on GPT-3.5 gives us:

Question: Carnows are gromulites. A busdriver is a carnow but not gorocks. Amy is a busdriver. What is Amy?

A) gorocks → 0.034

B) sherry → 0.039

C) witherton → 0.013

D) gromulite → 0.914

This model is obviously not calibrated for these new terms, as the clearly wrong answer (A) is still being assigned 3% probability of being correct. After conditioning on the correct logical explanation, this answer distribution becomes a singleton around the correct term (D).

Overall Significance: Please see the overall comment for discussion around extending to our confidence metric to open-ended generation.

[1] Kadavath, Saurav, et al. "Language models (mostly) know what they know." arXiv preprint arXiv:2207.05221 (2022).

We thank the reviewer for their careful reading of the paper and detailed response. In the main comment we discuss additional adaptive calibration metric results as well as stability and transductive terminology. We hope to address additional concerns below:

Bayesian Clarification: We thank the reviewer for this point about the Bayesian connection, and would like to add further clarification: if one considers the question itself as the evidence, we are computing a posterior likelihood of the explanation given the question using a Monte Carlo approach. Our explanation prior is implicit to the language model that we sample from: for example longer explanations will tend to be less likely and as a result sampled less often. With this in mind we have rewritten equation (5) to make the bayesian connection more explicit:

$

p(e|q)=\frac{p(q|e)p(e)}{p(q)}\approx \frac{\phi_{ent.}(q,e)}{\sum_{e'\in E}\phi_{ent.}(q, e')} =: \hat{p}(e|q),

$

We also state that we are calculating the ‘posterior predictive distribution’ on line 173 but will update references to the ‘Bayesian posterior’ to be more clear on lines 218 and 461.

Generalization to out-of-distribution test data:

• As noted by reviewer b15e, though somewhat standard the datasets selected are still “challenging” to LLMs, as indicated by lower overall accuracy. Additionally the GPT-4 technical report (Figure 8) suggests poor default calibration with datasets such as MMLU after applying post-training [1].
• As for one reason why explanation based methods can better generalize: consider that classical methods are restricted in evaluating uncertainty for a fixed set of hypothesis/answer classes. In contrast, our method can evaluate answers not seen during training. While using the LLM to predict something like P(True) [2] for this new answer will clearly not produce a calibrated likelihood that the answer is correct, we contend that the likelihood of answer+explanation pairs is still meaningful. This is because we make the assumption that while the train and test distributions may be different, both worlds follow the same underlying logic.
• You are correct that CoT-consistency also “adapts” to test data, however it is not guaranteed that this adaptation remains faithful to the data in the question [3]. Our method better ensures we are making a correct adaptation to the question and not just producing an explanation that drives the decision entropy to zero.

Additional Edits for Clarification:

• We have updated the mention of explanation clustering (line 216) to be more clear: instead of directly clustering explanations based on their pairwise similarity, we are effectively clustering based on their resulting answer distribution. If two explanations support the same answer, then they can be viewed as belonging to the same cluster. Our confidence for a specific answer is calculated by marginalizing over this cluster, i.e. all explanations that have non-zero probability mass assigned to that answer.
• We have added additional clarification to distinguish between token level probability $p_\phi(a|q)$ and our estimate that answer is correct $\hat{p}(a|q)$.

Exploring Additional Models Please see overall comment for discussion.

[1] Achiam, Josh, et al. "Gpt-4 technical report." arXiv preprint arXiv:2303.08774 (2023).

[2] Kadavath, Saurav, et al. "Language models (mostly) know what they know." arXiv preprint arXiv:2207.05221 (2022).

[3] Turpin, Miles, et al. "Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting." Advances in Neural Information Processing Systems 36 (2024).

We provide additional discussion below for some common reviewer concerns and questions.

OOD Generalization: One reason explanation-based methods can better generalize to unseen data is that classical methods are restricted in evaluating uncertainty for a fixed set of hypothesis/answer classes. Although one may naively have the LLM predict something like the probability that a new answer class is true (i.e. P(True) from [1]), this is not guaranteed to produce a calibrated likelihood that the answer is correct. In contrast, we contend that the likelihood of explanation+answer pairs is still meaningful. This is because we make the assumption that while the train and test distributions may be different, both worlds follow the same underlying logic.

For example, examining the default softmax of answer token probabilities for the following question on GPT-3.5 yields:

Question: Carnows are gromulites. A busdriver is a carnow but not gorocks. Amy is a busdriver. What is Amy?

A) gorocks → 0.034

B) sherry → 0.039

C) witherton → 0.013

D) gromulite → 0.914

This model is obviously not calibrated for these invented terms, as the clearly wrong answer (A) is still being assigned 3% probability of being correct. After conditioning on a CoT explanation, this answer distribution becomes a singleton around the correct term (D). Our method attempts to select for these logical explanations and marginalize over their resulting answer distributions.

Open-ended Extension: One natural extension of our method to open-ended generation would be to generate explanation+answer pairs, and then treat these answers as our new choices. We can then evaluate the likelihood of these choices given a specific explanation to determine the conditional answer distribution. This type of “cross-perplexity” evaluation has connections to semantic similarity metric proposed in [3] and from this perspective could be seen as marginalizing over the meaning of all logically entailed explanations associated with a question.

Additional Models: We primarily focus on testing our method on models at different scales rather than across model families for two reasons. First, existing literature [4] has already shown the effect of CoT explanations to be similar across a variety of architectures. Additional work [5,6] has also shown that logical reasoning capability such as binary entailment is primarily a function of model size. In particular, Parmar et al. 2024 [5] observe similar performance in logical reasoning for llama2-7b and mistral-7b models and observe consistent increases in performance as they scale to larger models such as Yi-34B and then GPT4.

[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." arXiv preprint arXiv:2302.09664 (2023).

[2] Nixon, Jeremy, et al. "Measuring Calibration in Deep Learning." CVPR workshops. Vol. 2. No. 7. 2019.

[3] Liu, Tian Yu, et al. "Meaning representations from trajectories in autoregressive models." arXiv preprint arXiv:2310.18348 (2023).

[4] Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022).

[5] Sanyal, Soumya, et al. "Minds versus Machines: Rethinking Entailment Verification with Language Models." arXiv preprint arXiv:2402.03686 (2024).

[6] Parmar, Mihir, et al. "LogicBench: Towards systematic evaluation of logical reasoning ability of large language models." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.