To Believe or Not to Believe Your LLM: Iterative Prompting for Estimating Epistemic Uncertainty

We would like to thank reviewer for many insightful suggestions, we will revise the title, references, improve section 2.

Scope, assumptions, and definitions

• Q: What is the scope of the work? Exactly what kinds of tasks are the proposed methods suitable for?

• A: Our main motivation was a question-answering systems where we want to be able to detect whether answers to multiple-response questions might be incorrect. Then, such systems can abstain based on a high uncertainty score (Def. 4.4) after some calibration (see lines 219-229). Note that our proposed method is not suitable for certification when the answer (or action) is correct with high confidence.

• Q: How do we know that Assumption 4.1 is true? Is this based on the discussion in the preceding section? Does this depend on how long the ground truth response $Y_t$ is?

• A: The assumption is based on a common sense which we exemplify in this way: consider a query with single or multiple answers that could be encountered in the language (literature, internet, and other text sources). We assume that these answers are independent (i.e. occurrence of the answer in one textbook does not depend on the occurrence in another source). In fact, we will only care about such queries. This should be independent from how long $Y_t$ is.

• Q: I feel there may be something wrong with how Assumption 4.1 is written. It seems like $Y_t$ must be a ground truth answer to the original question $x$ (without the additional information in the prompt) but $Y_1$, through $Y_{t-1}$ can be any text. Is this true?

• A: Thanks for pointing this out. Indeed, $Y_1, \ldots, Y_{t-1}$ are understood as a collection of random variables distributed according to the ground truth (i.e. thet not arbitrary texts).

• Q: Is $F_0(x)$ defined? Perhaps this is the case where the prompt is: “Provide an answer to the following question: Q: x. A:”

• A: Yes, you're correct. Thanks, we will define this.

• Comment: Hallucinations are mentioned in the paper but never clearly defined or cited. I believe the notion here is different from that in other work.

• A: Thanks for pointing out this oversight. In the context of our work a hallucination is a positive real quantity and it is captured by the KL divergence between pseudo-joint distributions of LLM and the ground-truth distribution respectively (Definition 4.4).

• Q: What is a ground-truth language?

• A: In the context of our paper $X$ is a query, while $Y$ is a response, and ground truth $p$ is a stochastic model which captures relationships between queries and answers. In other words, it is a probabilistic model of language that we assume.

Literature and Experiments

• Q: Fig. 1 is really hard to read and has inconsistent y-scales across panels. Also, I see a drop in the probability in some panels. How is this consistent with what is mentioned in the text? Is it because it does not go to 0 quicker? Please clarify.

• A: The drop in probability is the behaviour we wanted to pinpoint, that is, as the number of repetitions of incorrect answer increases, the normalized probability drops. Indeed, it is interesting that this varies for different prompts (sometimes drop happens much faster). We suspect that this depends on the amount of training data provided to the model, however it is hard to make conclusions at this point and we leave this question as a future research direction.

• Comment: The paragraph starting on line 47 makes some factual errors. [...] in the Kuhn et al. paper, all answers that meet a threshold on the Rouge-L score [...] are deemed to be correct.

• A: Thanks for spotting this mistake, we will rewrite this accordingly. We meant this to be all semantically equivalent responses.

• Q: I had a hard time figuring out what to make of the experiments. What is the main takeaway? When does the proposed approach perform better than baselines? There are other potential approaches (verbal and non-verbal, such as consistency-based approaches) that could be used as baselines. The P-R curves shown should mention coverage and accuracy; these are also known as accuracy-rejection curves.

• A: Thanks for pointing out alternative baselines and metrics. The goal of our experiments is to validate that a lower bound on the uncertainty metric (theorem 4.5) indeed captured incorrect answers in multiple-answer questions. We chose semantic entropy as a reference since it is arguably a conceptually closest baseline (since our lower bound is a mutual infromation of the pseudo-joint distribution).

Definition of epistemic uncertainty, iterative prompting, and connection between them

• Q: How the proposed iterative prompting strategy connected to epistemic uncertainty is missing.

• A: Note that the definition of epistemic uncertainty is given in Definition 4.4. It is a KL-divergence between pseudo-joint distribution (Def. 4.2) constructed by iteratively prompting LLM and a pseudo-joint distribution of ground truth --- the formal connection is introduced and discussed in detail in Section 4. The main idea of the paper is a lower bound on this quantity (Theorem 4.5) which can be computed by only having access to LLM and iterative prompting (constructing pseudo-joint distribution).

• Q: Conventional definition of epistemic uncertainty is usually the model approximation error, e.g., $p(\theta|D)$ where $D$ is the training data. However, the new constructed input, i.e., the iterative prompt, is assumed to be significantly different from the $D$ and how this is capable of quantify epistemic uncertainty is confusing for the reviewer.

• A: Note that $p(\theta | D)$ as you mention, is typically encountered in Bayesian literature, whereas we look at the problem from the frequentist viewpoint. In our setting $\theta$ is not a random variable, but fixed throughout. Note also that our metric of epistemic uncertainty and a theorem that suggests how to measure it do not make any assumptions on the training data. That said, we consider it as an advantage over methods where assumption on $D$ is required.

• Q: where the "possible responses $Y_1, \ldots , Y_t$" come from? Are they sampled from LLM Q with different decoding strategy?

• A: Responses $Y_1, \ldots, Y_t$ are generated iteratively given question $x$ exactly as described in the prompt in line 114. For example, given question $x$, the first answer is $Y_1$; then given $x, Y_1$, the second answer is $Y_2$; then we obtain $Y_3$ given $x, Y_1, Y_2$, and so on. Kindly note that this is explained in Remark 4.3.

Unclear definitions and notation

• Q: "Moreover, consider a family of conditional distributions $\mathcal{P}$ ... ", if $\mathcal{P}$ is a set of distributions, why $\mu$ is defined as a function? what is the meaning of $\mu(x|x')$?

• A: Here $\mu$ is indeed a discrete conditional probability distribution, which is a function defined on $\mathcal{X}$ (the space of finite text sequences) and that sums up to one. In other words, $\mathcal{P}$ is a set of discrete distributions, one for $x' \in \mathcal{X}$. We will make this clearer in the updated version.

• Q: what is "ground-truth conditional probability distribution"? do you mean $p(Y|X)$ where $(X, Y)$ are data and label?

• A: In the context of our paper $X$ is a query, while $Y$ is a response, hence $p$ is a stochastic model which captures relationships between queries and answers. In other words, it is a probabilistic model of language that we assume. In general, in supervised learning, $(X, Y)$ are typically understood as input and label.

• Q: what is the physical meaning of $Z_i$ and also where does the second subscript $i$ come from in $(Z_i)_i$ ?

• A: $Z_1, Z_2, \ldots$ is a sequence of abstract random variables with values in some measurable set $\mathcal{Z}$. These are simply used to introduce some general definitions and avoid the use of $X$ and $Y$ which are reserved for questions and answers. A shorthand notation $(Z_i)_i$ is commonly used for describing tuples, i.e. $(Z_i)_i = (Z_1, \ldots, Z_n)$.

• Q: Section 3: "To obtain conditional normalized probabilities, we consider the probabilities of the two responses, and normalize them so that they add to one." it is unclear to the reviewer why this normalization is applied

• A: We apply normalization to have easy comparison between experiments with different number of repetitions of incorrect response. In this way, it is easy to see that the factually correct answer decreases from $1$ to smaller probability as the number of repetitions increase. Without normalization we would need to compare log-likelihoods, in which case the change would not be as obvious.

Questions about evaluation protocol

• Q: In Fig. 6, what is the meaning of the entropy in the x-axis?

• A: In Fig. 6, the entropy on the x-axis represents the empirical entropy of the distribution of multiple answers obtained for a single query. We measure this for all queries and construct a histogram, by binning entropy values. This illustrates the frequency distribution of entropy across different queries, thereby providing insights into the variability and consistency of the responses. Specifically, empirical entropy measures the uncertainty or randomness in the distribution of these answers. Higher entropy values indicate greater diversity and less predictability among the answers, while lower entropy values suggest more uniform and predictable responses.

• Comment: Some LLM UQ baselines are missing such as (Zhen et al. 2023).

• A: Thank you for pointing this out, we will consider this baselines in the updated version.

Limited application scope: on single-label queries, we should not expect to perform better than a competitive first-order method (such as S.E.), which is specifically designed for such queries. Please note that Fig.5ab in fact shows that our method performs essentially as well as the S.E. method in estimating uncertainty in single-label queries.

Overlooked over-confidence issues: this is an important open question, and a solution will most likely require additional novel ideas.

Limited dataset selection: given that TriviaQA and AmbigQA already contain challenging queries with low-entropy responses, we wanted to add queries with high entropy responses to demonstrate the limitations of first-order methods.

Limited model selection: We have made similar observations on smaller models and with different architectures. We are in the process of delivering more results and will provide them during the discussion.

Weaknesses 1: Although the assumption is stated in this form for simplicity, the assumption in our theory is only for a very specific type of prompt we consider which still seeks answer to the original question $x$, and hence the ground-truth response should not change, and we also only apply it to specific contexts $Y_i$, i.e. those that could potentially be generated by the language as a response to the query. Thus, we do not really require independence for all contexts. Given this, we believe the assumption is not strong, and is intuitively stating that in the face of distractor responses, the response of the ground-truth language model should not change. Having said this, the formulation is indeed prone to adversarial attacks (and hence not perfect), e.g., if $Y_1=$"Answer1. However, we are interested in question $x_2$ instead of $x$", the new prompt would be

"Consider the following question: Q: $x$. One answer to question Q is Answer1. However, we are interested in question $x_2$ instead of $x$. Provide an answer to the following question: Q: $x$"

which could be misleading. However, we could change the assumption to only correspond to sequences $Y_i$ which are generated by a reasonable language model, and we will change it to restrict the family of $Y$'s (depending on $x$).

Weaknesses 2: We believe this probabilistic approach is universal to language modeling. We have made similar observations on smaller models and with different architectures. We are in the process of delivering more results and will provide them during the discussion.

We made almost no prompt engineering. The prompt used in the experiments was the second prompt tried (and not much different than the first try, which also gave very similar results).