LACIE: Listener-Aware Finetuning for Calibration in Large Language Models

Thanks for your comments and questions, and for highlighting our “excellent experimental setup” and positive results. We’ve sought to address each point in the review below:

1. Highting decrease in recall/accuracy due to abstention in the intro

We agree that this is an important point to highlight – we will propagate our discussion of the tradeoff between recall and precision from L263-269 to the introduction and abstract in the camera-ready version of the paper.

2. Further analysis of abstained cases

Thank you for the suggestion -- we have added the above-described analysis of the abstained cases in Figure 1 of our rebuttal pdf, with the full experiment described in our main rebuttal. Because of the complicated nature of estimating confidence using token probabilities on long-form generation, we follow the second suggestion and measure the diversity of answers when decoding with a high temperature. Specifically, we sample 100 abstained and 100 unabstained examples and decode 40 generations for each, extracting their answers and counting the number of unique answers among them. We find that the base model generally has substantially more unique answers on abstained examples, indicating that these have higher uncertainty in the base model, and that LACIE training allows the model to detect this uncertainty and abstain appropriately.

3. Adding TP/FP/FN numbers

We have added this to Table 3 of the rebuttal pdf; for the sake of space, we have only included Mistral-7B but the other models are similar in their trends. In summary, we find that the increase in precision in Table 1 of the original paper for LACIE-trained models is indeed driven by a ~56% reduction in False Accepts, from ~250 to ~109. The reduction in recall is driven by a ~34% increase in False Rejects from ~115 to ~173. These are indeed more dramatic trends than those seen in Table 2 of the original paper, where we saw a 47% decrease in False Accepts and no significant increase in False Rejects.

4. Question 1: do humans reject underconfident answers less?

We believe that this is indeed the case, and it points to a direction for future work, which is aligning the listener model more to human intuition. While the listener model generally is a good proxy for people, there are clearly deviations (e.g. rejecting answers that humans might actually accept). Another possible reason for the difference is that the listener model does not see the actual answer, while the human annotators do. While we filtered out examples that human annotators said they knew the answer to (L581-583), it could still be that they did not know the answer enough to report knowing it, but know what kind of answer is plausible (e.g. even if you do not exactly know the capital of Germany, you may know that “Paris” is not a plausible answer).

Smaller points:

• We will add a reference to the human results discussion in 4.2
• We will update Figure 3 of our final version accordingly (excluded from the pdf currently because of space constraints)
• We will add the following more descriptive caption for Table 3 in the original paper: “TruthfulQA performance using Mistral-7B as measured by finetuned Llama3 judge models for truthfulness and informativeness. Higher truthfulness reflects lower rates of hallucination. Informativeness reflects topicality, and penalizes abstention.”

Thanks for highlighting the importance of our research problem and our empirical improvements. We have sought to address the remaining questions/comments below:

1. Improvements on additional baselines

Thanks for your suggestion of non-pragmatic baselines. To clarify, our main preference tuning in our original paper’s Table 1 is based on the judgment of a Mistral-7B-base listener model, which for the first set of results is the same architecture as the speaker model (which is the only model being tuned) and is not pragmatic, in that it does not consider the distribution over answers, but only the implicit and explicit confidence of the answer.

To further highlight non-pragmatic baselines, we have added two new baselines from Tian et al. (2024) (https://arxiv.org/abs/2305.14975), described in our general rebuttal response, Table 1 of the rebuttal PDF. These baselines are non-pragmatic in that they do not take into account a listener model and act just as a speaker. In the "no listener" setting we also evaluate Tian et al.’s method by directly extracting the confidence (rather than using a listener model), i.e. using a non-pragmatic evaluation. Our results here show that LACIE outperforms the additional baseline (both when applied to base models and chat models), with a consistent increase in precision and AUROC over the new baseline across both Mistral-7B and Llama3-8B. This highlights the importance of pragmatic listener-aware training.

2. Qualitative analysis on abstained answers

Thanks for this question. We have added a qualitative analysis to our general rebuttal and the rebuttal PDF in Figure 1. Here, we show the answer diversity from the base model on abstained and non-abstained examples. We find that the base model has higher answer diversity on examples that LACIE abstains on (orange), and lower diversity on examples that LACIE does not abstain on (blue). This indicates that LACIE training allows the model to identify examples that have high uncertainty (i.e. high answer diversity) and then abstain on these examples. In other words, it does seem that abstention is worth it, since the model is abstaining on high-uncertainty examples.

3. Adding different listener models

Thanks for this suggestion; based on this comment we have added an ablation using Llama3-8B as the listener model. The full experiment is described in our general response and shown in Table 2 of the rebuttal PDF.

To summarize, we find that LACIE with a Llama3-8B listener still increases the precision and AUROC over the base model, but not as much as with a Mistral-7B listener. We do however find that it improves the ECE the most. Taken together, these results indicate that LACIE is compatible with different listener models.

Thank you for appreciating our human evaluation, and our analyses – we will address the remaining questions/comments in more detail below

1. “...Do you qualitatively/quantitatively find that justifications that can clearly be inferred to be incorrect (e.g., based on personal learning experiences that are clearly incorrect) reduce in the fine-tuned model variant?”

Thanks for this question! Qualitatively, we did not find that this behavior was reduced through LACIE training; we will add more examples in the final version. One hypothesis for why is that the listener model (used to generate the training data) is not informed that the outputs it is seeing are from an LLM (the same is true for the human evaluation). Thus, the model (and people) do not actually know that there is no way for the speaker (a model) to have learned something in school (or other such backstories) and so the speaker is not penalized for making such statements (as long as its answer is correct). However, it would be possible to explicitly penalize these statements by instructing the listener (or human evaluators) that the speaker is a model. In that case, statements like “I learned this in school” would be dispreferred, since they should lead to low confidence because the listener would know that the model is hallucinating at least one part of the response. Thus, we suspect that pragmatic finetuning with better listeners will be able to even further improve implicit confidence markers in speaker outputs.

2 “ it might be worth briefly discussing how the highly anthropomorphic language affects listener perception of the system and therefore how confidence is interpreted.”

Thanks for the suggestion – in our qualitative analysis in Figure 3 of the original paper, we find that the use of details does increase on correct examples after training; this does correspond to some extent to the anthropomorphic language referred in the comment above. Qualitatively, the listener model does often increase its probability of accepting an answer when these more human-like backstories are included; luckily, LACIE training results in an increase only for correct examples. However, to have a more faithful system, future work might explicitly penalize such responses (e.g. by using the listener prompt to downweight them) as mentioned in the response above. We will add a discussion of these effects to our final version.

Thank you for your attention to our work and for highlighting the importance of the problem we focus on and the excitement of our pragmatics-aware approach. We will address the remaining comments below

1. Application of LACIE to knowledge-intensive tasks

Like other work in training models to be better-calibrated, LACIE does require access to labeled data for determining correctness or factuality; this kind of data is widely available in many domains. Since calibration deals with the relationship between model uncertainty and model correctness, we would argue that any method either measuring or improving calibration requires access to some kind of labeled correctness data. In the scenario described by Reviewer dvWw, where users synthesize data from powerful LLMs, the “ground truth” data is provided by a teacher LLM, and thus the teacher’s answer is treated as the “correct” answer. In this case, LACIE could be applied in exactly the same way to train the model to be calibrated w.r.t. the synthetic data. The caveat here is that the model quality may be affected by mislabeled examples in the synthetic data; however, this is true of any method training on LLM-generated synthetic data (whether for calibration or other purposes). Assuming that the teacher model is strong enough to provide mostly factual answers, LACIE will work as expected. Moreover, LACIE does not need annotations for all claims in an answer, only for the final answer; this kind of data can generally be found in existing datasets or can be collected relatively cheaply.

2. Adding more baselines

Please see the general response for a full description of the additional baseline. To summarize, we have added two versions of Tian et al. (2024)’s method as an external baseline to Table 1 of our rebuttal PDF. Because this method asks for a direct confidence score from the model, we compare it to our methods when using a listener model to estimate confidence (as we do for all other baselines) and when extracting the confidence directly (giving their method an advantage over LACIE and other baselines, which are mediated by the listener).

We find that while Tian et al.’s method generally outperforms the base model, LACIE continues to outperform all baselines, with consistent improvements in AUROC and Precision across Mistral-7B and Llama3-8B.

3. “...I am also curious about the potential of such pragmatic-aware tuning in a variety of typical LLM tasks, beyond simply knowledge QA…”

LACIE’s results show that QA-based tuning is a valuable signal for helping the model distinguish between correct and incorrect responses; our TruthfulQA results show that this has promising implications to tasks like hallucination reduction as well. Future work might explore using the signal obtained from LACIE training on QA data (which is readily available and easily annotated for factuality) to long-form generation tasks like summarization or dialogue. More broadly, given that humans interact pragmatically when using language, we believe that imbuing models with pragmatic ability is an important step to having more natural, reliable, and interpretable interactions between humans and models.