Improving Uncertainty Quantification in Large Language Models via Semantic Embeddings

We sincerely thank the reviewer for their thoughtful and detailed feedback. We appreciate the opportunity to clarify the concerns.

Q1: In line 185, "cosine similarity provides a continuous metric". In fact, the bidirectional entailment can also offer continuous scores by using the output probability (or logit).

We appreciate the reviewer's insightful comment regarding the potential of using the output probabilities from the entailment classification to express the degree of entailment between different responses. Based on this suggestion, we have included the entailment probability as an additional baseline in our experiments. Furthermore, to enhance the robustness of our evaluation, we use a LLM as a judge (GPT-4o) to assess the correctness of the generated responses.We find that SEU outperforms the average pairwise entailment probability baseline in 8 out of 9 cases across different models and datasets. The following table lists the AUROC values for different uncertainty measurement methods across models and datasets.

Q2: ROUGE-L cannot capture the semantic equivalence well and is insensitive to word orders. The evaluation step needs further improvement by using better metrics such as BERTScore and LLM-as-Judge (though stated in the conclusion)

Based on this suggestion, we provide results using a LLM as a judge (GPT-4o) to assess the correctness of the generated responses (please see the above response)

Q3: It is important to report multiple metrics for uncertainty estimation, e.g., Brier and ECE.

We appreciate your suggestion to report multiple metrics for uncertainty estimation, such as the Brier score and ECE. However, in the context of free-form question answering (QA) and natural language generation (NLG), these metrics pose significant challenges due to the open-ended nature of the responses. More specifically, calculating the Brier score and ECE requires estimating the total probability mass assigned to all possible correct answers. In free-form text generation, there are nearly infinitely many ways to express a correct response, making this estimation infeasible. Language models assign probabilities to specific token sequences, not to the underlying semantic meanings. Therefore, these metrics cannot accurately capture uncertainty over the correctness of responses. As noted by Kuhn et al. (2023),

“The AUROC is a better measure of uncertainty for free-form question answering and NLG than calibration measures like the Brier score, which are often used in classification or for multiple choice QA. In order to estimate the Brier score, we would need to estimate the entire probability mass assigned to any possible way of saying the correct answer. This is intractable for free form text where we do not have access to probabilities about meanings.”

Q4: a key baseline is missing: INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection (ICLR 2024), which develops an EigenScore metric to measure the semantic consistency in the embedding space.

We thank the reviewer for the pointer to INSIDE. We will include this baseline in the next revision.

Q5: In Table 2, it is hard to see the benefit of SEU since it leads to a worse FPR

The intent of presenting Table 2 was to dissect the contributions to the AUROC from both TPR and FPR, thereby illustrating where SEU provides benefits over SE. Specifically, we aimed to show that the SEU’s higher AUROC stems from a higher TPR which demonstrates SEU's superior ability to detect when the language model is certain and likely to produce correct answers. In contrast, SE by using bidirectional entailment inflates uncertainty and is susceptible to flagging correct answers as having high uncertainty.

Reference:

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations, 2023

“The AUROC is a better measure of uncertainty for free-form question answering and NLG than calibration measures like the Brier score, which are often used in classification or for multiple choice QA. In order to estimate the Brier score, we would need to estimate the entire probability mass assigned to any possible way of saying the correct answer. This is intractable for free form text where we do not have access to probabilities about meanings.”

Thanks for your responses. I have a question about the metrics of Brier and ECE. If you can obtain labels for the generated answers, e.g., true or false, it is still possible to compute Brier and ECE. See one reference below:

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

Thank you for your follow-up and for pointing us to the paper "Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback." We appreciate your insightful suggestion.

You are correct that if we obtain binary labels (true or false) for the generated answers and elicit confidence scores from the language model, it is possible to compute calibration metrics like the Brier score and Expected Calibration Error (ECE). However, our study focuses on uncertainty estimation methods that do not rely on prompting the model to provide explicit confidence scores.

Moreover, recent research [1] (figure 2) indicates that language models' self-reported confidence scores may not be well-calibrated, especially when limited to a small number of samples per query. This challenges the reliability of calibration metrics derived from such confidence scores.

We acknowledge the value of calibration metrics and agree that incorporating them could provide additional insights. However, integrating these metrics would necessitate a different experimental setup that includes confidence elicitation, which is beyond the scope of our current work. Our aim was to compare uncertainty estimation techniques that operate without additional prompts.

We appreciate your suggestion, as it highlights a valuable avenue for future research.

[1] Measuring short-form factuality in large language models https://arxiv.org/abs/2411.04368.

Thank you for your feedback once again. We agree that reporting multiple metrics can strengthen our claims, and we appreciate the suggestion to consider the Brier score and ECE. However, as you've noted, calculating these metrics requires access to the model's probability estimates of correctness for each answer.

In the free-form setting of our paper, the model generates open-ended responses without accompanying confidence scores. The probabilities assigned by the model are over sequences of tokens rather than over the semantic correctness of the answers. To compute the Brier score or ECE, we would need to either:

1.) Elicit Confidence Scores via Prompting: This would involve modifying our methodology to include prompts that ask the model for its confidence in each answer i.e prompt based methods.

or

2.) Restructure the Task Format: By converting our free-form questions into multiple-choice or true/false formats, we could use the model's probabilities over these discrete options. However, this would change the nature of our task from free-form generation to a classification problem, hence not requiring an estimate of semantic uncertainty.

Given these constraints, we believe that including Brier score or ECE in our current work isn't feasible without altering our experimental setup. Could you please help us understand if there would be another way of calculating ECE/Brier? We do sincerely appreciate your comments and we believe a bit more clarification would help us improve.

We sincerely thank the reviewer for their feedback. We appreciate the opportunity to clarify the concerns.

Q1: Generalization to Other NLP Tasks?

In our work, we specifically focus on free-form question-answering (QA) tasks for hallucination detection, following the precedent set by prior research in this area (e.g., [1,2,3]). Free-form QA is particularly susceptible to hallucinations due to the open-ended nature of the task and the lack of constraints compared to some other tasks. Our goal was to develop and validate the SEU method within this context, where accurately estimating semantic uncertainty can have a significant impact on the reliability of language model outputs. While our experiments and validations are centred on QA tasks, we believe that the underlying principles of SEU and ASEU —quantifying semantic uncertainty through comparing response embeddings — are applicable to other NLP tasks.

Q2 (a): Computational Resources in Model Training?

While ASEU introduces a fine-tuning phase to learn the latent semantic representations, we designed the process to be efficient and practical for real-world applications. Specifically: we fine-tuned the ASEU model using 20,000 question-answer pairs from the TriviaQA dataset. The finetuning process only requires learning the additional adapter weights, while the LLM weights are held fixed. For this fine-tuning step, we used a single NVIDIA A100 GPU. The fine-tuning step can be done in less than 4 hours.

Q2 (b) On your question about How does this compare to traditional methods in terms of efficiency?

Could you please elaborate which traditional methods you are referring to in this question?

Q3 Reliability on Pre-trained Models?

While we acknowledge that the quality of semantic embeddings could impact SEU's performance. As we show in our experiments, the off-the-shelf sentence-BERT model delivers strong performance across a variety of datasets. The modular nature of our approach allows for easy substitution of the embedding model component. As better sentence embedding models become available, they can be readily incorporated to further improve performance.

To support this point, we provide additional experiments where we use NV-embed which is a new SOTA sentence embedding model [5].

The consistent performance we observe across different datasets and LLMs and 2 sentence embedding models, provides strong evidence for the robustness of our method.

Q4 Diversity in Model Size Analysis?

We appreciate the reviewer's suggestion about including a broader range of model sizes. However, we believe our current analysis already provides strong evidence for the effectiveness of our method across a meaningful spectrum of model sizes

• Phi-3.5B (~3.5 billion parameters)
• Mistral-7B (7 billion parameters)
• Llama-8B (8 billion parameters)

Following the reviewers suggestion we have added new models up to 27B parameters. The new models added are:

• Gemma-2-9B (9 billion parameters)
• Gemma-2-27B (27 billion parameters)

The consistent trends we observe across the current range of model sizes (3.5B to 27B) provide evidence that our method's effectiveness is not heavily dependent on model sizes. While experimenting with 70B+ models would be interesting, our current analysis focuses on model sizes that are readily accessible to most researchers. Please note that it is generally expensive for many academic researchers to experiment with models around or beyond the 70B parameter range.

Q5: Analysis of Close Model Variations?

We thank the reviewer for the suggestion. We have now added results from the gemma family of models [4], we have focused on 9B and 27B to show the consistency of our approach across size variations of the same model architecture. Please see the results in the response above.

Dear Reviewer 8m62,

We appreciate that you are likely to be reviewing multiple other papers; however, as we are approaching the end of the discussion period (less than one week remaining), we would greatly appreciate your feedback on our rebuttals. We appreciate your feedback and we hope that we have adequately addressed the concerns you raised. We have also included additional experiments based on your suggestions including bigger models.

Your additional insights would be valuable in helping us improve the paper further.

Kind regards, The authors of paper 10173

[1] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, Tie-Yan Liu. Representation Degeneration Problem in Training Natural Language Generation Models. ICLR 2019.

[2] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language

Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In International Conference on Learning Representations, 2023

Nikitin, A., Kossen, J., Gal, Y., & Marttinen, P. (2024). Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities.

We sincerely thank the reviewer for their thoughtful and detailed feedback. We appreciate the opportunity to clarify the concerns about our proposed Semantic Embedding Uncertainty (SEU).

On Weakness #1: Reliability of cosine similarity of embeddings as an absolute measure of semantic relatedness

We understand the concern about the isotropy of the embedding space, as discussed in Gao et al. [1], which can affect the reliability of cosine similarity measures. However, we would like to clarify that the representation degeneration problem highlighted in [1] primarily pertains to language generation models trained with likelihood loss, where the goal is to predict the probability of the next word. This can lead to embedding degeneration (of word embeddings), making cosine similarity an unreliable measure of semantic relationships. In contrast, our method employs Sentence-BERT [2], which is specifically designed to produce semantically meaningful sentence embeddings that are suitable for comparison using cosine similarity. Specifically, Sentence-BERT uses siamese and triplet network structures to fine-tune BERT models, optimising their embeddings for semantic similarity tasks. As stated in [2]: "We use siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity." Therefore we believe that this training approach can alleviate the issue of using cosine similarity which may arise when using embeddings from regular language models.

On Weakness #2: Advantages of SEU compared to SE

We appreciate the reviewer's concern regarding the interaction between responses in NLI-based methods versus our SEU approach. While NLI classifiers in SE consider both responses jointly, we argue in this work that the strict entailment criteria they rely on can be overly stringent for measuring semantic uncertainty. More specifically, for sentence A to entail sentence B, all the information in B must be present in A. This is a stringent criterion that does not tolerate additional or missing details, even if they are minor (we provide examples of such failure cases in Table 1.) This strictness means that even minor variations, paraphrasing, or inclusion of extra correct information can cause semantically very similar responses to be classified as non-equivalent, inflating the estimated uncertainty of SE. As we show in Table 2 in the paper: SE overestimates uncertainty as demonstrated by its significantly lower TPR. Lower TPR suggests that SE is classifying more cases as uncertain, even when the model’s response is correct.

On your concern about the experiments: To clarify, our experiments in the paper were conducted using different language models of varying sizes, trained by different research teams (Llama 8B, Mistral 7B, Phi 3.5B, Gemma-2-9B, Gemma-2-27B) to ensure that our findings are robust and not specific to a single model. In addition, for the entailment model used in the Semantic Entropy (SE) baseline, we used the DeBERTa-Large model, consistent with prior work proposing SE Kuhn et al. (2023), Nikitin et a. (2024). We also adopted the same evaluation methodology as these previous papers, using the ROUGE-L score to assess the correctness of generated responses and the Area Under the Receiver Operating Characteristic curve (AUROC) for performance comparisons. Using the same NLI model and the evaluation methodology as in these prior works ensures a fair comparison between SE and our proposed SEU method.

On Weakness #3: Bidirectional entailment criterion can only act as a binary measure

We appreciate the reviewer's insightful comment regarding the potential of using the output probabilities from the entailment classification to express the degree of entailment between different responses. Based on this suggestion, we have included the entailment probability as an additional baseline in our experiments. Furthermore, to enhance the robustness of our evaluation (following reviewer TjGS’s suggestion), we use a LLM as a judge (GPT-4o) to assess the correctness of the generated responses. We find that SEU outperforms the average pairwise entailment probability baseline in 8 out of 9 cases across different models and datasets. The following table lists the AUROC values for different uncertainty measurement methods across models and datasets.

Dear Reviewer vHa8,

We appreciate that you are likely to be reviewing multiple other papers; however, as we are approaching the end of the discussion period (less than one week remaining), we would greatly appreciate your feedback on our rebuttals. We appreciate your detailed feedback and we hope that we have adequately addressed the concerns you raised. We have also included additional experiments based on your suggestions.

Your additional insights would be valuable in helping us improve the paper further.

Kind regards, The authors of paper 10173

Once again we thank the reviewer for their thoughtful question and recommendations.

Q1.) There doesn’t appear to be evidence supporting cosine similarity's direct use as an absolute value.

While we agree that absolute cosine similarity values require careful interpretation, we would like to draw attention to Sentence-BERTs strong performance on the STS benchmark dataset (Cer et al., 2017), which has human labels of semantic similarity on a 0-5 scale. In this dataset, for a given pair of sentences, scores are defined with the following criteria [Table 1 (Cer et al., 2017)]:

Score 5: Complete semantic equivalence (e.g., 'The bird is bathing in the sink' vs 'Birdie is washing itself in the water basin')

Score 4: Mostly equivalent with minor differences (e.g., 'Two boys on a couch are playing video games' vs 'Two boys are playing a video game')

Score 3: Roughly equivalent with important differences (e.g., 'John said he is considered a witness but not a suspect' vs '"He is not a suspect anymore." John said')

Score 2: Not equivalent but sharing some details

Score 1: Not equivalent but same topic

Score 0: Completely dissimilar

The sentence-bert model achieves a strong Spearman correlation of 0.85 on this benchmark [1], demonstrating that higher absolute values of cosine similarity correspond to higher human-assessed semantic similarity. While this correlation doesn't guarantee that specific cosine similarity thresholds map directly to the 0-5 scale, it provides strong evidence that high absolute values of cosine similarity (for eg: greater than 0.9) will generally correspond to high semantic similarity assigned by humans.

Q2.) I think incorporating some additional NLI classifiers could provide valuable insights.

Once again we thank the reviewer for their valuable suggestion to help improve our work. We have now added Roberta Large as an additional NLI classifier. We are also commencing experiments on XLNet, but please note that these experiments can take a fairly long time (due to the large size of these datasets) and we may not be able to finish it in the remaining time for the discussion period (~1 day left). We will make sure to add these experiments into the main PDF once we are allowed to do so.

Q3) would be very interested to see how the methods perform when using different NLI classifiers

Please see the response above to Q2.

References:

Daniel Cer, Mona Diab, Eneko Agirre, Iigo LopezGazpio, and Lucia Specia. 2017. SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.

[1] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language