Graph-based Confidence Calibration for Large Language Models

We're grateful for the valuable comments and thank you for your time and expertise.

1. Thank you for raising this point. We’d like to clarify our contributions in more detail.

• Our work addresses an important research gap in confidence calibration for LLMs. Existing methods overlook the role of semantic similarity in the calibration process, which limits their ability to generalize effectively across diverse scenarios. To address this, our approach integrates semantic similarity using a graph-based model, enabling more robust and adaptable confidence estimation across various contexts. This generalization capability is further demonstrated through experiments on out-of-domain datasets, where our model consistently outperforms baseline methods.
  • While related work on semantic uncertainty is related to our work, it operates in a parallel domain without addressing confidence calibration directly. In contrast, our work proposes a new confidence calibration approach for LLMs, filling a critical research gap.
  • We also want to point out that our work focuses on confidence calibration, aiming to align confidence scores with empirical correctness, ensuring the model’s predicted confidence accurately reflects its true correctness. In contrast, uncertainty estimation provides a relative measure of uncertainty without requiring alignment to actual correctness.
• Our experiments demonstrate that the proposed method consistently outperforms state-of-the-art baselines across multiple datasets, including both in-domain and out-of-domain scenarios.
• In contrast to methods requiring access to internal LLM mechanisms or extensive task-specific fine-tuning, our approach is lightweight compared to baseline methods. This simplicity enhances its applicability to real-world use cases.

2. We thank the reviewer for pointing this out. We have carefully revised the writing to improve clarity and coherence throughout the manuscript. Additionally, we have refined Figure 1 to ensure they are more informative.

3. The baseline methods mentioned, such as Self-checkGPT, fall under the category of uncertainty estimation. However, they do not calibrate confidence scores based on empirical correctness. To ensure a fair comparison, we extended the Self-checkGPT method by applying additional calibration steps, such as Platt Scaling, which also needs training to align its outputs with correctness-based confidence. This adjustment allows for a more direct comparison between our method and these baselines.

4. Regarding the long-form answers We have included a long-form generation dataset: TruthfulQA dataset. We use GPT4 to create labels to focus on semantic meanings better.

Table 1: Experimental results for TruthfulQA with Llama3 model

Table 2: Experimental results for TruthfulQA with Vicuna model

Thanks for the responses.

Some of my concerns have been addressed by introducing additional experiments on TruthfulQA. Although the improvement is marginal, I would like to raise my score.

1. Regarding the computation: We acknowledge that sampling multiple responses can introduce computational overhead. However, our sensitivity analysis reveals that the method performs well even with as few as 10 sampled answers (as shown in Figure 3b in the paper), significantly reducing computational costs without compromising performance. We have added further discussion about this. While computational cost is a limitation, our method offers a key advantage: it works for black-box models, requiring no access to the internal states or logits of the LLM. In contrast, other approaches—such as internal-state-based methods or logit-based methods—rely on white-box access to the model, making them less applicable to black-box scenarios. This broader applicability makes our method a viable alternative despite the computational trade-offs.

2. Regarding the motivation for choosing GNN: Our motivation is to leverage semantic similarity among answers to calibrate the answers’ confidence.
   Treating this task as a typical text classification task, where the question and answer are inputs to a model, would require a model with semantic understanding significantly stronger than the base LLM. Such an approach would require training a highly capable model, potentially exceeding the complexity and performance of the LLM itself, which is impractical in most scenarios.
   In contrast, our graph-based approach offers a simpler and more efficient solution by focusing solely on the semantic similarity between answers. The GNN utilizes this similarity to aggregate information across clusters of answers, enabling effective classification without requiring a semantic understanding of individual answers. This design leverages the answer’s consistency pattern, avoiding the need for a more complex text classification model and making the task computationally feasible.
   Regarding the choice of \tau in Equation 4, \tau=0.3 is chosen following the previous work [1], which satisfies most cases.

3. We agree that there may be some bias in sentence-BERT embeddings for similarity computations. To test the robustness of various embedding strategies, we evaluated the sensitivity of our method to the quality of embeddings by experimenting with alternative embedding methods, such as Roberta and GPT3 embeddings. The results show that our approach remains robust across different embedding techniques, indicating that our method is not overly dependent on the specific nuances of Sentence-BERT.

[1] Kuhn, Lorenz, Yarin Gal, and Sebastian Farquhar. "Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation." arXiv preprint arXiv:2302.09664 (2023).

We thank the reviewer's constructive comments and valuable suggestions of our work.

1. Clarification of the k: We thank the reviewer for pointing this out. To clarify, the maximum number of clusters is set to K = 3, but the algorithm naturally returns fewer clusters in cases of no or little ambiguity. Note that the cluster memberships only provide one type of input feature to the GNN. The GNN also uses response similarities as edge features, so the GNN may give less weight to cluster memberships if they are noisy. We hope this alleviates the concerns about the specification of K in our context.
2. Regarding more datasets: We have included experiments on TruthfulQA in Table 1 and Table 2 in the experiment Section. The experiment results are shown below. They provide further evidence of the effectiveness of our model:

Table 1: Experimental results for TruthfulQA with Llama3 model

Table 2: Experimental results for TruthfulQA with Vicuna model

3. Regarding the limitations: We thank the reviewer for pointing this out. We’ve revised the limitations section in the conclusion accordingly.
4. Clarification about the writing: Thank you for pointing this out. We have clarified the distinctions between “calibration data,” “validation data,” and “test data” in the corresponding sections of the paper. The calibration data is the training data for our model, and validation data is a separate split used to tune the hyperparameters of the GNN calibrator. The test data is to test the performance of calibration capability.

We appreciate the reviewer's careful reading and suggestions for this paper. We believe these revisions help improve the manuscript a lot.

Dear reviewer e4j6,

We thank you once again for your valuable suggestions. We’d like to point out that we have conducted additional experiments on the more challenging TruthfulQA dataset (the accuracy is 56% on llama3). And our method demonstrates much better performance than the baselines. Specifically, our method achieves a 74% reduction in Expected calibration error (ECE) (from 0.1 to 0.026) on TruthfulQA with Llama3, and a 44% reduction, from 0.09 to 0.05, on TruthfulQA with Vicuna. This is generally due to the non-dependency on textual information. The results show more evidence of our method’s effectiveness.

If there are any remaining questions, we would be happy to provide additional clarification.

Kind regards, Authors

3. Through our experiments, we observed that samples within the same cluster tend to share the same meaning, even when expressed using different wording or reasoning paths. In the TriviaQA dataset, 91.4% of answers within a cluster converged to the same final answer. This strong alignment indicates that clustering based on semantic similarity serves as a reasonable proxy for grouping similar answers.
   Additionally, [5] reported a similar observation. Their study highlighted that responses within a cluster often share reasoning paths or underlying semantics, despite minor differences in expression. This further supports the rationale for aggregating answers within clusters. While we acknowledge the existence of outliers, the high consistency observed within clusters across datasets and corroborated by related work demonstrates the practicality and effectiveness of this approach.

4. We appreciate the reviewer's valuable feedback. For each question, the process of constructing the GNN training data is as follows:
   Sampling Answers: We generate multiple answers for each question using multinomial sampling to capture a diverse range of possible responses from the LLM.
   Constructing the Semantic Similarity Graph: A semantic similarity graph is created where each node represents an answer, and edges between nodes represent the semantic similarity between answers. The similarity is computed using a predefined metric. Node Features: Each node is assigned features that include the cluster ID, which groups semantically similar answers. This enables the GNN to aggregate information within and across clusters during training.
   We have provided an explanation in lines 213-232 of the Method Section. We will refine this section to include a more detailed description.

5. We thank the reviewer’s feedback. We would like to point out that the baseline method, Graph Spectral, utilizes a self-consistency mechanism to estimate uncertainty, but its confidence scores are not calibrated. To ensure a fair comparison, we enhance the baseline by applying Platt Scaling to calibrate its confidence estimates with the same training dataset to our approach, resulting in the Graph-Spectral + Platt Scaling baseline. This allows us to directly compare the effectiveness of our approach against a calibrated version of the self-consistency method. We have already included this comparison in the results section for a comprehensive evaluation.

[5] Xu, Tianyang, et al. "SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales." arXiv preprint arXiv:2405.20974 (2024).

We thank the reviewer's constructive comments and valuable suggestions of our work.

1. Regarding the assumption: Our assumption is that higher consistency among LLM’s responses indicates a higher probability of correctness. To validate this, we conducted an empirical evaluation on the TriviaQA dataset. On the TriviaQA dataset, we observed a strong statistical correlation between correctness and consistency with R² = 0.966. This suggests that statistically speaking, responses exhibiting greater consistency are more likely to be correct. We further validated this assumption using the TruthfulQA dataset by analyzing the proportion of highly consistent yet incorrect answers. Results showed that only 13% of all incorrect responses exhibited high consistency.
   Similarly, this trend was observed in other models, such as GPT-4-o1, on the SimpleQA dataset, where more consistent answers demonstrated higher accuracy (see Figure 2 in [1]). While this confirms the general reliability of our assumption, we acknowledge that there are always outliers where the LLM generates consistent but incorrect responses due to systematic biases or errors. While these cases exist, they are statistically rare, and the strong correlation observed in our study supports the utility of consistency as a proxy for correctness in most scenarios. We will include a detailed discussion in the appendix. Furthermore, this assumption aligns with insights from prior work [1][2], further reinforcing its validity as a practical method for calibration.

2. Regarding the difference of reward model:The primary difference lies in the goal of the auxiliary model versus the reward model. The reward model is designed to provide feedback to LLMs to help them align with human preferences, focusing on attributes like honesty, helpfulness, and harmlessness. Specifically, reward models are trained to mimic human preferability by ranking responses based on human feedback, and they are used to fine-tune language models via reinforcement learning (e.g., RLHF). These models aim to optimize for preference, not confidence.
   In contrast, our auxiliary model is focused on confidence calibration—determining the probability that a generated answer is correct. This distinction is crucial, as preference and correctness are not synonymous. For example, a response may align with human preferences but still be factually incorrect. Our work addresses this gap by calibrating confidence, not preference.
   Further differences include:
   Neural Network Architecture: Reward models typically use language models like BERT, RoBERTa, or GPT to process the semantic content of responses and predict human preferences. These architectures are parameter-heavy and computationally intensive. Conversely, we employ a Graph Neural Network that focuses on the semantic similarity among responses. This allows us to efficiently calibrate confidence scores among multiple responses without requiring the deeper contextual understanding needed for preference modeling.
   Loss Function: Reward models use a ranking loss, which encourages higher-ranking probabilities for responses with greater preference alignment. In contrast, our model is trained with a loss function specifically designed for confidence calibration, ensuring the model predicts accurate confidence scores based on the consistency among answers. We will incorporate a discussion of the works on reward models, such as [3] and [4], in the related work section, clarifying these distinctions and providing a comprehensive context for the contributions of our approach.

[1] Wei, Jason, et al. "Measuring short-form factuality in large language models." arXiv preprint arXiv:2411.04368 (2024).
[2] Lin, Zhen, Shubhendu Trivedi, and Jimeng Sun. "Generating with Confidence: Uncertainty Quantification for Black-box Large Language Models." Transactions on Machine Learning Research.
[3] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems, 35 (2022): 27730-27744.
[4] Stiennon, Nisan, et al. "Learning to summarize with human feedback." Advances in Neural Information Processing Systems, 33 (2020): 3008-3021.