Lookers-On See Most of the Game: An External Insight-Guided Method for Enhancing Uncertainty Estimation

1. The method requires constructing a dataset with the same domain as the dataset used for evaluation.

   • The paper lacks details on the construction process.
   • How different data selection strategies influence the performance is unclear.
   • The construction needs the data to have the same domain as the target evaluation domain, which is difficult to scale/generalise to estimate uncertainty in real applications. The generalisation issue has been revealed by the performance decreases in Table 4.

2. The "external insight-guided method" and "lookers-on see most of the game" refer to using an external model to estimate the possibility of correctly answering the given question. The practical method is restricted by looking into a small constructed in-domain dataset and finetuning a RoBERTa-based model to estimate such probability. There could be different methods to discuss if the method focuses on estimating the probability of correctly answering the given question.

3. It is unclear how the method can be applied in non-question-answering tasks.

4. The paper discusses the preliminary in 2.5 pages, mixing with existing methods and the author's opinions. Most parts are not related to the proposed method, which confuses readers about the paper's contribution.

1. Missing Baselines: This paper proposes a post-processing calibration method that requires task-specific training data, yet the baselines used for comparison are confidence elicitation techniques (pre-processing methods) that do not require training data, making the comparison potentially unfair. Established post-processing techniques like isotonic regression [1], Platt scaling [1], and recalibration with Bayesian neural networks [2] should be considered. Additionally, existing work has explored training an external calibrator specifically for LLMs [3]. A more balanced evaluation would fix the confidence estimation method and compare post-processing techniques that require training. The authors should cite relevant work, include applicable baselines, and justify any inapplicable baselines.
2. Generalizability Concerns: The authors claim generalizability across domains and models, but the approach’s intuition does not clearly support this. The corrector learns from an LLM’s biases on a specific task, so it’s unclear why it should generalize across models and tasks without similar biases. The current generalizability may stem from both tasks being relatively easy factoid QA, where models often exhibit comparable confidence biases. To substantiate generalizability, the authors should include more diverse tasks, such as those with varying difficulty (e.g., math), and test across models with different capabilities.
3. Lack of Method Analysis: The rationale behind this approach and its specific design remains unclear. The corrector functions as an evaluator trained on LLM data, yet it’s unclear:

• Why use an external model as the corrector rather than training the original model (or classifier based on the original model's hidden states)?
• How does this corrector complement the original uncertainty score? If given full weight, the corrector could solely determine the score—does improvement come from relying more on the corrector, or is there genuine complementarity?
• The paper claims the corrector mitigates the original model’s biases, but it’s unclear what specific biases (over-confidence, under-confidence) are present and addressed. An analysis of the corrector’s impact on these biases would clarify its effectiveness.
• The method’s success seems highly dependent on the corrector’s performance. Would substituting it with a strong LLM evaluator (e.g., GPT-4o) yield similar or superior results?

4. Excessive Conceptual Background: The paper devotes two pages (Section 2) to defining uncertainty concepts, which are widely understood and don’t directly influence the methodology or offer theoretical bounds. This lengthy section could distract readers and reduce space for more critical analysis and intuitive explanations of the method.

[1] Chuan Guo, et al. "On calibration of Modern Neural Networks". https://arxiv.org/abs/1706.04599
[2] Juan Maronas, et al. "Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks" https://arxiv.org/abs/1908.08972
[3] Sabrina J. Mielke, et al. "Reducing conversational agents' overconfidence through linguistic calibration" https://arxiv.org/abs/2012.14983

1. Lacking comparisons of internal state-based methods (Azaria et al., [1]; Kapoor et al., [2]). Both approaches use a probing network to estimate model confidence from internal states. Since this work also involves training a corrector using additional constructed data, including these methods as comparisons would be valuable. Besides, Liu et al. [3] generated their training set in a manner similar to this work and introduced a calibration mechanism to adjust logits, reducing the novelty of the proposed data construction process. Additionally, Ulmer et al. [4] propose an external model to estimate the confidence of LLM outputs, closely resembling the proposed method, which also trains an external model for confidence prediction. This potentially diminishes the novelty of the external corrector.
2. The theoretical analysis in Section 2.3 states the limitations of using logits to estimate uncertainty in cases of over-confidence and under-confidence, but this is not applicable to other prior methods.

• [1] Azaria, A. and Mitchell, T., 2023. The internal state of an LLM knows when it's lying.
• [2] Kapoor, S., Gruver, N., Roberts, M., Collins, K., Pal, A., Bhatt, U., Weller, A., Dooley, S., Goldblum, M. and Wilson, A.G., 2024. Large Language Models Must Be Taught to Know What They Don't Know.
• [3] Liu, X., Khalifa, M. and Wang, L., 2024. LitCab: Lightweight Language Model Calibration over Short-and Long-form Responses.
• [4] Ulmer, D., Gubri, M., Lee, H., Yun, S. and Oh, S.J., 2024. Calibrating Large Language Models Using Their Generations Only.

• (Major) Lack of novelty:

  • The mathematical formalization of overconfidence and underconfidence is not very novel since it is a known issue in the literature. Formalizing it into a mathematical framework is a nice addition, but it's not clear its purpose since it is discarded soon after its definition and not used in the following sections.
  • The proposed Corrector is not novel since training an accuracy classifier is a known practice in the literature (e.g, p(IK), Accuracy Probes) as well as training a more advanced classifier like BERT

• (Critical) The evaluation is quite weak, not thorough and with several issues:

  • The results proposed for RBS baseline methods seem suspiciously low and in contrast with known results in the literature. For example, on TriviaQA AUROC results of base methods are close to or below the 0.5 baselines in contrast with known results in the literature (https://aclanthology.org/2023.emnlp-demo.41.pdf , https://arxiv.org/abs/2406.15627 , https://www.nature.com/articles/s41586-024-07421-0). Please double-check your evaluation protocol.
  • Missing trivial baselines. The paper is missing several trivial single-sample logit-based baselines like Perplexity, Maximum Sequence Probability and Mean Token Entropy. Please take a look here and add these baselines (https://aclanthology.org/2023.emnlp-demo.41.pdf and https://arxiv.org/abs/2406.15627).
  • Missing more advanced baselines. The method proposed is based on training a classifier based on the correctness score of the base model on the task. However several methods in this same class (trained methods) are not evaluated and compared (e.g, p(IK), Accuracy Probes). You should add these methods to your evaluation.
  • The results are presented in a weird way and no real comparison with other methods has been performed. Your method is presented as a method to boost a base score with a convex combination of your trained classifier. However, you did not compare with convex combinations of other baseline methods. The performance of the Corrector without the convex combination is not reported.
  • Your method has different requirements (training FLOPs, hparam tuning via grid search) that other simpler methods do not have. You did not take into consideration these different requirements when comparing methods and this analysis has been completely overlooked. You mention to optimize $w^*$ via grid search, but you provide no indication of the extensiveness of this search that is not performed on other methods. Your comparison should be fair and you should compare methods with equal requirements.
  • The results on CBS show minimal improvement w.r.t. to baseline methods on Llama-3 even though your method uses training. You should also report the confidence interval of your estimates to understand whether these improvements are meaningful. Performance on OPT-2.7B is misleading since the base model is not good enough and UE base methods rely on its logits.
  • Results are just on two datasets (TriviaQA and SciQA) and with two models (OPT- 2.7B, LLaMA-3-8B-Instruct). It's difficult to conclude something with this little evidence. It's not clear how the method would perform on different datasets and models. Furthermore, it's unclear whether the method scales with better base models compared to other approaches like p(True). This is important since UE methods rely on the model logits.

• (Minor) The paper contains several inaccuracies:

  • It's not clear the purpose of sections 2.1 . Framing uncertainty as a classification task (section 2.1) is the standard practice in the literature and you never clearly propose the definitions of relative uncertainty and absolute uncertainty.
  • I don't agree with equation 2. Equation 2 implies that uncertainty is the complement of the probability given by the model for a specific prompt. This is not always the case because some Uncertainty Estimation methods are not defined as probability (e.g., INSIDE).
  • I think equations 8 and 9 are wrong. Equation 9 should be $SE(x,\theta) = -\frac{1}{C} \sum_{i=1}^{C} P(c_i|x,\theta) \ln P(c_i|x,\theta)$ . Please check both of them
  • "internal state-based methods often lack stable theoretical frameworks". This is not true, there are some methods like INSIDE that have a "theoretical frameworks" that it's not based on output probabilities for trivial reasons. It's not clear what is your definition of "stable theoretical frameworks".
  • "showing their performance [internal state-based methods and verbalized methods] generally falling short of advanced logits-based methods". This is in contrast with results in the literature (e.g., INSIDE)

• (Minor) The code proposed is missing several parts and files to understand how the evaluation has been performed, it's not usable and contains not-anonymized paths.

• (Minor) Your template is broken since it does not show lines, this is possibly due to compilation with latex errors.

1. The datasets/evaluation setup provides insufficient evidence to support the effectiveness of the corrector model. It seems fairly trivial that training a classification model on labeled examples would improve relative uncertainty scores, which are strongly correlated to the misclassification error for that domain and task. However, doing so worsens its objectivity in providing any complementary signal to the task performance. Moreover, the generalization of data domains shows negligible improvements across domains, which contradicts the claim that the corrector model or the method is general-purpose.

2. The writing of certain sections could be improved for clarity. For instance, the terminology of "representative" vs "challenging" is a confusing way of categorizing baseline methods in the context. Additionally, the claim that the relatively lower uncertainty scores of the former category are due to its "ease of implementation" isn't sufficiently justified. The experimental setup section also doesn't elaborate on the key information about the training data and uncertainty correction steps. It is also unclear from the results section how the proposed method addresses the over/under-confidence issue in LLMs over existing baselines, which makes the overall storyline less coherent.