ConfTuner: Training Large Language Models to Express Their Confidence Verbally

Thanks for your valuable suggestions! We address your main concerns as follows:

W1: Limited exploration of complex contexts

ConfTuner can generalize to more complex contexts. We conduct tests with a more open-ended setup. Specifically, we change the original prompt from {Question + output confidence (0-100)} to {Question + Please express your uncertainty when providing the answer}. This new setting significantly expands the LLM's output space to diverse natural language expressions of uncertainty, such as "I'm fairly certain, but ..." or "I'm not sure" We evaluated these implicit confidences by inputting them to GPT-4o to assess the implied confidence levels. The results of AUROC and ECE are presented in Tables 1 and 2 below, showcasing ConfTuner's robustness in diverse conversational contexts.

Table 1: AUROC

Table 2: ECE

W2: Comparison of black-box calibration methods for GPT-4o.

Based on your suggestion, we further add a black-box calibration baseline, Ensemble [1], which prompts LLMs to generate the top K guesses and their corresponding confidence, then inputs the same prompt multiple times, and finally computes the average confidence. The results are shown in Tables 3 and 4. We can see that ConfTuner has significantly better ECE (by 5.3%) and slightly lower AUROC (by 1.4%). Please note that ConfTuner only uses a smaller model and prompts once, while Ensemble uses GPT-4o and prompts 3 times, which is more expensive.

Table 3: ECE

Table 4: AUROC

Q1: Ablate against a regression head baseline

ConfTuner has two strengths over a regression head: 1) ConfTuner enhances the model's intrinsic calibration ability, thus has better generalization ability. ConfTuner could generalize to out-of-distribution datasets, different forms of confidence levels (Section 4.2), and to diverse and more complex confidence expressions (W1), which cannot be done by a simple regression head. 2) It offers a more seamless and natural way to express confidence.

Brier loss is used for various confidence level tokens instead of a single numeric value, so we further compare the performance of ConfTuner and a classifier on the in-distribution and out-of-distribution datasets. We conducted an experiment where we fed the token logits after "Confidence: " from the last hidden state into a two-layer MLP classifier to predict confidence scores from 0 to 100, trained with Brier loss. The results, presented in Tables 5 and 6 below, compare this approach to ConfTuner. Our findings indicate that the classifier-based method yields poorer performance compared to ConfTuner, especially on OOD datasets.

Table 5: AUROC

Table 6: ECE

Q2: Comparison to models where confidence estimation is derived from logits

We further use a method [3], P(True), where confidence estimation is derived directly from logits of the token “True”. The results of ECE and AUROC are shown in Tables 7 and 8, demonstrating that ConfTuner performs better than P(True).

Table 7: AUROC

Table 8: ECE

Q3: Comparison to Ensemble (pair-rank).

We provide the results of Pair-rank in Tables 9 and 10. The results demonstrate that ConfTuner outperforms Ensemble (Pair-rank).

Table 9: ECE

Table 10: AUROC

Q4: Sensitivity of ConfTuner to varying prompting strategies & Comparison to improved prompt engineering and top-k response aggregation

Even trained only on numeric confidence (0-100%) prompting, ConfTuner could generalize to different confidence elicitation prompting strategies, such as "high/medium/low" (Table 3 in our paper), and implicit confidence elicitation (W1).

We have compared ConfTuner with Ensemble (Tables 1 and 2 in our paper), where the prompting and aggregation strategies are recommended by [1]. We also compare ConfTuner with Ensemble (pair-rank) in Q3. The results show that ConfTuner still achieves better performance.

We further conduct an experiment on ConfTuner + Ensemble, i.e., use the prompting and aggregation strategies of Ensemble for ConfTuner. The results in Tables 11 and 12 show that improved prompting and aggregation strategies could yield better performance for ConfTuner.

Table 11: ECE

Table 12: AUROC

Q5 and Limitation 1: Ablation on Training Distribution Shifts

We further train LLaMA on GSM8K (math problems) instead of HotpotQA (general knowledge from Wikipedia). As shown in Tables 13 and 14. ConfTuner trained on GSM8K performs better on GSM8K and StrategyQA, but worse on HotpotQA, TriviaQA, and TruthfulQA.

Table 13: ECE

Table 14: AUROC

Limitation 2: fairness implications of ConfTuner

The primary focus of this paper is to establish effective verbalized confidence calibration. Analyzing calibration quality across demographic groups or ambiguous queries is a critical topic and a valuable direction for future work. Indeed, [2] finds that many existing calibrators tend to focus on calibrating overall performance, which can lead to miscalibration in underrepresented groups. We believe that methods that more deeply integrate with the LLM’s own capabilities like ours have potential to alleviate this problem, but a deeper analysis is left for future work. We appreciate your suggestion and will include a discussion on fairness implications and potential biases in our revised manuscript.

​​Thanks again for your constructive and valuable suggestions! We'll incorporate them in future revisions.​​ Please let us know if you have any further questions.

[1] Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms

[2] Proximity-Informed Calibration for Deep Neural Networks.

[3] Language Models (Mostly) Know What They Know

Thanks for your positive feedback and valuable suggestions! We address your main concerns as follows:

W1: Concerns of exploiting spurious correlations in (question, answer, confidence) tuples and high out-of-distribution calibration errors.

The absolute levels of metrics on a given dataset depend significantly on the dataset’s difficulty and the model’s accuracy on that task, which vary from one dataset to another. However, in many cases, the improvements contributed by our approach over the Base model on out-of-distribution cases are often better than on in-distribution ones. The results are shown in Table 1 below. Under LLaMA and Qwen models, the AUROC improvements on OOD datasets generally surpass those on the in-distribution dataset (HotpotQA). This suggests that ConfTuner is not merely exploiting distributional similarities in the training data but is learning robust confidence calibration capabilities.

ConfTuner not only generalizes to OOD datasets, but also generalizes to different forms of confidence expressions (e.g., training on confidence expressed as percentages, and then expressing confidence linguistically as high/medium/low at test time), as demonstrated in Section 4.2. This reinforces that ConfTuner has internalized calibration ability rather than relying on spurious correlations.

Furthermore, we conduct a new experiment to show that ConfTuner also generalizes to implicit confidence expressions. Specifically, we modified the original prompt from {Question + output confidence (0-100)} to {Question + Please express your uncertainty when providing the answer}. This new setting significantly expands the LLM's output space, allowing it to generate diverse natural language expressions of uncertainty, such as "I'm fairly certain, but there’s a chance I could be mistaken" or "This is a tough one, so I’d say it’s likely but not guaranteed." We evaluated these implicit confidences by inputting them to GPT-4o to assess the implied confidence levels. The results of AUROC and ECE are presented in Tables 2 and 3 below, demonstrating that ConfTuner enhances the model's intrinsic calibration ability instead of simply learning spurious correlations.

Table 1: Improvement of AUROC compared to the Base model

Table 2: Implicit confidence expression performance (AUROC)

Table 3: Implicit confidence expression performance (ECE)

W2: Comparison to a related work.

Our method focuses on verbalized confidence calibration and differs from the work [1], which uses logits to infer the confidence. Thus, we initially did not include it as a baseline. However, per your request, we have conducted experiments incorporating this baseline (P(True)) on the LLaMA base model. The results of ECE and AUROC in Tables 4 and 5 below show that ConfTuner outperforms P(True) [1]. We will add this baseline to our revised manuscript.

Table 4: Comparison to P(True) (AUROC)

Table 5: Comparison to P(True) (ECE)

We really appreciate your valuable suggestions; they’ll be extremely useful for our future work on revising the manuscript.​​ Please let us know if you have any further questions.

[1] Language Models (Mostly) Know What They Know

Thank you for your kind and constructive feedback on our manuscript! We address your main concerns as follows:

W1 & Q1: Difference between ConfTuner and classification method.

ConfTuner is superior to the classification method for two reasons: first, using LLM to directly verbalize its confidence is more seamless and intuitive than having an external classifier generate confidence scores. Second, by fine-tuning the LLM with a tokenized Brier score, ConfTuner enhances the model's intrinsic calibration ability. This is evidenced by:

1. ConfTuner has good generalization ability to various scenarios and different forms of expressions. ConfTuner achieves significantly lower ECE scores on out-of-distribution datasets, and also generalizes to different forms of confidence levels (Section 4.2). We further compare the performance of ConfTuner and the classification method on the in-distribution and out-of-distribution datasets. We conduct an experiment where we feed the token logits after "Confidence: " from the last hidden state into a two-layer MLP classifier to predict confidence scores from 0 to 100, trained with Brier loss. The results, presented in Tables 1 and 2 below, compare this approach to ConfTuner (based on LLaMA). Our findings indicate that the classifier-based method yields poorer performance compared to ConfTuner, especially on OOD datasets, indicating that the classifier has relatively higher usage of spurious factors that do not generalize well to diverse scenarios.

2. This improved calibration not only boosts trustworthiness but also supports broader LLM capabilities, such as self-correction (Figure 4).

3. ConfTuner also generalizes to implicit confidence expressions, which cannot be done by a classifier. We conduct further tests with a more open-ended setup. Specifically, we modified the original prompt from {Question + Output confidence (0-100)} to {Question + Please express your uncertainty when providing the answer}. This new setting significantly expands the LLM's output space, allowing it to generate diverse natural language expressions of uncertainty, such as "I'm fairly certain, but there’s a chance I could be mistaken" or "This is a tough one, so I’d say it’s likely but not guaranteed." We evaluate these implicit confidences by inputting them to GPT-4o to assess the implied confidence levels. The results of AUROC and ECE are presented in Tables 3 and 4 below, demonstrating that **ConfTuner can handle not only classification-like confidence expressions but also more complex and implicit confidence expressions.

Table 1: Classifier vs ConfTuner comparison (AUROC)

Table 2: Classifier vs ConfTuner comparison (ECE)

Table 3: Implicit confidence expression performance (AUROC)

Table 4: Implicit confidence expression performance (ECE)

W2: Discussion of two related works.

The regression-based methods [1] rely on an external classifier to calibrate confidence, and can not generalize to different forms or implicit confidence expressions. Similarly, using a single "I don’t know" token limits expressiveness to a binary uncertainty indicator [2], lacking the granularity of our approach. By contrast, ConfTuner fine-tunes the LLM itself, enabling more accurate and diverse verbalized confidence expressions. We will include this discussion in our future revision.

Q2: Would outputting the confidence prediction before the answer change how well the method works?

It’s an interesting experiment and could further elucidate the internal mechanics of ConfTuner. We prompt ConfTuner to generate the confidence score prior to providing the answer. The results of AUROC and ECE are presented in Tables 5 and 6. Our findings indicate that outputting confidence before the answer yields poorer performance compared to outputting it afterward, suggesting that the hidden states of the answer tokens are indeed informative about the certainty of the response. And ConfTuner still outperforms Base model when outputting confidence first. We will include this experiment in our revised manuscript.

Table 5: Performance of outputting confidence first (AUROC)

Table 6: Performance of outputting confidence first (ECE)

Thanks again for your valuable suggestions, which are extremely useful for our future work on revising the manuscript.​​ Feel free to reach out if anything comes up. Please let us know if anything needs further clarification!

[1] Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approach

[2] I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Token

Thank you sincerely for your ​​thoughtful and constructive feedback​​ throughout this process! Your insights and suggestions will ​​significantly enhance our work​​. All new experiments and discussions during this review cycle will be ​​rigorously integrated​​ into the final paper. Specifically, we will:

​​1. Expand Section 4.5​​ to discuss ConfTuner's underlying mechanisms (comparison to classifier), emphasizing its ​​system-level confidence calibration capabilities​​ beyond verbalized uncertainty expressions.

2. ​​Incorporate new analysis​​ on mathematical robustness (GSM8K results in Table 3) to reinforce our reliability claims.

We are profoundly grateful for your decision to ​​raise the score to 5 (Accept)​​. Your engagement has been ​​exceptionally valuable​​ in advancing this research.

Thank you for your encouraging feedback and insightful comments! We address your main concerns as follows:

W1: Importance of confidence estimation vs core LLM advancement​

Improving the general capabilities of LLMs is indeed important. However, even as LLMs become more capable, there will inevitably be questions that exceed their knowledge or reasoning abilities. In such cases, the model’s tendency to be overconfident or fail to accurately express their uncertainty is a fundamental limitation, undermining user trust and making it hard for downstream systems or users to know when caution is warranted.

Moreover, the ability to express calibrated confidence can also support better reasoning and decision-making. For instance, in retrieval-augmented generation (RAG) or tool-use settings, a model that can identify and verbalize its uncertainty within its chain-of-thought could trigger additional information-seeking steps, such as searching for more evidence or invoking a tool, to validate or revise its initial answer, allowing for iterative self-correction and better performance.

Q1: LoRA setting.

The alpha value is set to 32, with adapters applied to all layers - specifically attached to the query and value projection modules. We will incorporate this implementation detail in our manuscript revision.

Q2: Could the proposed loss be combined with other losses to improve both the quality of the answers and the calibration in the same training?

Yes. As a plug-and-play loss, the Brier score is highly flexible and compatible with standard LLM fine-tuning pipelines. Brier loss can be seamlessly combined with other loss functions, such as those optimizing answer quality, to enhance both calibration and response accuracy during training simultaneously. In this work, we focused exclusively on improving verbalized confidence calibration, so we only use the Brier loss. Future work could explore integrating it with answer-focused losses to achieve dual improvements.

Q3: Does the confidence calibration carry over to tasks/prompts where there aren’t explicit instructions to evaluate its confidence?

ConfTuner could generalize to scenarios without explicit instructions. We conduct new experiments on ConfTuner (based on LLaMA). Specifically, we change the original prompt from {Question + output confidence (0-100)} to {Question + Please express your uncertainty when providing the answer}. Under this instruction, the LLM also produces implicit confidence expressions, such as "I'm fairly certain, but there’s a chance I could be mistaken" or "This is a tough one, so I’d say it’s likely but not guaranteed." We evaluate these implicit confidence by inputting them to GPT-4o to assess the implied confidence levels (0-100). The results of AUROC and ECE are shown in Table 1 and Table 2, demonstrating that implicit confidence calibration of ConfTuner is comparable to explicit confidence calibration.

Table 1: AUROC

Table 2: ECE

Thank you again for your supportive review. Your constructive comments are greatly appreciated for revising our paper. We welcome any additional questions you might have.