Response to Reviewer GoW5

We appreciate the reviewer for the insightful and detailed comments. Please find our response below:

1. [W1] Definition of LLM confidence distribution

Thank you for this valuable suggestion. In the main part, we focus on the multiple-choice QA task, where an LLM's confidence refers to the softmax probability of the token(s) corresponding to the chosen answer option (e.g., A, B, C, D) [1,2]. In Section 5 (discussion), we extend our method to open-ended QA tasks and employ the P(True) approach [3], where models first generate answers and then assess the probability, P(True), that their responses are correct. Following the suggestion, we will explicitly introduce the definition in Section 2 of the final version.

2. [W2] Clarification on the repetition in Eq6 and Eq7

Thank you for the detailed review. We clarify that Eq. (6) defines the loss function for DACA's temperature scaling, while Eq. (7) presents the optimization objective for extending DACA to other post-hoc calibration methods, such as vector scaling (VS) and matrix scaling (MS). Specifically, Eq. (6) is a special case of Eq. (7), where the parameter $\theta$ represents either a scalar temperature $\tau$ for temperature scaling (TS) or vector/matrix parameters for VS and MS. Without the generalized objective in Eq. (7), the extension to various calibration methods might be ambigurous to readers.

3. [W3, W7] Clarification on applicability

Thank you for raising this concern. Below, we address each point regarding the applicability of our method with clear explanations.

• "white-box". This work focuses on calibrating logit-based confidence [1,2,3,4], which is accessible in both open-source and many API-based LLMs (e.g., GPT-4o, Grok-4 and Llama-4-Maverick). Therefore, our method is broadly applicable to various LLMs (including APIs) with access to logits.
• "require access to the pre and post training version". We clarify that the effectiveness of DACA is agnostic to the choice of PLMs (See Tables 2 and 10), so we do not require access to its pretraining version for calibrating a PoLM. For example, we can leverage a small-scale, open-sourced PLMs (e.g., Qwen2.5-7B) to calibrate large-scale PoLMs (and APIs) such as GPT-4o and Deepseek-V3. This approach reduces the ECE of GPT-4o on MedMCQA from 21.23% to 7.82%, demonstrating that access to the pretrained version of the PoLM is not required.
• "multiple inference of the pretrained model". We clarify that post-hoc calibration methods, including TS and our DACA, perform optimization (Eq.(7)) during the calibration phase, using a small validation set to tune scaling parameters (e.g., $\tau$). Our method only queries the pretrained model during this phase, not for every test instance. In deployment, DACA incurs no additional computational cost compared to standard post-hoc calibration methods like temperature scaling.

4. [W4] Clarification on the limitation to multiple option QA

In Section 5 (Discussion), we extend the proposed method to open-ended QA tasks. Specically, we employ the P(True) approach [4], where models first generate answers and then assess the probability, P(True), that their responses are correct. Here, "Agreement" occurs when both models give consistent "Yes" or "No" answers. Figure 4 illustrates the effectiveness of DACA in these tasks, notably reducing the Expected Calibration Error (ECE) of Qwen2.5-32B-Instruct on TruthfulQA from 30.955% to 5.244%.

5. [W5] Additional results with larger PLMs

Thank you for this valuable suggestion. We conduct new experiments by employing larger PLMs to calibrate GPT-4o, and present the results on MedMCQA in the table below (including the original results of Table 2 for convenience). The results indicate that leveraging more powerful PLMs can indeed further enhance the calibration of the PoLM.

6. [W6] Discussion of Linguistic Confidence

Thank you for introducing the related works. We agree that "Linguistic Confidence" is an interesting and promising approach to expressing LLM uncertainty. In this work, we focus on calibrating logit-based confidence and reveal the benefits of PLMs in confidence calibration. For future work, it will be an interesting direction to design calibration methods for verbalized or linguistic confidence. We will include a discussion of this concept with proper references in Related work of our final version.

7. [Q1] Why DACA can outperform TS with labels?

Thank you for the insightful question. We conjecture that this pheonomenon might be related to the size of the calibration set. Previous studies [5,6] have shown that Temperature Scaling (TS) performs poorly with limited calibration data. Table 1 presents the average performance across 57 MMLU subjects, where TS’s scaling parameter $\tau$ is separately tuned on each subject’s validation set. Many subjects have small validation sets (e.g., $\leq 20$ samples), leading to degraded performance of TS. To validate this, we report the performance of Qwen2.5-7B-Instruct using TS and DACA across subjects with varying calibration set sizes in the table below. The results confirm that DACA outperforms TS in subjects with limited calibration samples, while TS excels when abundant data is available. This highlights DACA’s superior data efficiency, making it particularly valuable in data-scarce scenarios.

[1] Shen, Maohao, et al. "Thermometer: Towards Universal Calibration for Large Language Models." ICML (2024).

[2] Achiam, Josh, et al. "Gpt-4 technical report." (2023).

[3] Zhu, Chiwei, et al. "On the Calibration of Large Language Models and Alignment." EMNLP (2023).

[4] Kadavath, Saurav, et al. "Language models (mostly) know what they know." (2022).

[5] Mozafari, Azadeh Sadat, et al. "Attended temperature scaling: a practical approach for calibrating deep neural networks." (2018).

[6] Liang, Gongbo, et al. "Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification." BMVC (2020).

Response to Reviewer uRXE

Thank you for the constructive feedback. Please find our response below:

1. [W1] Clarification on the inference costs

Thank you for highlighting this concern. We clarify that post-hoc calibration methods, including TS and our DACA, perform optimization (Eq.(7)) during the calibration phase, using a small validation set to tune scaling parameters (e.g., $\tau$). Our method only queries the pretrained model during this phase, not for every test instance. In deployment, DACA incurs no additional computational cost compared to standard post-hoc calibration methods like temperature scaling. Therefore, we can perform inference on test instances using only the post-trained LLM, without needing to deploy or run the pretrained model alongside it. As for the calibration phase, practitioner can first generate the target confidences with (small-scale) PLMs and then use these to tune the scaling parameters for PoLMs, without need to load both models simultaneously. In summary, our solution is practical in real-world scenorios without introducing heavy computational costs.

2. [W2, Q2] Generalization of DACA across tasks

Thanks for raising the concern. To evaluate the transferability of the temperature parameter learned by DACA, we conducted experiments by training a temperature value on the MMLU dataset and applying it to the MedMCQA and MathQA datasets. The results in the table below demonstrate that DACA consistently reduces the ECE of several PoLMs on both MedMCQA and MathQA, which validates the practical transferability of the learned temperature $\tau$ (despite it might be not the optimal).

Secondly, a more practical scenario is learning a temperaute value on a few tasks and then generalize to other more tasks, similar in spirit to the setting of OOD generalization [3]. We conducted experiments by randomly selecting 10 subjects from the MMLU dataset for calibration and testing on the remaining 47 subjects. The table below presents the mean results with standard deviations across 10 random trials. The results demonstrate that DACA consistently improves the calibration performance of three PoLMs across the 47 unseen subjects, indicating DACA's robust generalization to new tasks and robustness to domain shifts.

As for learning a single $\tau$ for all 57 subjects in MMLU, we conduct new experiments by learning a single $\tau$ on the validation set of all MMLU subjects and present the average results of test set in the table below. The results show that a single $\tau$ indeed significantly improves the calibration performance of PoLMs on all subjects of MMLU.

3. [Q1] Incorporating KL Loss as a post-training constraint

Thank you for this insightful suggestion. We agree that it might be useful to incorporate the KL loss into the post-training process as a regularization. It is worth noting that most popular post-training methods —such as PPO [1] and DPO [2]— indeed employ an explicit or implicit KL-divergence constraint between PoLM and PLM. Their purpose is to improve the training stability by limiting its deviation from the PLM. However, it may introduce a tradeoff: a weaker KL regularization allows the model to better align with preference data, but often at the cost of its calibration performance. Conversely, strong regularization may compromise the performance of preference alignment. As a result, those post-trained models are still poor calibrated (as shown in Table 3), although they are trained with the KL-divergence constraint.

[1] Schulman, John, et al. "Proximal policy optimization algorithms." (2017).

[2] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." NeurIPS (2023).

Response to Reviewer APYD

Thank you for the constructive feedback. Please find our response below:

1. [W1] Dependence on the selected PLMs

Yes, we agree that the calibration level of the chosen PLM may affect DACA's performance. However, prior studies [1,2] demonstrate that most PLMs exhibit significantly better calibration than PoLMs, as validated by Figure 1 of our manuscript. This makes it straightforward to choose an appropriate PLM for implementing DACA. Importantly, the excellent performance of our DACA is not tied to specific PLMs as shown in Table 2. Across various PLMs, including Llama-3-8B, Qwen2.5-7B, and Gemma-3-12B, DACA consistently reduces GPT-4o's ECE from 21.23% to below 8%. This highlights that the choice of PLM does not constrain DACA's practical applicability.

Regarding the term "well-calibrated PLMs" (line 128), we clarify that it refers to the well-calibrated confidence scores of PLMs compared to those of PoLMs, not to restricting PLM selection (as most PLMs perform exceptionally well). In the final version, we will revise the phrase to "align PoLMs' confidence levels with those of PLMs" for greater clarity.

2. [W2, Q1] Transferability of DACA

Thanks for raising the concern. To evaluate the transferability of the temperature parameter learned by DACA, we conducted experiments by training a temperature value on the MMLU dataset and applying it to the MedMCQA and MathQA datasets. The results in the table below demonstrate that DACA consistently reduces the ECE of several PoLMs on both MedMCQA and MathQA, which validates the practical transferability of the learned temperature $\tau$ (despite it might be not the optimal).

Secondly, a more practical scenario is learning a temperaute value on a few tasks and then generalize to other more tasks, similar in spirit to the setting of OOD generalization [3]. We conducted experiments by randomly selecting 10 subjects from the MMLU dataset for calibration and testing on the remaining 47 subjects. The table below presents the mean results with standard deviations across 10 random trials. The results demonstrate that DACA consistently improves the calibration performance of three PoLMs across the 47 unseen subjects, indicating DACA's robust generalization to new tasks and robustness to domain shifts.

As for learning a single $\tau$ for all 57 subjects in MMLU, we conduct new experiments by learning a single $\tau$ on the validation set of all MMLU subjects and present the average results of test set in the table below. The results show that a single $\tau$ indeed significantly improves the calibration performance of PoLMs on all subjects of MMLU.

3. [W3,Q2] The potential limitation of filtering examples

Yes, the filtered examples could potentially enhance calibration performance, offering a promising avenue for future research. However, this limitation does not affect the contribution of this work, which achieves calibration performance comparable to, or surpassing, supervised temperature scaling. Notably, our method still significantly improve the calibration performance even in cases of high disagreement ratio. The table below presents the results of DACA using Qwen2.5-7B-Instruct (PoLM) and Qwen2.5-7B (PLM) on the three MMLU subjects with the highest disagreement ratios. The results show that our method can outperform supervised TS, even when there are only four agreement samples.

Besides, we report the statistics of disagreement sample ratios $\pi$ across 57 MMLU subjects using three models in the following table.

4. [Q3] Definition of agreement in open-ended QA

Thank you for the insightful question. In Section 5 (Discussion), we extend the proposed method to open-ended QA tasks. Specically, we employ the P(True) approach [4], where models first generate answers and then assess the probability, P(True), that their responses are correct. Here, "Agreement" occurs when both models give consistent "Yes" or "No" answers. Figure 4 illustrates the effectiveness of DACA in these tasks, notably reducing the Expected Calibration Error (ECE) of Qwen2.5-32B-Instruct on TruthfulQA from 30.955% to 5.244%. Additionally, a potential direction is to define a soft metrics for the agreement via measuring the semantic distance between the responses of PLM and PoLM, offering greater flexibility for chatbot applications. This will be included in the final version of our discussion.

5. [Q4] Grammar typo

Thank you for pointing this out. We will revise this typo in our final version.

[1] Achiam, Josh, et al. "Gpt-4 technical report." (2023).

[2] Zhu, Chiwei, et al. "On the Calibration of Large Language Models and Alignment." EMNLP (2023).

[3] Liu, Jiashuo, et al. "Towards out-of-distribution generalization: A survey." (2021).

[4] Kadavath, Saurav, et al. "Language models (mostly) know what they know." (2022).

Response to Reviewer 8hAp

Thanks for your positive and valuable suggestions. Please find our response below:

1. [W1] The potential weakness of filtering examples

Thank you for the thorough review. The filtered examples could potentially enhance calibration performance, offering a promising avenue for future research. However, this limitation does not affect the contribution of this work, which achieves calibration performance comparable to, or surpassing, supervised temperature scaling. Notably, our method still significantly improve the calibration performance even in cases of high disagreement ratio. The table below present the results of DACA using Qwen2.5-7B-Instruct (PoLM) and Qwen2.5-7B (PLM) on the three MMLU subjects with the highest disagreement ratios. The results show that our method can outperform supervised TS, even when there are only four agreement samples. Additionally, a promising direction for utilizing filtered examples is to generate target confidences using specialized agents for confidence alignment, which could be particularly valuable in some extreme cases.