Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting

Thank you for your effort and valuable comments and feedback! We would like to make some clarifications and answer your questions below:

W1: The framework of 4.2 and 4.3 is wrong. We want good calibration performance for alignment models but not necessarily the same answer uncertainty as the same pre-trained LMs.

A: Thanks for pointing this out! We agree with your statement that changing the answer uncertainty does not necessarily cause overconfidence and miscalibration. Indeed, keeping the answer uncertainty of the pre-trained LMs is not desired during alignment since updating it could also boost the accuracy. Hence, the claim (such as "Ideally, ...") in our initial submission is inaccurate and inappropriate, and we removed them.

Here, we want to make some clarifications about the somewhat convoluted phenomenon when aligning pre-trained LMs. The key point here is changing the answer uncertainty in some human preference dialog data during alignment would cause overconfidence in MCQs like MMLU, which is usually beyond the scope of and not present in the dataset of alignment. This problem is similar to the in-distribution (ID) and out-of-distribution (OOD) settings, where optimizing the answer uncertainty on the ID dataset (in this case, the human preference data) would cause overconfident on the OOD dataset (in this case, the MCQs that is not presented in the alignment data). Tuning the answer uncertainty would result in good accuracy and calibration in the ID dataset, but the accuracy of the OOD dataset is basically unchanged, and the confidence is affected by the alignment process. This is especially true for larger LMs (e.g., Llama-2-70B in our experiments), where the alignment process does not improve the accuracy of MMLU but only increases confidence.

In Section 4.2 of our revised version, we demonstrate that the common alignment processes update the LMs' answer & format uncertainty simultaneously and cause overconfidence in MCQs, which is undesired since the alignment process does not involve much domain knowledge required for performing the MCQs in our evaluation. In Section 4.3, we add new synthetic schemes using pair-wise preference data with DPO and show that we could increase the accuracy of the synthetic task (illustrated in Table 1 in Appendix C.4) while preserving the calibration and changing format preference on MMLU.

W2: The experiments setup in Section 3 is unclear.

A: Thank you for asking this. We add a reference in Section 3.1 to the detailed evaluation protocol in Appendix B.1. The accuracy, ECE, and confidence are calculated based on the target generation position for a choice letter (no matter which choice format, i.e., A or (A), it is). So, these three metrics are computed based on one position that is supposed to be a choice letter.

As for the probability of the format identifier, for an autoregressive LM, we could get all conditional probabilities $p(x_i|x_{<i})$ for all $x_i, i \in [2, L+1]$ within a single run of inference on the input, where $L$ is the length of the input text. Hence, we could get the probability of the format identifier without letting the LMs generate it by themselves.

W3: Equation 6 is not great.

A: Thank you for your carefulness! We have fixed it.

Q1: How does the proposed TS technique affect MC accuracy?

A: Temperature scaling preserves the accuracy since the argmax of the scaled distribution is still the same. The cost of TS is that it requires an additional calibration set for optimizing the temperature, which may not be available for all cases.

Q2: The binary-class QQP task will wreck the performance of the 4-class MMLU task.

A: That's a great critical point. In response to that, we changed the dataset in Section 4.3 to a synthetic four-class MCQ dataset, where the LM is asked to pick the word that appeared in the question from four given options.

Q3: It would be interesting to see Llama-1 + Format, ICL.

A: We present the results here (we use the setup in our revised version, which is conducted with Llama-1 7B):

The result shows that the performance of pre-trained Llama-1 and Llama-1 + Format has similar performance under ICL.

Underspecified experiment settings

W5: How the ICL samples are selected is not specified; it is also unclear if the observations in this paper hold when varying the demonstrations in ICL.

A: Thank you for asking this. In the initial submission, we put the details about the selection of ICL samples and prompt sensitivity analysis in the Appendix without referring to them in the main text. In our revised version, we add the reference to these details in Section 3.1. For a short overview, we perform all evaluations with three sets of randomly selected in-context examples, and the conclusion is still valid. We also find that larger pre-trained LMs are more robust to the changes of ICL examples.

W6: The SFT and LPF datasets used in Figure 5 are not specified. It is thus unclear if the SFT dataset has a fixed or preferred format.

A: We add a reference to the detailed information of these LMs in Appendix B.2 in our revised manuscript. The LMs in Fig.5 adopt common preference data similar to the aligned LMs in Section 3 (e.g., dialogs) for alignment. We use the official checkpoints to evaluate them on MCQs.

W7: Fig.5 does not explain why there are there are multiple points for Llama-2 between pre-trained and SFT models.

A: They indicate different training steps during SFT. In our revised version, we replaced the Llama-2 with another model that employs DPO for the LPF stage for better demonstration.

Readability of Figures

W8: Fig.4 and Fig.7 are unreadable

A: Sorry for the inconvenience. We improved the readability of all figures in our manuscript.

About Section 4.3

W9: The method proposed in Section 4.3 is not practical, highly limited to the MCQ setting, and not straightforward to adapt to other settings. Besides, it makes us unable to add new knowledge to the LLM during SFT.

A: Thank you for asking this! We would like to clarify that the alignment schemes in Section 4.3 do not serve as novel practical solutions. Instead, by optimizing the answer and format uncertainty separately, their role is to help understand the source of miscalibration during alignment and to support that altered answer uncertainty during alignment will cause overconfidence in MCQs. However, our initial submission does not convey this point well, and we rewrite this section thoroughly.

Besides, in our revised version, we add three pairwise preference learning schemes with DPO to make it closer to realistic settings and show that format learning with DPO could add new knowledge to the LLM (illustrated in Table 1 in Appendix C.4 due to space limitation) while preserving the calibration and changing format preference on MMLU.

Question

Q1: Have you considered instructing the LLMs to answer in the parentheses format to eliminate the format uncertainty instead of using ICL?

A: That's a great question! We tried this by adding the sentence like Begin your response by selecting one option from [(A),(B),(C),(D)] in the instruction, and we have the following ZSL results on MMLU (we mark the setting with such sentence as + Hint):

For choice format A:

For choice format (A):

While the hints could partly eliminate the format uncertainty of LMs, it is not as effective as in-context demonstrations for the LMs we evaluate.

Thank you for your devotion and careful review! We address the weaknesses and answer your question below:

Unclear sentences and claims

W1: The sentence Pre-trained LMs are calibrated in-context learners is confusing.

A: Sorry for the confusion. We want to highlight the fact that pre-trained LMs need ICL to be well-calibrated compared to ZSL. We revise the sentence to Pre-trained LMs achieve good calibration by ICL.

W2: The sentence Format identifier decomposes format preference in MCQs and the corresponding paragraph is incomprehensible.

A: We change the sentence to The probability of format identifier indicates the format preference of LMs in MCQs. and we rewrite the corresponding paragraph. The main point we want to convey is the difference between the choice format A and (A), where the choice letter in the former case is responsible for representing both the answer and the format of MCQs, whereas in the latter case, the format preference is separate from the choice letters.

W3: Proposition 1 is unclear. I would expect a proposition to be something that can be derived from rigorous mathematical proof.

A: Thank you for pointing it out. We admit that using propositions here is inappropriate, and we reframe this part to improve clarity. The revised version replaces the proposition with an assumption for pre-trained LMs that comes from the empirical observations that they have well-calibrated answer uncertainty $p_{\theta}^{\rm{PT}}(y_c|x, F_{\rm{MC}})$. And we intuitively and briefly discuss how ICL estimates $p_{\theta}^{\rm{PT}}(y_c|x, F_{\rm{MC}})$ with $p_{\theta}^{\rm{PT}}(y_c|x, S_K)$.

To achieve this, ICL should (1) provide the signal for the format $F_{MC}$, i.e., $p_{\theta}^{\rm{PT}}(F_{\rm{MC}}|x, S_K) = 1$; (2) have minimal impact on the answer uncertainty of the test example, i.e., $p_{\theta}^{\rm{PT}}(y_c|x, F_{\rm{MC}}) \approx p_{\theta}^{\rm{PT}}(y_c|x, S_K, F_{\rm{MC}})$. A rigorous demonstration of (2) would require introducing a number of premises [1]. Here, we just utilize the intuition that when the LMs comprehend the MCQ task, their decision to the test example will not be influenced by the ICL examples since they are independent of each other.

[1] An Explanation of In-context Learning as Implicit Bayesian Inference (Xie, Sang Michael, et al., ICLR 2022).

W4: Proposition 2 is unclear either.

A: We admit that Proposition 2 could not provide a solid claim for explaining the source of the miscalibration of aligned LMs, and we remove that. The key point here is changing the answer uncertainty in some human preference dialog data during alignment would cause overconfidence in MCQs like MMLU, which is usually beyond the scope of and not present in the dataset of alignment. This is analogous to the problem of in-distribution (ID) and out-of-distribution (OOD), where optimizing the answer uncertainty on the ID dataset (in this case, the alignment dataset) will cause overconfidence on the OOD dataset (in this case, the general MCQs). Hence, for the ID dataset, the changed answer uncertainty and increased accuracy may result in a calibrated model, while for the OOD dataset, the accuracy remains unchanged, but the answer uncertainty is altered toward the direction of overconfidence. That's basically what happens when aligning the pre-trained LM and could not be demonstrated in the way in the original Proposition 2.

In the revised version, we demonstrate this phenomenon based on the existing empirical results and support the claim that altered answer uncertainty during alignment will cause overconfidence in MCQs with our synthetic alignment schemes in Section 4.3.

Thank you so much for the discussion, which we believe is extremely helpful to further improve our work. We are also glad to see our revisions are recognized.

We will try our best to address your remaining concerns and keep refining our work based on your valuable feedback. Here is our response:

Why can we assume the ICL examples do not affect the accuracy when the model understands the task?

A: First, let's qualify the scope to ICL based on general MCQs, which is exactly the setting in our work. The fact that ICL examples have a marginal effect on the accuracy of MCQs has also been established by the previous empirical study [2]. The empirical study in [2] extensively demonstrates that randomizing the input-label mappings of ICL examples has a marginal impact on the accuracy of MCQs compared to the cases of using correct mappings, which, in our opinion, suggests that When the LM understands the multiple-choice task, the LM's prediction under ICL would be largely independent of ICL examples. Moreover, to better support the intuition that $p_{\theta}^{PT}(y_c |x, F_{MC})\approx p_{\theta}^{PT}(y_c |x, F_{MC}, S_{K})$, we add an additional empirical study in Appendix C.4 by using task-irrelevant MCQs as $S_K$ (refer to ICL-Mismatch) and compare the performance with the normal ICL. Here we present a brief overview of the results:

Results on MMLU validation set under the choice format (A):

As shown in the result, the effects of ICL on the accuracy and confidence of pre-trained LMs are still valid when using task-irrelevant $S_K$ that corresponds to the correct format $F_{\rm{MC}}$. Although the task-relevant $S_K$ does provide more positive results in accuracy, it is not that significant compared to the totally task-irrelevant ones. Besides, the confidence in these two settings exhibits a strong consistency compared and is clearly distinguished from ZSL. We hope the additional results could help to convey our intuitive explanation better.

The approximation $p_{\theta}^{PT}(y_c |x, F_{MC})\approx p_{\theta}^{PT}(y_c |x, F_{MC}, S_{K})$ can not be established by solely citing [1]. I cannot agree with the statements here if this approximation cannot be rigorously verified.

A: Thank you for pointing it out. We revised this by only referring to the established empirical evidence [2] in Section 4.1 and adding a clearer discussion about [1] and [2] in Appendix C.4.

[1] An Explanation of In-context Learning as Implicit Bayesian Inference (Xie et al., ICLR 2022).

[2] Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (Min et al., EMNLP 2022)

I also disagree that ICL examples and the test sample are independent.

A: We admit that it is not a rigorous way to support our intuition, and we avoid using such an expression in the paper.

There has been a lot of evidence showing that ICL is very unstable due to the demonstrations.

A: Indeed, ICL does exhibit different biases in specific scenarios [3] or on some manually curated tasks where the LMs must rely on the in-context examples to be correct [4]. However, when we qualify the setting to ICL for general, self-contained MCQs, we think there would be enough evidence to support the intuition that The in-context examples have a marginal effect on LM's answer uncertainty for MCQs.

[3] Calibrate Before Use: Improving Few-Shot Performance of Language Models (Zhao et al., ICML 2021)

[4] Larger language models do in-context learning differently (Wei et al, 2023)

Overall, in order to make the intuition $p_{\theta}^{PT}(y_c |x, F_{MC})\approx p_{\theta}^{PT}(y_c |x, F_{MC}, S_{K})$ more sound, we revise our presentation by addressing the focus on ICL with general MCQs, adding the relevant empirical study as a reference, and using extra empirical evidence to support it, we also updated our manuscript accordingly. We hope our response could ease your concerns.

Again, we appreciate your rigorous attitude and active engagement, which helps us keep polishing our work. We look forward to your further feedback and would be happy to answer any questions and take suggestions.

Thanks for your effort and valuable review! We address the weaknesses and clarify the questions below:

W1: The scope is limited to MCQ, and whether the findings will generalize to free-form generation is unclear.

A: Indeed, our main concern is the calibration of LMs in MCQs, and we do not expect our findings to be naturally extended to free-form generation. We focus on understanding and mitigating the degradation of calibration from pre-trained to aligned LMs with logit-based UQ that originate from the evaluation of MCQs, where the LM's confidence is well-defined [1, 2]. For free-form generation, the pressing issue is how to quantify its confidence rationally, and logit-based UQ may not be a good solution. Hence, our work is somewhat orthogonal to the work related to UQ for free-form generation.

W2: Most of the findings in the paper are expected and have already been established in the literature. We should expect ICL to mitigate the miscalibration issues with answer format uncertainty.

A: While some of our work's findings and results may align with expectations and intuitions, we believe that we still need to present them rigorously and comprehensively. Besides, the focus of our study is markedly different from existing work. Here is a brief overview of previous work related to logit-based UQ with MCQs in the context of pre-trained and aligned LMs:

• [1]: Demonstrate the calibrated performance of the pre-trained LMs in MCQs and show that larger LMs with ICL are well-calibrated, with ECE and the reliability diagram as metrics. Show that the aligned LMs suffer from miscalibration and propose the TS with a constant temperature method.

• [2]: Show that aligning GPT-4 with PPO degrades the ECE on a subset of MMLU with a single pair of reliability diagrams.

• [3]: Present a comprehensive benchmark for all available LMs and compare different schemes (e.g., using the likelihood of sentence or choice letters) for logit-based calibration evaluation.

It follows that even with the simple MCQ setting, there are still questions to be answered, such as how aligning LMs with human-preference data leads them to be overconfident in MCQs, which is not quite straightforward to answer.

Our study addresses this question by specifically pairing pre-trained LMs with their aligned counterparts for comparative evaluation. We report additional metrics previously unexplored in depth by other research, employ a framework for decomposing answer and format uncertainty in LMs, study different alignment stages, and design synthetic align schemes to investigate the different roles of the alignment process, which helps enhance the understanding of the source of miscalibration in aligned LMs.

Besides, our work shows that ICL could only mitigate the miscalibration of pre-trained LMs caused by underconfidence while having a marginal effect on aligned LMs' miscalibration due to overconfidence, which suggests the alignment process impairs the answer uncertainty of the pre-trained LMs. Again, we believe that this kind of difference between pre-trained and aligned LMs is not obvious and needs to be carefully discussed across model sizes, alignment algorithms, and tasks.

[1] Language Models (Mostly) Know What They Know (Kadavath, Saurav, et al., 2023).

[2] GPT-4 Technical Report (OpenAI, 2023).

[3] Holistic Evaluation of Language Models (Liang, Percy, et al., TMLR 2023).

W3: Formalizing some of the paper findings as propositions is unnecessary and does not add anything to the paper. I would expect the paper to briefly discuss these expected findings and then move on to attacking the issue.

A: Thank you for your feedback! We admit that the propositions are unnecessary and may cause confusion, and we accept your suggestion to remove Proposition 4.2 and re-organize the Proposition 4.1 part to an empirically established assumption with some intuitive discussions. We still preserve the formalization of the decomposition for a concrete demonstration of "answer" and "format" and enrich the experiment part with new results in Section 4, where we think careful discussion of the findings is necessary.

Thank you for your supportive comments and constructive reviews! We would like to address the weaknesses below:

W1: The study is restricted to MCQ, and the identified problems and proposed solutions for calibration issues are only applicable to MCQ settings. Some discussion on how the methods could generalize to open-ended generation would be helpful.

A: Thank you for pointing this out. First, for the decomposition of answer & format uncertainty and performing controlled optimization for them, we think this idea has the potential to be generalized in the open-ended generation. Although, as you mentioned, the quantification of format uncertainty, in this case, is not as straightforward as it is in MCQs. Nevertheless, we could still try to optimize the format preference only during general alignment processes.

For example, in LPF, we could select the pair of $\boldsymbol{y}_w$ and $\boldsymbol{y}_l$ that share the same semantic meaning while structured with a different format. In our revised version, we design a new synthetic scheme following this idea by letting $\boldsymbol{y}_w$ = (A) and $\boldsymbol{y}_l$ = It's (A) and fine-tune the LM with DPO. The result aligned LM exhibits similar behavior to the original Format scheme using SFT, i.e., preserving the pre-trained LM's calibration on MMLU. Though such an alignment scheme still performs on MCQs, it's closer to the alignment objective on open-ended generation compared to only increasing the likelihood of the format identifier in the original Format scheme.

Second, generalizing the proposed TS technique to open-ended generation scenario depends on whether the premises of the method are met. The key fact to making this approach work is that the pre-trained LMs are well-calibrated while the corresponding aligned LMs are not, which is well-established in MCQs. This would be true for short-form answers where our method could be naturally applied. However, to the best of our knowledge, for open-ended generation, the likelihood of a long answer may not be a proper indicator of LMs' confidence [1], and other forms of confidence quantification, such as linguistic confidence, do not show a clear superiority of pre-trained LMs over aligned LMs [2]. Hence, our proposed TS method may best fit the scenario similar to MCQs where logit-based UQ works well.

[1] Holistic Evaluation of Language Models (Liang, Percy, et al., TMLR 2023) (The ablation study of calibration in MCQs).

[2] Llamas Know What GPTs Don't Show: Surrogate Models for Confidence Estimation. (Shrivastava, V., Liang, P., & Kumar, A, 2023) (Table 6 in Appendix A.6.).

W2: The proposed TS method requires access to the non-aligned LMs and a few examples for every new task that may not always be feasible. Is there some way to generalize this such that we only really need to use the pre-trained model once?

A: We admit that our proposed TS method has the limitations you mentioned, and we also addressed them in the manuscript. In the context of post-hoc calibration, generalizing our task-wise approach to a model-wise one is not trivial since the degree of overconfidence is not quite the same on different tasks. Achieving this goal may need to include extra and potentially heavier procedures, and we would like to explore possibilities in future works.

Thank you again for your support and we sincerely appreciate your participation in the discussion phase. Also, if you don't mind, we respectfully hope you could be able to give us more feedback on your potential concerns, which remain unresolved by our rebuttal. We would be willing to address them.