Taming Overconfidence in LLMs: Reward Calibration in RLHF

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point. We would appreciate it if you could let us know whether your concerns are addressed by our response.

Q1: Have you considered jointly optimizing (1) and (2) to enforce $y_c$ over $y_r$ as well.

A1: Thank you for your valuable insight. Equation (2) is intended to function as a stand-alone objective. In our experiments, when fine-tuning an existing reward model on our calibration dataset, we did not observe any diminishment in the model’s ability to distinguish between chosen and rejected responses. As shown in Figure 6 of Section 4.2 in the updated PDF, we provide a direct comparison between our calibrated reward model and its pre-calibrated version. Specifically, in the answer_only row, which uses the original RewardBench datasets without modifications, the chosen/rejected preference ratio remains largely unchanged.

Additionally, in Appendix E.8 and Figure 24, we present the training plots of the reward model trained using equations (1) and (2) in combination, compared to training with equation (2) alone. The model trained solely with equation (2) maintains a similar trend of distinguishing between chosen and rejected responses as the model trained with the combined loss.

We also conducted experiments comparing PPO-M using the reward model trained with equation (2) alone versus in combination with equation (1), as shown in Table 13 of the updated manuscript. For your convenience, we have provided the table below.

To summarize:

• The results indicate that PPO-M using the reward model trained with the combined loss does not achieve better calibration results. Moreover, combining equations (1) and (2) would require additional computational resources since we need to process two extra samples. Given these observations, we have chosen to set equation (2) as a standalone loss.

Furthermore, in Figure 24 (middle column), when comparing cwh_reward (the reward for a chosen response with high confidence, $(y_c, h_c)$ and rwl_reward (the reward for a rejected response with low confidence, $(y_r, l_r)$, we observe that the rewards of $(y_c, h_c)$ consistently exceed the rewards of $(y_r, l_r)​$ without the need for an explicit combined loss.

Q4: Regarding PPO-M, is it intended to be an additional objective to equation 1

A4: Thank you for your question. Equation (2) is intended to function as a stand-alone objective. In our experiments, when fine-tuning an existing reward model on our calibration dataset, we did not observe any diminishment in the model’s ability to distinguish between chosen and rejected responses. As shown in Figure 6 of Section 4.2 in the updated PDF, we provide a direct comparison between our calibrated reward model and its pre-calibrated version. Specifically, in the answer_only row, which uses the original RewardBench datasets without modifications, the chosen/rejected preference ratio remains largely unchanged. Additionally, in Appendix E.8 and Figure 24, we present the training plots of the reward model trained using equations (1) and (2) in combination, compared to training with equation (2) alone. The model trained solely with equation (2) maintains a similar trend of distinguishing between chosen and rejected responses to the model trained with the combined loss.

Furthermore, we conducted experiments comparing PPO-M using the reward model trained with equation (2) alone versus in combination with equation (1), as shown in Table 13 of the updated manuscript. For your convenience, we have provided the table below.

The results indicate that PPO-M using the reward model trained with the combined loss does not achieve better calibration results. Moreover, combining equations (1) and (2) would require additional computational resources since we need to process two extra samples. Given these observations, we have chosen to set equation (2) as a standalone loss.

Q5: Line 270: What do you mean by “biased response”

A5: Thank you for pointing this out. We mean that including a confidence score within the response can lead to a biased reward score (as shown in our preliminary study in Sec 2.2). To address this, we first parse and remove the confidence score to obtain a clean (unbiased) response for evaluation.

Thank you for your constructive feedback and comments. We have incorporated the suggested changes into our updated manuscript and will address your concerns below. Please let us know if our response addresses your concerns.

Q1: The conclusion drawn from Figure 2 does not align with the plots. Why are data points so sparse. How many prompts were used in this experiment?

A1: We agree with your observation that pre-RLHF models also show some degree of overconfidence. However, our point is to show that RLHF extensively increases the overconfidence of LLMs. In this experiment, we use all samples from CommonsenseQA validation set (1221 prompts in total), and bin the output confidence scores into 20 intervals (0.05 range per interval). For each bin, we aggregate all responses in that confidence interval to calculate the bin accuracy (with the last bin merging confidence 0.95 and 1.0). Empty bins mean no confidence scores fall into those intervals. RLHF-trained models have more empty bins in the lower confidence range, indicating that they have a stronger bias towards expressing high confidence scores (darker bins mean more samples falling into those bins). In contrast, pre-RLHF models show a broader range of confidence distributions across multiple intervals. This suggests that RLHF tends to make LLMs more overconfident. We have made Figure 2 caption clearer in our revised version of the paper.

Q2: Appending the prompt with a random confidence score to both chosen and rejected answers.

A2: Thank you for your suggestion. We also recognize the potential concern that the reward model may be biased by the confidence-query system prompt, which is often included to enable the model to verbalize its confidence. We have conducted an analysis to evaluate the impact of this prompt on additional 15 reward models, as detailed in Appendix E.2, and Figures 17, 18, 19. The left panel of these figures presents the exact setting as our preliminary experiments, while the right panel explores a new setting on the same set of reward models where the confidence-query prompt is removed, and only random confidence scores are appended to the model responses. To summarize the findings:

• When the confidence-query prompt is removed, under the confidence_reversed setting, where random high scores are appended to rejected responses and low scores to chosen responses, many models continue to assign higher scores to the chosen responses. This contrasts with cases where the confidence-query prompt is included.
• Moreover, when comparing the same responses with high versus low confidence scores, we consistently observe a tendency for the reward models to prefer high-confidence responses regardless of their correctness, indicating that the bias persists.

Thank you for your valuable feedback. We have incorporated the feedback into our updated manuscript and will address your concerns below:

Q1: The “PPO-M” loss could be confusing.

A1: Thank you for raising the concern. To avoid any confusion, we have revised the terminology in the updated version. We rename the loss in Eq. (2) to “CRM” to better reflect its purpose in Calibrated Reward Modeling.

Q2: The use of verbal confidence to measure a model’s confidence in its response may not be ideal.

A2: Thank you for your comment. We recognize that using verbal confidence to measure model confidence, particularly for smaller models under 70B, may have limitations due to their constrained reasoning capabilities. Nonetheless, the use of 7B models is widely adopted in related works within this field [1][2][3]. In our study, we use verbalized confidence because it provides direct feedback in natural language to LLM users without access to logits, and previous studies [4][5] demonstrate that using verbalized confidence generally leads to lower expected calibration error than logit-based methods.

In response to your query about examining the logits for the 0 to 10 range when evaluating verbal confidence, we have conducted further analysis. In Appendix E.5, Figure 23 of the updated PDF, we present density plots of log probability of number 0 to 10, specifically at the locations corresponding to confidence scores on responses for TruthfulQA dataset. To summarize the observations:

• We observe that certain numbers demonstrate notably high density. For example, the number 10 exhibits high density for PPO, while the number 9 shows a similar behavior for PPO-M. The log prob for numbers are not uniformly distributed, which suggests that the model is not randomly generating number values but rather exhibits a tendency to favor certain numbers in its confidence outputs.

[1] Xu, Tianyang, et al. "SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationales." The 2024 Conference on Empirical Methods in Natural Language Processing.

[2] Tao, Shuchang, et al. "When to Trust LLMs: Aligning Confidence with Response Quality." arXiv preprint arXiv:2404.17287 (2024).

[3] Han, Haixia, et al. "Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience." arXiv preprint arXiv:2404.10315 (2024).

[4] Tian, Katherine, et al. "Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback." The 2023 Conference on Empirical Methods in Natural Language Processing.

[5] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Teaching Models to Express Their Uncertainty in Words." Transactions on Machine Learning Research.

Q3: The confidence metric presented in Figure 2 lacks clarity.

A3: Thank you for your concern. It is true that Llama3-8b achieves over 70% accuracy on CommonsenseQA. To provide a broader perspective, we have included results on additional datasets in Appendix E.1, and Figures 12 through 16, covering 5 other datasets. For example, on TruthfulQA (Figure 16), where the model achieves approximately 40% accuracy, we observe that RLHF-trained LLMs’ confidence tends to concentrate within high-confidence bins. In contrast, the pre-RLHF version exhibits a broader distribution of confidence scores. This suggests that observed confidence is not solely attributable to the simplicity of the dataset.

Q4: It would be valuable to include a distribution of confidence scores after tuning with these proposed methods.

A4: Thank you for your suggestion. We have added a comparison of the confidence distributions for PPO, PPO-M, DPO, and CDPO (an extension of PPO-M to DPO-based pipelines) in Appendix E.4 and Figure 22 of the updated version of our paper. To summarize our observations, we note that the confidence distribution for PPO-M is slightly shifted to the left compared to PPO, showing a decrease in high-confidence scores (e.g., confidence level 10, which represents a highly overconfident state) and an increase in lower confidence categories. For CDPO, this phenomenon is more obvious; compared to DPO, the confidence distribution of CDPO is more widely spread across categories, with a noticeable increase in lower-confidence levels.

Thank you for the effort you put into the rebuttal. I am generally satisfied with the additional evaluations and the improvements in the paper's clarity, therefore I will improve my rating on this paper toward acceptance.

Quick question:

I appreciate the making of Figure 23 in Appendix E.5. The figures look good, though they were slightly different from what I had in mind at the time. If I understood correctly, each logit distribution corresponds to a subset of QA that has been given a particular confidence score by the model, which means the size of the subset could vary significantly across different confidence scores and I can foresee that lower confidence score will be a much lower amount of samples. If we can get a sense of how many QAs are used to construct every distribution that would be helpful statistically speaking.

I was thinking about the logit distribution over the verbal confidence tokens for a single QA. Concretely, $\rm{P}(\rm{confidence}|x_{t<i}), \rm{confidence} \in [<0>,<1>, ...,<10>]$. I want to see if the model truly understands rating confidence as a smooth single-modal curve centered around the confidence score with the highest probability, and I also would like to learn if you believe that this figure would be helpful or not.

We sincerely thank you for your comments and recognition of our work and additional evaluations.

Regarding your question, since it is not permissible to upload new figures at this moment, we will provide the statistics here and include the plot in the camera-ready version.

For the current Figure 23, it incorporates all 817 samples from the TruthfulQA dataset. We extracted the log probabilities for 10 numbers from each sample and aggregated them to produce the figure.

To demonstrate the distribution for a single QA sample, we randomly select one sample from the TruthfulQA dataset (round to 4 digits for clarity here):

For PPO:

• {1: 1.7925e-06, 2: 1.9081e-06, 3: 5.1868e-06, 4: 1.9271e-05, 5: 0.0004, 6: 0.0020, 7: 0.0348, 8: 0.2575, 9: 0.3746, 10: 0.3306}

For PPO-M:

• {1: 1.0372e-07, 2: 7.5881e-08, 3: 1.9377e-07, 4: 9.8404e-07, 5: 2.1040e-05, 6: 0.0002, 7: 0.0096, 8: 0.1705, 9: 0.7641, 10: 0.0553}

To summarize, we observe that both distributions are unimodal, with PPO-M being more concentrated than standard PPO. This aligns with our expectations, as standard PPO is not explicitly optimized for outputting verbalized confidence scores. Overall, we agree that this visualization is a helpful tool for understanding model behavior in assigning confidence scores, and we appreciate your suggestion to include it.

Thank you once again for your valuable feedback that makes our paper better.

Thank you very much for your valuable comments. We have revised the manuscript to incorporate your suggestions, and we will address your concerns in detail below:

Q1: From a scientific perspective, the method does not offer much insight.

A1: We thank the reviewer for the concern regarding the insights provided by our work. To clarify, we summarize the key insights offered below:

• Identification of overconfidence root cause: we identify the potential root cause of overconfidence in RLHF-trained LLMs and demonstrate how reward model biases can propagate through RLHF, leading to overconfident behavior in LLMs, which hasn’t been recognized by previous studies.
• Effective bias mitigation through simple intervention: We show that simple intervention of reward scores on binary preferences can already effectively address the reward model bias.

We agree that more fine-grained confidence levels could potentially lead to better calibration. However, this would require k-wise preference datasets, while many current preference learning datasets are based on binary human preferences [1][2][3]. We have already shown that binary distinctions can already improve model calibration, and we leave more fine-grained levels of confidence to future studies.

[1] Wang, Zhilin, et al. "HelpSteer2-Preference: Complementing Ratings with Preferences." arXiv preprint arXiv:2410.01257 (2024).

[2] Bai, Yuntao, et al. "Training a helpful and harmless assistant with reinforcement learning from human feedback." arXiv preprint arXiv:2204.05862 (2022).

[3] Ji, Jiaming, et al. "PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference." arXiv preprint arXiv:2406.15513 (2024).

Q2: From a particle perspective, the methods don’t seem that useful either

A2: Thank you for raising the concerns. While we appreciate your perspective, we respectfully disagree with the argument. Reward scores can be challenging to interpret due to different scales across prompts, as illustrated in Figure 1, and are typically compared in a pairwise manner. These scores are generally designed to measure the quality of model responses rather than assess model confidence. Furthermore, reward models are not consistently available in real-world applications. For example, when users interact with LLM chatbots, it is often more practical to integrate confidence calibration directly into the model generation process rather than relying on external scoring models. Additionally, verbalized confidence in LLMs has become an active area of research, with prior studies demonstrating the effectiveness of verbalized confidence over logit-based scores. While overconfidence phenomena have been observed, they remain unexplained [1] [2].

From a practical standpoint, our proposed methods offer several key advantages:

• Seamless integration: They can be integrated into existing PPO or DPO pipelines, providing better calibration results without compromising model performance on downstream tasks or reducing instruction-following capabilities compared to standard PPO or DPO.
• No “golden” labels required: Unlike previous training-based methods for improving model calibration [3][4] that require “golden” confidence labels for supervised fine-tuning, our proposed calibration training simply augments existing pairwise datasets with randomly assigned confidence scores.
• Efficiency: While PPO-M involves the calibration of a reward model, PPO-C introduces no additional computational overhead compared to standard PPO.

Therefore, we believe our proposed methods hold practical values due to its ability to enhance calibration seamlessly, maintain performance and efficiency, and eliminate the need for label requirements.

[1] Tian, Katherine, et al. "Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback." The 2023 Conference on Empirical Methods in Natural Language Processing.

[2] Xiong, Miao, et al. "Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs." The Twelfth International Conference on Learning Representations.

[3] Lin, Stephanie, Jacob Hilton, and Owain Evans. "Teaching Models to Express Their Uncertainty in Words." Transactions on Machine Learning Research.

[4] Han, Haixia, et al. "Enhancing Confidence Expression in Large Language Models Through Learning from Past Experience." arXiv preprint arXiv:2404.10315 (2024).

Thank you for your rebuttal. The new version of PPO-C makes more sense to me. The new experiment is interesting, but did you forget to include the scaling factor of $w = 0.5$, i.e. would it not be more comparable to take the reward to be $r_i=\hat r_i+w*(\hat r_i-\Delta r_t)*(s_i-0.5)$ with $w=0.5$? Failing to include this scaling factor would make the reward adjustment twice as large, which may explain why the new version favors calibration over instruction-following. Perhaps the running average of reward is close to 0, making the methods almost identical anyway (modulo the scaling factor issue). If so, I'd suggest using the new version, as it makes more theoretical sense, even though the two may be almost identical in practice.