Rethinking Reward Modeling in Preference-based Large Language Model Alignment

3. When cross-prompt comparisons can be particularly effective.

We thank the reviewer’s interest in the cross-prompt comparisons. We will try to answer this question from both theoretical and empirical perspectives.

(1). Theoretically, we have added a section in Appendix G (pages 34-37) that theoretically justifies the superiority of using cross-prompt comparisons in preference annotations. To highlight the high-level takeaways of this section, we show that in expectation, cross-prompt comparisons can improve annotation quality since they increase the expected differences between prompt-response pairs. More specifically, we have

• Example 1 and corresponding numerical results to illustrate how annotation quality changes with different response diversity;
• Proposition G.1 to show that using cross-prompt comparisons can improve comparison diversity;
• Theorem G.2 to show that cross-prompt comparisons can improve annotation quality.

In our revised manuscript, those added contents are highlighted with blue text. We additionally used gray text for the proofs to enhance the clarity.

(2). Empirically, we studied the performances of cross-prompt comparisons under diverse annotation qualities. According to the results in Section 4.3, we can conclude from Figure 4 that cross-prompt comparisons generally improve the alignment performance.

In order to understand when those cross-prompt comparisons can be particularly effective, we controlled the generation diversity in response generations. According to the results in Figure 5, we find that cross-prompt comparison can significantly improve the alignment performance when the responses lack diversity. To further verify such a finding, we evaluated the correlation between response diversity and the improvement achieved by cross-prompt comparisons (over normal comparisons). We observed a high correlation between those two variables (Figure 6). Those observations validate the insights in the above theory.

Once again, we would like to thank the reviewer for their insightful comments and thorough reading to improve our paper. We hope the responses above have addressed the outstanding questions and the reviewer would consider raising their score if those questions have been appropriately answered. Should there be any leftover concerns, please don’t hesitate to let us know and we will do our utmost to address them!

2. Reward model calibration.

Clarification: with calibration, we suppose the reviewer is referring to the property that a higher reward model score should correspond to a higher oracle utility score. Please correct us if we understand the reviewer’s comment incorrectly!

We thank the reviewer for pointing out the importance of having calibrated reward models, and we fully agree such a property is essential for the success of the reward modeling process. In our Theorem 2.2, the objective is to demonstrate why the BT model works in estimating reward function, hence the analyses in Section 2 mainly focus on minimizing errors in score predictions.

In section 3 of our paper, we discussed the essential property of order consistency (i.e., reward model calibration) in reward modeling — in our work, the order consistency formally defined whether a reward model is well-calibrated, and we highlighted the objective of reward modeling should be optimization toward better order consistency rather than toward more precise win rate predictions as in the BT models. This is because the final objective of reward modeling is to optimize LLMs to generate better responses using those reward models, instead of knowing the exact win rate between any two responses. In our work, we have Proposition 3.3 to theoretically justify the importance of having order-consistent (i.e., calibrated) reward models, and we explained in section 3.2 and section 3.3 why both the BT models and the classification-based reward models are optimizing toward such an objective.

Empirically, the percentile performances we reported above (in Table 3 and Table 4 of the previous response) work as an additional evaluation metric that directly quantifies whether different reward models are properly calibrated (or with the concept defined and used in our paper, order consistent).

3. Discussion on preference transitivity

We thank the reviewer for raising the question and inspiring a discussion of the potential non-transitive preference annotations.

In the canonical RLHF framework, the reward modeling step builds a deterministic reward model that assigns a scalar value to each of the responses, and the fundamental assumption is that such an oracle utility value exists. RLHF chose to build reward models from pairwise preference annotations rather than direct score-based annotations due to the consistency challenge in the latter approach.

Therefore, the existence of such an oracle utility value lays the foundation of the policy optimization step that seeks to maximize the utility value. An interpretation of the oracle utility value can be the population-level preferences. From such a perspective, the non-transitive annotations manifest the imperfectness of annotators in identifying the population-level preferences, or this can be interpreted as an individual-dependent bias from another point of view.

Due to space limit, we deferred an extended discussion linking those individual-level bias and reward modeling in Appendix B (pages 16-17 in the updated manuscript, we have highlighted those contents with orange text). We also elaborated on the underlying assumptions and motivation for applying the BT models to the preference data annotated by different individuals.

To sum up, the individual annotations in a preference dataset can contain non-transitive preferences, yet the reward modeling objective is to learn the population-level average of those imperfect preference annotations, which is transitive by definition.

Once again, we would like to thank the reviewer for their insightful comments and thorough reading to improve our paper. We hope the responses above have addressed the outstanding questions and the reviewer would consider raising their score if those questions have been appropriately answered. Should there be any leftover concerns, please don’t hesitate to let us know and we will do our utmost to address them!

Q3. explanation of this unexpected robustness to annotation errors, and potential implications of this finding for practical applications

We thank our reviewer for raising this insightful point for further discussion.

One way to understand the classification-based method is that instead of predicting P(i wins j), we instead predict P(i wins), which is an expectation of P(i wins j) over the randomness of j. One can show that P(i wins) is a lower variance object to be estimated: because it is an average over j of P(i wins j). It can be seen by the law of total variance. Consider the variance

$Var_{i,j}(P(i \textrm{ wins } j|i,j)) = Var_i(\mathbb{E}_j(P(i \textrm{ wins } j|i,j)|j)) + \mathbb{E}_i(Var_j(P(i \textrm{ wins } j|i,j)|j)) = Var_i(P(i \textrm{ wins }|i)) + \mathbb{E}_i(Var_j(P(i \textrm{ wins } j|i,j)|j))$

The first term on RHS is the classification-based target’s variance and the second term is non-negative. So the classification-based method targets an objective with less noise in it.

The discovery in our experiments suggests using classification-based methods when annotation quality is limited and considering using cross-prompt comparisons to improve annotation quality when feasible. On the other hand, when annotation quality is high and the preferences are more deterministic (e.g., a practical case is individualized alignment where all preference labels are from a single user), we should consider using BT models instead of the classification models, since those models, in general, performs better in those cases. (c.f. Figure 2 and Figure 3, when there is less annotation noise, the BT models are better)

Q4. Complement to Existing Benchmarks

We thank the reviewer for raising the question on benchmarking the methods.

In our paper, we choose to use fixed embeddings as inputs for reward models to isolate the source of gains in reward modeling choices from the representation learning. Our current experiments are designed to verify the insights and theories introduced in the paper and make comparisons between different reward modeling approaches by controlling other variables. Hence our discoveries are more of a complement to existing studies on reward modelling and have the potential to be combined with them.

Based on our discoveries, combining the improvement of reward modeling choices and annotation strategies introduced in our work with other orthogonal advances in the literature such as representation learning [5], data merging [6] and regularizations [7] can be important and promising future directions yet out of the current research’s scope.

References

[1] Cui, Ganqu, et al. "Ultrafeedback: Boosting language models with high-quality feedback." arXiv preprint arXiv:2310.01377 (2023).

[2] Lee, Harrison, et al. "Rlaif: Scaling reinforcement learning from human feedback with ai feedback." arXiv preprint arXiv:2309.00267 (2023).

[3] Bai, Yuntao, et al. "Constitutional ai: Harmlessness from ai feedback." arXiv preprint arXiv:2212.08073 (2022).

[4] Wu, Tianhao, et al. "Meta-rewarding language models: Self-improving alignment with llm-as-a-meta-judge." arXiv preprint arXiv:2407.19594 (2024).

[5] Yang, Rui, et al. "Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs." arXiv preprint arXiv:2406.10216 (2024).

[6] Dong, Hanze, et al. "Rlhf workflow: From reward modeling to online rlhf." arXiv preprint arXiv:2405.07863 (2024).

[7] Wang, Binghai, et al. "Secrets of rlhf in large language models part ii: Reward modeling." arXiv preprint arXiv:2401.06080 (2024).

Once again, we would like to thank the reviewer for their insightful comments and thorough reading to improve our paper. We hope the responses above have addressed the outstanding questions and the reviewer would consider raising their score if those questions have been appropriately answered. Should there be any leftover concerns, please don’t hesitate to let us know and we will do our utmost to address them!