Off-Policy Corrected Reward Modeling for Reinforcement Learning from Human Feedback

We would like to thank the reviewer for their helpful and kind review!

We will reconsider the presentation of the importance weights and extend the algorithm in the appendix, to prevent any misunderstandings.

We will also share our implementation after acceptance, to prevent any potential issues with reproducibility.

We would like to thank the reviewer for their helpful and kind review.

• More iterations beyond m=3

We initially used m=3 due to computational constraints. We now ran an additional experiment with m=4 and m=5 on Pythia 1B on the TL;DR task and indeed found a further improvement for m=4, but it seems to saturate for m=5:

We will add this experiment to the paper, thank you for your suggestion!

• Accuracy of the Gold Reward Model

When generating the reward model dataset we follow the methodology of Gao et al. [1] and always choose the completion with a higher gold RM score as preferred completion, the Gold RM thus has a perfect accuracy.

For comparison we show the reward model accuracies on the RM training dataset here:

Please let us know if there is any other information about the reward models that would be helpful!

References:

[1] Gao et al. “Scaling Laws for Reward Model Overoptimization”, https://arxiv.org/abs/2210.10760

We would like to thank the reviewer for their thoughtful and helpful review.

• Only two tasks and synthetic Gold RM setting

We acknowledge that this is a limitation. We decided to focus on fewer datasets and models in order to provide more complete ablations instead, which we hope to provide a better understanding of our method. We unfortunately do not have the resources to perform training runs with real human feedback instead of a synthetic setting.

• Qualitative analysis of effect on outputs

We performed an additional analysis to investigate the diversity of the model outputs, based on the methods suggested in Kuhn et al. [1]: On a subset of 256 prompts from the validation set, for each prompt, we sample 20 completions and use deberta-large to cluster them into semantic clusters. We report the average number of clusters. The results show that our method maintains a similar output semantic diversity compared to the PPO baseline:

We will further extend this experiment along with more example outputs and add it to our submission, thank you for the suggestion!

We also provide statistics of the gold reward distribution on the validation set, indicating that the improvement is not limited to specific prompts:

• Model size limited to 1B

We would like to note that we also ran experiments on Pythia 2.8B, shown in Table 3 in the submission and reproduced here, which shows that our method still is beneficial with increased model sizes:

We understand that this is still not comparable to large-scale model training runs, unfortunately we do not have the computational resources to scale the experiments to large LLMs.

Thank you for the suggestions, we hope that we could address your concerns. We would be grateful for additional suggestions on specific qualitative analyses for the output.

References:

[1] Kuhn et al. “Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation”, ICLR 2023

To address the reviewer's concerns about the synthetic gold model setup we ran an additional experiment with an API model as a replacement for the gold model. We train Pythia-1B on the TL;DR summarization task with pairwise comparisons provided by GPT 4.1 Nano, both to create the preference labels for the RM training dataset and for the final evaluation.

The results are consistent with the gold reward model experiments shown in the paper:

We will add this experiment to the paper and hope it can address the reviewer's concerns about our experiments. Thank you for the suggestion!

To address the reviewer's concerns about the model size, we now added an additional experiment with GRPO to train a Pythia 6.9B model on the summarization task:

The results show that our method also works for larger models.

We would like to thank the reviewer for their suggestion and hope that that we could address their concerns about the experimental evaluation.

We would like to thank the reviewer for their helpful advice and questions.

We will revise Table 1 and move the results for individual random seeds to the Appendix.

• “Online RLHF” terminology

While we note that the terminology has been used in recent papers [1,2], we agree that it can be misleading as no new preference data is obtained online in our problem setting, instead only new completions are sampled online from the policy. We will revise the relevant sections to replace this terminology.

• Meaning of $P_{\pi^i}(s,a_\mathrm{w},a_\mathrm{l})$

We consider a data generation process in which we first sample the prompt $s \sim P(s)$, then independently sample two completions from the model $\pi^{i}$, $a_1\sim \pi^i(a|s)$, $a_2 \sim \pi^i(a|s)$ and finally sample a human preference label $y \sim P(y|s,a_1,a_2)$, which is 1 if $a_1$ is preferred over $a_2$ and 0 otherwise. Together, this yields the joint probability

$P_{\pi^i}(s,a_1,a_2,y) = P(s)\pi^i(a_1|s)\pi^i(a_2|s)P(y|s,a_1,a_2) .$

In the current version of our submission the preference label $y$ is absorbed into the ordering of terms $a_\mathrm{w}$ and $a_\mathrm{l}$, respectively for the winning and losing reply. We hope this explanation clarifies the term and we will improve the presentation in the submission.

We believe these changes will improve the clarity of our submission, thank you!

References:

[1] Dang et al., “RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs”, EMNLP 2024

[2] Ethayarajh et al., “KTO: Model Alignment as Prospect Theoretic Optimization”, ICML 2024