REBEL: Reinforcement Learning via Regressing Relative Rewards

Thank you for your valuable review of our paper.

The statement "REBEL ... be extended to handle intransitive preferences ...." in the abstract is not adequately presented in the main content of the paper. As the major influence brought by intransistive preferences is the degradation of reward score accuracy, which is not addressed by this paper.

We appreciate your observation and agree with your assessment. Since our focus is not on the preference model itself, we did not conduct experiments specifically targeting preference models. However, we provided a method to extend REBEL to preference models. The extension of REBEL to address intransitive preferences is discussed in detail in Appendix D, with further analysis provided in Appendix G. How to pre-train a good preference model without any degradation of reward score accuracy is beyond the scope of our paper.

I would suggest the authors to summarize the limitations of the proposed method in a separate "Limitations" section.

Thank you for the suggestion. We will include a dedicated "Limitations" section in the next version of the paper.

Thank you for your constructive feedback. We address each of your points below.

Insufficient experimental validation and limited baseline comparisons

Performance comparison with baselines

Our experimental section is comprehensive compared to previous works on RLHF [1, 2], incorporating a general chat dataset and an image generation task to evaluate the robustness and generality of REBEL. We compared REBEL with several algorithms, including DPO, PPO, iterative DPO, REINFORCE, RLOO, and APA. Detailed hyperparameter settings are provided in Appendix H.1.4, H.2.3, H.3.3.

The lower performance of DPO compared to PPO is not due to implementation or hyperparameter choices. On TL;DR, our DPO results are better than the ones reported in [3]. Results for PPO, REINFORCE, and RLOO are directly obtained from prior papers [2, 4] which are exclusively focusing on these algorithms on TL;DR. For general chat, we directly report winrates from the released starling-alpha model by APA's authors. Thus, we believe that our comparison to baselines is fair.

In Figure 3, the comparison at an intermediate checkpoint is intended to highlight the sample efficiency of REBEL. There is no indication that PPO has overfitted at this epoch, as the image quality of PPO continues to improve afterward.

Sample efficiency and convergence guarantees

For the image generation task, as illustrated in Figure 4, REBEL converges faster during the initial training phase and eventually achieves performance comparable to that of PPO. In addition, we plot the reward vs. step for TL;DR dataset in the pdf of the global rebuttal. The plot demonstrates REBEL's faster convergence compared to iterative DPO and PPO.

Lack of support for certain claims and limited exploration of key aspects

Applicability to deterministic MDPs

Choice of base distribution \mu

Since our focus is not on the preference model itself, we did not conduct experiments specifically targeting preference models. However, we provided a method to extend REBEL to preference models in Appendix D, with further analysis provided in Appendix G.

In our experiments, we found that setting $\mu=\pi_t$ yielded better results. We attribute this to the lower quality of the offline dataset. In TL;DR summarization and Ultrafeedback, our trained policies can generate better responses than the ones in the datasets. This is shown in Table 1, where the 2.8B and 6.9B models can easily reach high winrates compared to offline human demonstrations. In this case, setting $\mu$ to $\pi_{ref}$ or the offline data does not help significantly. Therefore, we use $\pi_t$ as $\mu$ in all experiments.

Since transitions are deterministic, we could formulate this as a bandit problem where $x$ is the initial state and $y$ is the trajectory consisting of a sequence of actions [1, 2, 10]. Applying token generations in LLMs and image generation to deterministic MDPs has been proposed in prior RLHF works [5, 6, 7, 8]. We adapt this bandit setup to make a fair comparison to the previous works.

Inconsistencies and potential conflicts with previous statements

Stochastic MDPs and the need for critic-based variance reduction

Lack of experimental support in stochastic MDPs is not a significant limitation as deterministic MDPs have wide applications in LLMs and image generations, which are high-impact areas. We emphasize that all prior RLHF works [1, 2, 4, 6] all focus on deterministic MDP.

There is also no conflict in our criticism of PPO's complexity. Our paper focuses on deterministic transitions; in this context, many prior work have argued that PPO and critic-based variance reduction methods are not necessary [2, 8, 9]. We agree that in highly stochastic settings, critics might still be necessary, which is indeed stated in the paper.

Relationship between regressor performance and dataset quality:

A high-quality dataset, i.e. a dataset that has diverse coverage, would increase the generalization ability of the learned regressor. E.g., if the training distribution is diverse, then the learned regressor can achieve better performance under the comparator policy $\pi^*$ distributions, which, as our theory indicates, will ensure convergence to $\pi^*$. Our work focuses on online RL and studying the dataset quality is not the main focus of the paper.

[1] Rafailov R, Sharma A, Mitchell E, Manning CD, Ermon S, Finn C. Direct preference optimization: Your language model is secretly a reward model.

[2] Ahmadian A, Cremer C, Gallé M, Fadaee M, Kreutzer J, Üstün A, Hooker S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.

[3] Rafailov R, Chittepu Y, Park R, Sikchi H, Hejna J, Knox B, Finn C, Niekum S. Scaling laws for reward model overoptimization in direct alignment algorithms.

[4] Huang S, Noukhovitch M, Hosseini A, Rasul K, Wang W, Tunstall L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL; DR Summarization.

[5] Ramamurthy R, Ammanabrolu P, Brantley K, Hessel J, Sifa R, Bauckhage C, Hajishirzi H, Choi Y. Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.

[6] Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, Radford A, Amodei D, Christiano PF. Learning to summarize with human feedback.

[7] Chang JD, Shan W, Oertell O, Brantley K, Misra D, Lee JD, Sun W. Dataset reset policy optimization for rlhf.

[8] Black K, Janner M, Du Y, Kostrikov I, Levine S. Training diffusion models with reinforcement learning.

[9] Oertell O, Chang JD, Zhang Y, Brantley K, Sun W. Rl for consistency models: Faster reward guided text-to-image generation.

[10] Wu T, Zhu B, Zhang R, Wen Z, Ramchandran K, Jiao J. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment.

Thank you for your valuable review of our paper. We respond to your individual questions below.

I believe that at least a brief section on related work should be included in the main paper, the in-depth one can be deferred to the appendix. In terms of space, I personally do not think Section 2.2 adds much value to the main paper.

Thank you for your suggestion. We will include a brief section on related work in the main paper and have a detailed discussion in the appendix.

How significant a problem is reward model over-optimization [1] for REBEL?

In the context of RLHF for LLMs, we choose reward models that are highly ranked on the Reward Bench [2]. These reward models are trained using extensive datasets that include responses generated from a variety of different policies [3]. This diversity makes it challenging to over-optimize a reward model, even for methods such as REBEL. In addition, following previous work [4], we apply an addition KL penalty to the reward, $r(x, y) = RM(x, y) - \gamma (\ln \pi_{\theta_{t}}(y|x) - \ln \pi_{\theta_{0}} (y|x))$, to prevent over-optimization for both REBEL and the baseline methods.

In the image generation setting, following previous work [5, 6], no KL regularization is used. We observe that both PPO and REBEL over-optimize the reward model towards the end of the training. They tend to ignore the input prompt and generate the same image that maximizes the reward model score. For fair comparison to [6], we also did not include KL.

It would be interesting to see and understand the differences between reward-weighted regression baseline (RWR) and REBEL as they have some close connections.

RWR can be understood as reward-weighted imitation learning. Prior work on RL for diffusion models shows that RWR is not as effective as RL (PPO in that case) [5]. REBEL, on the other hand, is a full RL algorithm from its connection to NPG where it generalizes NPG.

Is there an optimal choice of $\mu$? What are the intuitive differences between using the $\mu=\pi_{ref}$ and $\mu=\pi_{t}$?.

Setting $\mu$ to $\pi_{ref}$ or another distribution can encourage exploration, especially when the policy is still learning and the quality of the generations is not yet optimal. This approach can help in discovering diverse and potentially higher-reward samples that the current policy might not generate.

In our experiments, we found that setting $\mu=\pi_t$ yielded better results. We attribute this to the lower quality of the offline dataset. In TL;DR summarization and Ultrafeedback, our trained policies can actually generate better responses than the ones in the datasets. This is shown in Table 1, where the 2.8B and 6.9B models can easily reach high winrates compared to offline human demonstrations. In this case, setting $\mu$ to $\pi_{ref}$ or the offline data does not provide significant benefits. Therefore, we use $\pi_t$ as $\mu$ in all our experiments.

Why are datasets not aggregated? Instead, only the most recently collected dataset is used for training.

Our approach can be understood as online RL, e.g. PPO, where only on-policy data is used for update. We could aggregate the datasets which makes the approach more off-policy. While we have not tested this in our experiments, it could be an interesting direction to explore if being off-policy would lead to better sample efficiency.

In addition, previous RLHF works [4, 8, 9] also use batches collected by the current policy (i.e. online batch), with each batch containing a new set of prompts. To ensure a fair comparison with previous methods, we primarily focus on the on-policy approach.

[1] Gao, L., Schulman, J., & Hilton, J. (2023, July). Scaling laws for reward model overoptimization. In International Conference on Machine Learning (pp. 10835-10866). PMLR.

[2] Lambert N, Pyatkin V, Morrison J, Miranda LJ, Lin BY, Chandu K, Dziri N, Kumar S, Zick T, Choi Y, Smith NA. Rewardbench: Evaluating reward models for language modeling. arXiv

[3] Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., Zhang, T. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint. 2024. arXiv

[4] Huang S, Noukhovitch M, Hosseini A, Rasul K, Wang W, Tunstall L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL; DR Summarization.

[5] Black K, Janner M, Du Y, Kostrikov I, Levine S. Training diffusion models with reinforcement learning. arXiv

[6] Oertell O, Chang JD, Zhang Y, Brantley K, Sun W. Rl for consistency models: Faster reward guided text-to-image generation. arXiv

[7[ Korbak T, Shi K, Chen A, Bhalerao RV, Buckley C, Phang J, Bowman SR, Perez E. Pretraining language models with human preferences. In International Conference on Machine Learning 2023 Jul 3 (pp. 17506-17533). PMLR

[8] Stiennon N, Ouyang L, Wu J, Ziegler D, Lowe R, Voss C, Radford A, Amodei D, Christiano PF. Learning to summarize with human feedback. Advances in Neural Information Processing Systems. 2020;33:3008-21.

[9] Ahmadian A, Cremer C, Gallé M, Fadaee M, Kreutzer J, Üstün A, Hooker S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.

Thank you for your encouraging review and comments. We respond to your individual questions below.

Do the authors have any idea why REBEL seems to have a slightly higher KL than the other methods?

The KL divergence is generally close across methods. For the TL;DR experiments, following previous work [1], we apply an addition KL penalty to the reward: $r(x, y) = RM(x, y) - \gamma (\ln \pi_{\theta_{t}} (y|x) - \ln \pi_{\theta_{0}} (y|x))$. PPO incorporates a clipping mechanism on top of this regularization, which allows it to control the KL-divergence more strictly. This difference in approach can lead to REBEL exhibiting a slightly higher KL-divergence compared to PPO.

Although in image alignment REBEL seems to do similarly to PPO, it also has higher variance. Do you know why that might be?

To fairly compare REBEL and PPO, we tuned all hyperparameters, including parameters related to reward queries for PPO. Afterward, to ensure that REBEL was compared fairly to PPO, we kept those hyperparameters related to reward queries constant so that REBEL and PPO had the same number of reward queries per update. We believe that if we were to modify the reward queries per update, REBEL would show a lower variance than our current runs.

Are the results for the 6.8B model significant? It seems as though REBEL produces very similar performance to e.g. PPO. For the smaller models the separation seems larger, is there a reason why the separation in performance between REBEL and other methods is bigger for smaller models?

The discrepancy in our training setup could be the reason for the smaller separation in the larger model versus the smaller models. Specifically, we performed hyperparameter tuning using LoRA [3] on the 1.4B and 2.8B models. We then directly applied these hyperparameters to the larger 6.9B model for full-parameter training without additional tuning (detailed in Appendix H.1.2). The results for the baselines that we compared to (REINFORCE, PPO, RLOO) for the 6.9B models are obtained from [1,2], which might have directly tuned the 6.9B model. These differences contribute to the decreasing performance gains as model size increases.

What are the error bars in Table 1? Is that standard deviation?

The error bars are standard deviations. Results are averaged over three seeds and the standard deviations across seeds are in parentheses.

[1] Huang S, Noukhovitch M, Hosseini A, Rasul K, Wang W, Tunstall L. The N+ Implementation Details of RLHF with PPO: A Case Study on TL; DR Summarization.

[2] Ahmadian A, Cremer C, Gallé M, Fadaee M, Kreutzer J, Üstün A, Hooker S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms.

[3] Hu EJ, Shen Y, Wallis P, Allen-Zhu Z, Li Y, Wang S, Wang L, Chen W. Lora: Low-rank adaptation of large language models. arXiv