Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback

Thank you for your review, and noting that our work is clearly written, with an easy to follow narrative, detailed experimental setup, and multiple interesting observations. We clarify key points of our experimental setup and choices (e.g. datasets) below, which hopefully address your key concerns. We will add these additional details and clarifications to our updated paper.

Concerns:

1. Algorithm choice & Hyperparameter choice

In particular, for DPO, we tested values of {0.1, 0.01, 0.001} for beta {5e-6, 5e-7, 5e-8} for LR for both HH-RLHF and UltraFeedback. For PPO, we experimented with KL penalty coefficients of {0.01, 0.025, 0.0325, 0.05}, as well as varying the batch size (including taking more samples per gradient step, and taking multiple steps over the same generations). We discuss some of the details around our hyperparameter choices in Appendices F and G, and will explicitly add the details given above. For PPO hyperparameters we did not search extensively over (clip threshold, discounting factors, learning rates, adam betas), we borrow primarily from InstructGPT [2] and Llama-2 [3], and open-source repositories (Rainier [4], Finegrained-RLHF [5], AlpacaFarm [6]), which are common reference points for RLHF work. We provide additional notes on hyperparameter choice and additional observations using PPO in Appendices F and G. As such, we believe we made a reasonable attempt to tune both DPO and PPO hyperparameters independently, although we note that further performance gains could be made by tuning on a per-dataset basis.

2. Dataset choice in Table 2

We apologize for not explaining the dataset choice in Table 2: we explicitly chose the top-performing dataset of each source from Table 1 (StackExchange from Web, HH-RLHF from human, Ultrafeedback from synthetic). We also include the second-best performing dataset overall (Nectar) and an additional human-annotated dataset from the popular evaluation platform (Chatbot Arena; we chose 2023 over 2024 to reduce computational cost). We will add these details to our updated manuscript.

3. Why does synthetic data perform well?

We hypothesize that the synthetic data we test performs well for a few reasons: (1) annotation quality is generally high due to the use of GPT-4 as a judge [1] and GPT4 labels may be hypothetically more consistent, (2) the prompts chosen cover a wide range of topics useful for downstream evaluations (math, reasoning, chat, coding, etc.), (3) there is some alignment between synthetic preferences and our evaluation setting - for example, Chatbot Arena data is human-sourced, but shows a clear degradation in safety which hurts its overall performance, and using AlpacaEval in our evaluation means we have a slight bias to GPT-generated data (although this is not the case for the many other evaluations we examine). We believe that further exploring how well synthetically generated data compares to expert-level human annotations, and exploring differences between the two, is an interesting avenue for future work.

4. Preference-based learning’s effect on factuality.

We agree and will adjust the wording to make it clearer that our observations are based on the datasets we consider in our introduction, conclusion, and section 3. We agree that a carefully-made factuality dataset may indeed improve performance (even if it is non-optimal compared to SFT-based approaches).

5. Identifying the performance contribution independent of other components.

We agree that there may be ways in which the contribution of a different component varies based on the setting (for example, some datasets not tested may scale better than those tested for the reward model experiments). Due to computational constraints, we did not test every single possible combination of {rm size, prompt, dataset, hyperparameter choice}, but focussed on running ablations using the best choices from the previous steps (e.g., focussing on UltraFeedback for PPO ablations, and using the best datasets found for DPO for comparing to PPO).

6. Marginal vs non-marginal improvement

We agree that both changes give improvements of similar size, although we believe that the PPO claim is slightly stronger due to being a result of tests across multiple datasets (in fact, the p-value for the observed difference is .04 under a two-tailed paired t-test), while the larger RM size increase in performance was only from one experiment. As such, we have less confidence in the claim that the larger RM helps, although the boost on GSM8k appears large. We will adjust the wording in each section accordingly to better match this (explicitly noting the statistical significance of the DPO & PPO result, and noting that the jump in performance from using the larger RM is of similar scale, but is from a single run and driven largely by GSM8k improvements).

Additionally, we were somewhat surprised that a 10x increase in RM size (and almost 2x increase in best-of-N improvements, from 5.8 to 10.3) did not yield similarly large changes in the overall performance, and this coloured our reporting. We will adjust our wording accordingly (small changes in light of large BoN & compute increases).

Questions:

1. How did the authors select the preference datasets to compare DPO vs PPO in Table 2?

Please see our response to concern 2 above.

---

[1] Zheng et al (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. NeurIPS.

[2] Ouyang et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.

[3] Touvron et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. ArXiv.

[4] Liu et al. (2022). Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering. EMNLP.

[5] Wu et al. (2023). Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. NeurIPS.

[6] Dubois et al (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. NeurIPS.

Thank you for your review and noting that our work is well written and clear to follow, and that the insights in our work are useful for practitioners in the field. We address concerns and questions below:

Concerns:

1. Novelty

While we do not propose entirely new methods ourselves, the novelty of this work is more on the comprehensive setup for comparing current resources for RLHF with LMs, and the findings supported by a large number of experimental results. To our knowledge, there is little prior work comparing and examining RLHF methods in detail with extensive empirical backing. Additionally, various findings in our work (e.g., the hardship in transferring reward model performance to policy model, the sensitivity to the prompt distribution in PPO) are the first instance of these observations/results being discussed publicly to the best of our knowledge. We believe such analysis should have novelty in itself and are valuable given the current status of LLM research, and believe that multiple reviewers do point out that our insights have practical use for researchers in the field.

Questions:

1. Extending to multimodal models?

In our work, we decided to focus on understanding popular methods and datasets across a range of evaluations, working on popular text-only models. Multimodal RLHF approaches are still very new compared to the language model space, and there are not as many available instruction tuning or preference datasets to explore. Nonetheless, we believe that exploring how our findings extend to the multimodal regime would be an interesting direction for future work.

Thank you for your detailed review and questions. We address your concerns and questions below and hope this provides further insights into our work.

Concerns:

1. Results could be improved with more ablations.

Please see point 1 of our general response, where we additionally provide new results with a new base model (llama 3).

2. It would be helpful to draw deeper insights.

Our aim in this work is primarily to explore and test the performance of popular RLHF methods on existing datasets, and is primarily empirical. We hope that our work serves as a useful starting point for further, more focused studies into RLHF (for example, investigating further why using much larger RMs does not lead to much larger gains). We also note that most reviewers agree our work does contain observations and results that are of interest to the broader community.

Questions:

1. The quality of pairwise judgements.

Quality here is primarily referring to the use of overall vs fine-grained judgements when deciding the chosen/rejected pairs in UltraFeedback, which has been shown to lead to improved reward model performance [1, table 2], even when using the same chosen/rejected pairs (and we find similarly improves DPO performance). We focus on the effect of data on downstream performance here, noting that it appears improving the chosen/rejected judgements can improve downstream performance.

2. Why is PPO better?

We believe the offline vs online aspects of DPO and PPO are quite important for performance (as noted at the end of section 2.1), and there is concurrent work also suggesting this [2,3]. Additionally, we hypothesize that the RM being finetuned on preference data without any regularization terms (e.g. the beta in DPO) may allow it to more closely fit human preferences, which may result in a stronger policy downstream.

3. Why does 70B mix rm have a lower RewardBench score than the 13B model?

RewardBench consists of outputs from varied models across varied prompts, and so may be somewhat out-of-domain compared to best-of-N (BoN) evaluations for the purpose of identifying good RMs for PPO. In contrast, our BoN evaluation indicates that both 70B RMs outperform the 13B models, suggesting that larger models are better at identifying superior outputs from the same initial policy used in PPO, which more directly relates to how the RM is used during PPO training. Examining BoN and RewardBench scores in more detail in Tables 12 and 13, we see that 70B mix RM outperforms the 13B mix RM on math (GSM), while degrading in instruction following (AEval 1 & 2, IFEval), which matches with the worse performance on the chat subsets of RewardBench. We find it surprising the 70B mix RM underperforms in the reasoning subset, considering the improved GSM performance, potentially due to differences in evaluation distributions (RewardBench reasoning subset consists mostly of coding questions).

4. Can we get some principles for dataset selection for RM?

Figure 3’s best results are found when using the GSM8k train set as prompts, which is somewhat different to the typical zero-shot setting used for modern LMs, where we assume our models will be tested on tasks not explicitly selected for during training. We did try training our models with Mix RMs and mixture prompts for longer but still did not observe improvements in performance (see answer to Q5). Overall, the fact that RewardBench and BoN does not translate neatly to downstream performance from PPO makes determining good methods for dataset selection for RMs difficult. It would be interesting to further explore how to improve RM dataset selection with an eye towards downstream performance in future work.

5. What if we use all UF prompts and some further code and math prompts to train?

We find that further training the model to use all the UltraFeedback prompts on top of additional math and code prompts does not result in consistent improved performance at the 13B scale, suggesting that using the full UF prompt with the mix prompts set does not provide consistent additional gains:

6. Are the four datasets (UF all, Strong, Middle and Weak) downsampled to the same size?

Yes (see Appendix E line 668).

7. Does page 19 equation (5) \pi_{ref} refer to the SFT policy?

Pi_ref does refer to the SFT policy, we will clarify this in our updated paper.

8. Can more data provide more improvements?

We first note that as seen in Table 1, datasets that are simply larger are not necessarily more performant - SHP-2 and StackExchange are by far the largest datasets we test, but underperform much smaller datasets. Similarly, Nectar is 3x larger than UltraFeedback but performs slightly worse for DPO. However, we do also observe some evidence that increasing dataset size (or at least, increasing up to 60k samples) helps: training on all UltraFeedback data outperforms training on the {weak,middle,strong} subsets (all of which consist of 10k datapoints) - see Table 7 in the Appendix.

9. I am curious about the training process during these experiments.

We recorded logs of reward, KL divergences, average response lengths, and more for all our experiments. We observed that average model output length tended to increase over training, as observed in prior work. Interestingly, we also observed that performance on different specific evaluations behaved differently over training, and provide further details in the additional rebuttal material PDF - please see the PDF attached to the general response.

[1] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML.

[2] Xu et al (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study.

[3] Tajwar, et al (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data.

Thank you very much for your review and feedback, and for noting that our results are a valuable reference for the community and enhance understanding of RLHF. We address your feedback and questions below:

Concerns:

1. Testing the influence of the training dataset size on reward model & downstream performance.

Our focus in these experiments is to see if improvements to the reward model (either through incorporating different data or increasing the RM size) lead to improvements in the downstream policy trained through PPO. In this sense, we are interested to see that both types of changes did not yield consistent large changes in downstream performance in Table 3, despite the fact that we observe improvements in RewardBench and Best-of-N settings, which more directly test RM performance. We will clarify that "both increasing the reward model dataset (‘Mix’) and reward model size (from 13B to 70B) improve performance" means that taking either option may improve performance, as indeed we agree that we do not see evidence that a larger dataset together with a larger RM further improves performance.

2. Limited number of models tested.

Please see our general response, point 1. We provide additional results using Llama 3 and think further extending our observations to other models would be interesting future work.

Questions:

1. What is the data ratio of Mix and UltraF?

We provide exact sizes of the subsets and overall sizes of Mix and UltraF in Table 6 in the appendix. Mix contains roughly 260k samples from Stack Exchange, HH-RLHF, HelpSteer, PRM800k, and Nectar. UltraF contains roughly 60k samples. UltraFeedback itself is made up of prompts and completions (from varied models) from FalseQA, Evol Instruct, TruthfulQA, Flan V2, ShareGPT, and UltraChat (see Table 7 in Appendix E or the UltraFeedback paper itself [1] for more details). We add HelpSteer, Nectar, and HH-RLHF as they are the next best-performing datasets after UltraFeedback, and downsample HH-RLHF to avoid it making up most of the mixture. We then add Stack Exchange and PRM800k data to further diversify the data mixture, adding in more code and math-related data, which we empirically confirm aids reward model performance (see Table 3).

2. What is the codebase used in this paper?

We extend EasyLM (https://github.com/young-geng/EasyLM) with our own DPO and PPO implementations to make the training work on our infrastructure, similar to Tulu 2. We compare our implementation details to InstructGPT, Rainier/Crystal, Finegrained-RLHF, AlpacaFarm, and Quark in Table 10 in Appendix F.2, and referenced the hyperparameters used in these approaches (especially InstructGPT, AlpacaFarm, and the Llama-2 paper) when choosing hyperparameters for our own experiments. We will release a link to the code after the anonymity period.

3. Table 3 vs Figure 3 unmatched GSM scores

Yes, the different numbers reflect different experimental settings. In Figure 3, we limit training to 20k prompts due to the small number of GSM8k training prompts, and also to ensure that we are able to pick more relevant prompts for the mined set - we will clarify this in the figure caption.

[1] Cui et al. (2023). UltraFeedback: Boosting Language Models with Scaled AI Feedback. ICML.