The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization

We thank the reviewer for their in-depth review and are glad they find our paper to be an important contribution to the community and furthering open science. We agree that these details are important and deserve to be discussed.

On DPO vs PPO RM

Though the title of the DPO paper is “Your Language Model is Secretly a Reward Model”, only by training with DPO’s objective is this true. In particular, DPO has a formula that allows the trained LLM to be used as a reward model in conjunction with the reference LM), as defined in page 5 of https://arxiv.org/pdf/2305.18290:

$\hat{r}_\theta(x, y) = \beta \log\frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$

But this isn’t the case for PPO. PPO’s policy LM is trained exclusively to maximize RM and so its logprobs will not correspond to an implicit RM since it isn’t part of the objective function. Since both methods use an RM (DPO’s implicit RM, and PPO’s separate, explicit RM) we compare those.

Novelty of our details

The most similar work to ours is Delve into PPO by Zheng et al, 2023. Notably, our work distinguishes itself by building a scaling behavior – we showed our RLHF pipeline scales with model sizes and works across random seeds. Our investigation is also more extensive, and our findings differ (we find reward whitening to be optional and advantage whitening to be beneficial, whereas Zheng et al find the opposite). Though we find the EOS trick to be the most underreported detail, our work proposes a collection of different details to form an overall “correct” setup for the popular benchmark summarization task, which we hope will be adopted by the community. In contrast, Zheng et al focus on the best combination of PPO details for a single model on HH-RLHF.

Table 1

This is a good point. We have already changed it to another, more appropriate sample.

SUBREDDIT: r/relationships
TITLE: Me [19 F] should I be trying to help my
brother[16 M] with his life?
POST: This is my first Reddit post and I’m not
sure if I’m doing it right

Detail 23 Values

Are you referring to “values'' in Detail 13 (not 23)? We propose to treat the reward function as a per-token value function and get its outputs at non-eos tokens. We find that, in general, the reward model does not learn to be a per-token RM and gives nonsensical outputs at non-eos tokens.

We are glad the reviewer found our work to be a good manuscript to reproduce RLHF summarization.

Though the comprehensive study appears, it is somehow hard to show the connections between these different details, this potentially could be crucial.

We address this by releasing the source code and have tried to make it as minimal as possible, reducing the mental burden of understanding how these details connect with each other.

The application summarization task is specific, however, it would be better to make some details that could be transferable. Or it should be study some of these items.

This is a great point. The main objective of this work is to dive really deep into TL;DR summarization to make sure the RLHF pipeline works (and scales with model sizes). Given this knowledge, our next steps include adding other benchmark tasks such as anthropic’s HH.

The base model GPT-2 is somehow old now, at least LLaMA would be a better replacement.

We actually used pythia models instead of gpt-2 models. Furthermore, we choose pythia models for several reasons:

• pythia models compare more fairly with base models used in Stiennon et al., (2020). In particular, Pythia models were trained with ~207B tokens, and Stiennon et al., (2020) suggested that they trained the base models for “200-300 billion tokens”. In comparison, even LLaMA 1 models are trained with 1000 to 1400 billion tokens (https://arxiv.org/pdf/2302.13971).
• LLaMA models don’t have 1B models. LLaMA 1 models only have 8B to 60B models, which makes it difficult to conduct the RLHF scaling experiments shown in this paper.
• Pythia models are truly open. Unlike many open weights models like LLaMA, Pythia models release the training code, source code, and are fully transparent.

We are glad the reviewer found our work to be a good reference and appreciates how significant our contribution is. We agree our work does not propose a novel method, but we do provide novel, open results and novel findings about the most common benchmark and baseline in the field.

Many of our details are not commonplace in standard RLHF libraries which highlights their novelty and necessity to researchers e.g. TRL’s PPO 1) does not use reward normalization, 2) does not use a separate value network, 3) does not use the EOS trick, 4) reported issues with negative KL divergence (https://github.com/huggingface/trl/issues/355).

The strength of our open and reproducible baselines is also a novelty. For example, the DPO paper (https://arxiv.org/pdf/2305.18290) reported its PPO model to have 57% max win rate against reference summaries (judged by GPT4). Whereas our same-sized model has a ~80% win rate (in a separate GPT4 eval; just one random seed). By releasing code and baselines, we hope to ground the field in reproducible and significant results.

Finally, we believe there is some novelty in the correctness of our preprocessing and eval guidelines that enables the validity of the benchmark. For example, we are the first to openly replicate the scaling behaviors on summarization and we hope there is novelty for the community to now use this as a benchmark for scaling properties of RLHF algorithms.

Given that the discussion period ends tomorrow, let us know if there are any concerns you feel we haven't addressed or clarifications to improve our score. Thank you!

We are glad the reviewer found our work to be a great guide to all practitioners in the community. While we agree our work does not propose a novel method, we believe there is novelty in our investigation of details and the strength of our open and reproducible baselines. Our work helps the community build upon a better foundation (and create better science).

For example, our PPO models are well-calibrated and have a very high GPT4 win rate. The DPO paper (https://arxiv.org/pdf/2305.18290) reported its PPO model to have 57% max win rate against reference summaries (judged by GPT4). Whereas our same-sized model has a ~80% win rate. Our paper can become a strong baseline for future works and reviews to refer to. Below, we take the example completions at Table 5 from the DPO paper and compare them to ours :

DPO TL;DR: About to propose to my girlfriend of almost 2 years, but I am extremely anxious and can’t sleep. How can I relax?

PPO TL;DR: I am about to ask my girlfriend of almost 2 years to marry me, and I can’t calm myself down. Any tips on how to relax? I would like to be able to sleep tonight (and the night before)..............................................................................................................................

Our PPO TL;DR: I am about to propose to my girlfriend of almost 2 years in a couple days and I am freaking out. I can't seem to relax leading up to it. Any suggestions on how to calm myself down and relax?

As demonstrated, the PPO baseline used in the DPO paper seems broken (with excessive periods), whereas our completion is coherent and does nothing like that.

Comparing deepspeed-chat and trl: This is a good point. We noticed that TRL’s PPO 1) does not use reward normalization, 2) does not use a separate value network, 3) does not use the EOS trick, 4) reported issues with negative KL divergence (https://github.com/huggingface/trl/issues/355). We noticed deepspeed-chat does not use reward normalization.

Overall, direct comparisons are difficult to make because the full-stack setups are very different. It may be better to compare the final model checkpoints, but deepspeed-chat / trl did not release checkpoints for TL;DR summarization (the exact challenge this work aims to address).

Given that the discussion period ends tomorrow, let us know if there are any concerns you feel we haven't addressed or clarifications to improve our score. Thank you!

We are glad that the reviewers found our paper contributing to advancing the understanding and application of RLHF. We hope our careful study is helpful for future practitioners when building out their RLHF pipeline.

a) (Detail 2) How do the experiment observations change with different restrictions to the length of the query tokens (say 256 v/s 512 v/s 768)?

That is a great question! However, it might be difficult to evaluate because the reference summary has less than 53 tokens using the Pythia tokenizer, so comparing with 256 / 512/ 768 tokens might not be as meaningful.

That said, length is definitely a confounding factor, so it motivated us to do length-controlled comparisons in Figures 12 and 13.

How do the differences in SFT and Preference datasets affect the final performance of the trained model? Is there a control study, where the authors restrict the length of the examples in both datasets to check their impact on the final performance?

A reasonable comparison might be to filter out the portion of the preference dataset that has more than 53 tokens. This might be an interesting experiment for the camera-ready version if our computes allow.

What's the effect of multiple epoch training on the same SFT prompts? Do the authors observe differences in the quality of the different SFT prompts?

Great question. Multiple epochs on the SFT dataset might cause overfitting (e.g., https://www.fast.ai/posts/2023-09-04-learning-jumps/) In this work, we have used a single epoch SFT training to follow the setting in Stiennon et al., (2020) closely (e.g., “For supervised baselines, … We use a batch size of 128, and run for a single epoch.”). We find preliminary results at 1.3B scale show overfitting at more than one epoch. Nevertheless, performing multiple epochs training would be a good experiment to be included in the camera-ready version

d) "Reward whitening makes the model generate shorter outputs." Is this helpful only for summarization tasks?

Human preference datasets generally have longer answers as preferred causing length bias, so reward whitening may be a possible tool that can remedy this in regular chat tasks as well.