Understanding the Effects of RLHF on LLM Generalisation and Diversity

We thank the reviewer for their positive comments on the paper’s presentation and clarity. We’re glad you found the study comprehensive. We now address your comments in detail.

the claims made in the paper are not very well-justified by experiment results, and some experiment details are lacking. Only two sets of experiments, namely summarization and instruction following, are conducted on one single model (Llama-7B), yet the paper makes a general claim about the effects of RLHF. More experiments on different models at potentially different scales could be helpful, but still, the contribution seems to be incremental.

In fact, for the summarisation task we ran experiments with both LlaMA (7B) and OPT (125M, 350M, 1.3B, 2.7B, 6.7B) models. The results with OPT are described in Appendix J. These results show the same trends as those with LlaMA, demonstrating that our conclusions generalise well across different base models, thus further strengthening our claims about the comparison between RLHF and SFT. While we cannot be entirely sure whether these conclusions hold in all possible settings, we conducted extensive experiments on two different base models, two different tasks with five evaluation sets in total, across six different model sizes, using five different evaluation metrics for generalisation and diversity, resulting in training and evaluating over 80 models. All these results point to similar conclusions. These experiments are very extensive, and required non-trivial resources. While it would always be better to have more experiments, we believe our analysis is rigorous and extensive enough to sufficiently support our conclusions.

Following your suggestion, we will make sure to revise our claims to ensure we are not overclaiming and caveat that these conclusions are based on certain settings and models.

In addition, we would like to note that the tasks we tested models on are some of the most commonly used in practical applications of LLMs, namely summarization and instruction following (which is also similar to single turn dialogue, another popular application), so we think our results can be of broad interest to researchers and practitioners alike. Even if our results only held in the instruction following setting this should still be considered valuable for the community. Instruction-following is clearly a very important and timely topic: it has a workshop at NeurIPS (https://an-instructive-workshop.github.io/), lots of interest both in academia and industry, and is one of the most common applications that LLMs are fine-tuned and then deployed for. Thus, we believe improving our understanding of methods even in this domain alone is useful and relevant for the ICLR community.

[Response Continued Below]

Dear reviewer,

We appreciate the time you have dedicated to reviewing our paper. In the response above, our responses have addressed your concerns regarding the novelty and clarity of our work, and so we hope you will consider raising your score. Otherwise, please let us know if there is anything preventing you from recommending acceptance of the paper.

Thanks for your response. We are glad to have addressed your minor concerns.

We believe the insights provided by our paper are new, and especially that they have been shown robustly to hold across a variety of settings. To compare our experiments and conclusions to each of the works you shared:

1. We think the reviewer is pointing to the Modal representativeness paragraph of this work as evidence for a lack of diversity from RLHF. However, this experiment doesn't show that RLHF (as opposed to SFT, or some other part of the pipeline for training text-davinci-003 that we don't know about) is the cause of this lack of diversity. Further, the experiment is limited to multiple choice question answering, and only considers one model.

2. This work does show that something similar to RLHF (RKL-DPG in that work) does reduce per-input output diversity and increases in-distribution reward. However, the metrics for diversity are limited, they don't investigate across-input output diversity, then don't compare RLHF to SFT or BoN, and they don't look at out-of-distribution generalisation. Further, their work is on much smaller models and simpler tasks than the ones we investigate.

3. This work again shows that RLHF increases in-distribution reward but decreases per-input output diversity, and does compare to SFT in this setting. However, it again only uses limited diversity metrics, doesn't evaluate cross-input output diversity, and doesn't investigate out-of-distribution generalisation. Further, the red-teaming setting investigated in that paper is plausible quite different from standard settings like instruction-following that we investigate, making it unclear whether their results would hold in these other settings.

4. This work shows that RLHF (Ziegler in that work) decreases per-input diversity somewhat while increasing in-distribution reward, but it's experiments are on small models (GPT-2), they only investigate per-input diversity, they use limited diversity metrics, and they don't investigate out-of-distribution performance.

5. This work compares SFT, BoN and RLHF, but does not have any experiments on out-of-distribution generalisation or output diversity, and so is quite unrelated to our work.

Overall, none of these works investigate out-of-distribution generalisation, all of them use more limited diversity metrics, and none of them evaluate cross-input output diversity. There has been no study comparing the tradeoff between out-of-distribution generalisation and output diversity (both across-input and per-input), and hence our insights concerning the trade-off between these metrics of importance are novel.

At the minimum, could you plot the pareto frontier of diversity metrics versus OOD generalization across models and tasks?

We have added a plot to this effect in the summarisation setting, with ID and OOD winrates vs per-input and across-input diversity measures, in appendix L (Figure 24).

What would be potential ways that could push the pareto frontier outwards? Why is KL-divergence not sufficient in maintaining diversity? How about other entropy-based regularization?

While we agree these are all interesting research questions, and we are interested in further research investigating them, we believe our work as it currently stands is sufficient for acceptance to ICLR, and that answering those research questions is future work. Even if you believe the work is not worthy of acceptance, would you consider raising your score from a 3 to a 5, given our changes addressing your minor comments and our response above?

We thank the reviewer for their positive comments. We’re glad you found the work well-written and empirically sound. We’ll now respond to your comments in detail.

The major concerns stem from the fact that the empirical evaluation heavily relies on the training quality of each algorithm, and the reviewer is uncertain about whether RLHF is trained to a high standard

Given the strong performance of the RLHF and BoN policies relative to SFT when evaluated by humans and GPT-4, and the high accuracy of the reward models (in fact slightly higher than reported in [2] - our RM gets .76 while the best model in [2] gets ~.74) we believe this sufficiently demonstrates that our RLHF models are trained to a high standard.

Further, our conclusions rely on the relative ranking of RLHF, SFT and BoN and gaps in performance between methods. Even if our RLHF implementation is not fully optimal, our conclusions would still be correct, as an improved RLHF performance doesn’t change the ranking of methods in terms of generalisation as it already performs best in absolute terms on all evaluation sets.

Do empirical conclusions heavily depend on the training status of the reward model and PPO? The reviewer observed that this paper freezes some layers of LLMs when using RLHF, which may limit the representation capacity. Thus, the reviewer questions whether the training of RLHF is of good quality.

We agree that empirical conclusions will depend on the performance of the reward model and PPO algorithm, as well as on the performance of the SFT optimisation.

For the summarisation experiments, we believe our implementation is high quality for all these components, as our results mostly reproduce those in the literature: Our reward models gets .76 validation accuracy compared to ~.74 in [2]; our gpt-4 preference win rate for RLHF is .46 and for SFT is .36 while [1] gets ~.57 for RLHF and ~.41 for SFT (note there are slight differences in how GPT-4 is prompted). Slightly lower numbers should be expected given that we are freezing 80% of parameters. On freezing of layers, this happens for SFT, reward model training and PPO, so is unlikely to affect one method more than another. Thus, we’d expect the conclusions of the paper to hold even if all parameters were fine-tuned. Also, work such as [3] shows that impressive capabilities such as long-form dialogue and improved adversarial robustness can still be achieved when freezing a similar proportion of layers. Finally, in table 14 in the appendix we present some comparisons of frozen and unfrozen SFT and reward models based on OPT, and show only a small drop in performance when freezing 80% of the layers, the same as in the rest of our experiments.

For the instruction-following experiments, we use the models released in AlpacaFarm [5]. These models get good performance in both GPT-4 and human evaluations in both their work and ours, and we believe that their implementation is of high quality.

Could this paper provide the evaluation accuracy of the reward model and training curves of PPO?

Thank you for the suggestion to improve the thoroughness of the paper. We have updated the appendix and added a section (Appendix K) with the training reward and KL divergence curves for the PPO model and evaluation accuracy of the reward model, in the summarisation task. The reward model (with 80% parameters frozen) still achieves 75.8% validation accuracy, which is approaching the maximum given the levels of inter-annotator agreement reported in [2] being close to 25%. The PPO model converges at a KL divergence of approximately 15, and a validation reward of approximately 0.8.

For the instruction-following task we used models released in AlpacaFarm ([5]), so we don’t have access to training reward curves or reward model accuracy.

[Due to character limits, the response continues in a child comment.]

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, https://arxiv.org/abs/2305.18290

[2] Learning to summarize from human feedback, https://arxiv.org/abs/2009.01325

[3] Improving alignment of dialogue agents via targeted human judgements, https://arxiv.org/abs/2209.14375

[4] TextGAIL: Generative Adversarial Imitation Learning for Text Generation, https://arxiv.org/abs/2004.13796

[5] AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback, https://arxiv.org/abs/2305.14387

[6] Evaluating the Evaluation of Diversity in Natural Language Generation, https://aclanthology.org/2021.eacl-main.25

Thanks for the detailed response.

• The concerns about the quality of reward models are well-addressed.

• I find the KL divergence to be quite large. According to the reviewer's experience, the LLM tends to overfit when the KL divergence is larger than 1, as observed while working with Llama2-7B on datasets such as full-hh-rlhf. I have reviewed the literature and found that the provided KL results are consistent with those reported in [1]. The provided figures seem to suggest that KL regularization does not prevent the achievement of high reward scores. Therefore, the reviewer remains skeptical about the PPO training, but can accept the current results.

• A typo exists in Equation (4) in Appendix.

[1] Learning to summarize from human feedback, https://arxiv.org/abs/2009.01325

Dear reviewer,

We appreciate the time you have dedicated to reviewing our paper. In the response above, our responses have addressed your concerns regarding related work and the quality of the training algorithms, and so we hope you will consider raising your score. Otherwise, please let us know if there is anything preventing from more strongly recommending acceptance of the paper.

We thank the reviewer for their positive review. We’re glad they found our investigation thorough and valuable, and the paper clearly presented. We now respond to your comments on the summarisation task.

Firstly, for clarification and to avoid confusion, we’re using the LLaMa 7B model, not LLaMa 2 7B, although your comments are still valid. Secondly, are you aware of any papers that illustrate the performance of these pretrained models on CNN daily mail? We would expect them to do reasonably well, but not as well as models after fine-tuning. For example figure 4 in [1] shows pretrained models still doing comparably to transferred SFT models but much worse than RLHFed models in a similar setting (train on TL;DR, test on CNN/DM), although how the pretrained models were trained is not discussed.

In your comment you imply that TL;DR summarisation is harder than CNN/DM summarisation. We haven’t seen evidence for this in our work or in previous works; could you share where you’ve seen this result so that we can cite and discuss it in our work? We might expect CNN/DM to be a harder summarisation task, as the inputs to summarise are much longer, and are likely to be more densely packed with information (as news articles) that is hard to summarise concisely as compared to reddit posts.

Finally, it’s unfortunately not possible with our current resources to train RLHF models on CNN/DM since it would require collecting a lot of high-quality human preference data for training a reward model on this dataset which is expensive to do. The preference datasets we use were open-sourced by OpenAI in [1], but they didn’t release a large enough preference dataset for CNN/DM to enable reward model training. We agree it would be an interesting experiment to run, and we thank the reviewer for their suggestion. Our prediction is that the results would be the same: RLHF outperforms SFT both in-distribution and out-of-distribution, but at the cost of output diversity.

We thank the reviewer again for their complementary review and useful comments. We hope we've answered all your concerns, but please let us know if you have any remaining questions.

[1] Learning to summarize from human feedback, https://arxiv.org/abs/2009.01325

Thank you for the thorough response. I correct my statement that CNN is an easier summarization task. As the authors stated, when seeing prior work such as [1], we see that CNN/DailyMail summarization is a more difficult summarization task than TL;DR.

I also understand and agree with the authors' point about the lack of high-quality preference data for CNN barring them from performing the same analysis that they did on the TL;DR summarization task.

I will be maintaining my score.

[1] Statistical Rejection Sampling Improves Preference Optimization, Liu et al 2023

We have just uploaded an updated revision with Figure 5 updated with error bars. Figure 5 shows the across-input diversity, and for SFT and RLHF we calculate error bars using the standard deviation of the metric across different samples of the output for each input.