Learning to Generate Better than your Large Language Models

$\pi_{SFT}$ reasonably covers $\pi^{*}$

Here we assume $\pi^*$ as the policy that generates the references in the dataset. Then under the assumption that $\pi_{SFT}$ is well trained on the task with references as labels, we claim that $\pi_{SFT}$ has non-zero support over the space of generations that $\pi^*$ may cover. This claim is empirically supported by the fact that SFT has bounded non-infinity validation perplexity: given the definition of perplexity, the only way we can have bounded validation perplexity is for our model to have non-zero probability of generating the references. In other words, this means that the SFT model does have non-zero probabilities generating the references in the dataset.

Training on Positive

We would like to clarify that this choice is standard for the IMDB positive text generation task. The original IMDB dataset does not have any reviews that start negative and finish positive. Thus when training an SFT model, it is standard to focus on the positive -> positive reviews as the task-relevant data for supervised learning. This was done in RL4LMs [1], Direct Preference Optimization [2], as well as popular RLHF codebases such as TRLx [3].

[1] Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, Ramamurthy et al. 2023

[2] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, Rafailov et al. 2023

[3] https://github.com/CarperAI/trlx

Thank you for your review. We hope to address your concerns here

Chatbot Arena with GPT4 Evaluation

Thank you for your suggestions. We would like to clarify the scope of our paper: we focuses on RL finetuning of specific downstream tasks (e.g., generating positive reviews or reddit post summary) instead of generating a general purpose chat model. None of our tasks were for dialogue and general chat/instruction following and thus we don’t find Chatbot Arena to be a relevant evaluation platform for this work. Our evaluation and tasks closely follow other published RLHF papers in 2023, e.g., RL4LM [1], which is published in ICLR 2023.

Also we aimed to use open-source models for both training and evaluation of our models. Open-source models remain publicly available, guaranteeing the reproducibility of our experiments and results. Whereas closed source models access can be depreciated [2] at any time, making it impossible to reproduce our results.

[1] Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, Ramamurthy et al. 2023

[2] https://openai.com/blog/gpt-4-api-general-availability

Performance Gain

Please see our updated results we posted along with the rebuttal for more results with TL;DR. For a quick summary, we show that when doing best-of-n sampling with our RL policies (PPO included), PPO++ shows greater performance improvement than PPO.

Perplexity Goes up but Output Perplexity does down

We would like to clarify that perplexity is a measure of fluency rather than a measure of how well a generation does on a task. Our evaluation model for output perplexity is GPT2-large. While we agree GPT2-large is not good at the task in terms of optimizing the reward, it is still a valid model that has been widely used to evaluate whether or not the generation is natural and fluent (e.g., see [1-10]).

The overall objective that we are optimizing for is $J_{ppo}$, which includes three terms: reward, KL-constraint (i.e., a fluency constraint), and MLE-constraint (i.e., a fluency constraint). We attempted to capture each term’s significance during evaluation through reward, output perplexity, and perplexity respectively. Both output perplexity and perplexity are not perfectly aligned with the terms in the objective but are somewhat correlated. The policy that we are computing a KL-constraint with respect to is different from the output perplexity policy. Furthermore, we would like to point out that there could be high-scoring generations with low fluency as well as high-scoring generations with high fluency. The output perplexity being better after the RL procedure shows that we are maintaining fluent generations while improving the reward that we are optimizing on.

[1] DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts by Liu et al. 2021

[2] PLUG AND PLAY LANGUAGE MODELS: A SIMPLE APPROACH TO CONTROLLED TEXT GENERATION by Dathathri et al. 202

[3] Plug-and-Blend: A Framework for Plug-and-play Controllable Story Generation with Sketches by Lin et al. 2021

[4] POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training by Zhang et al. 2020

[5] COCON: A SELF-SUPERVISED APPROACH FOR CONTROLLED TEXT GENERATION by Chan et al 2021

[6] FUDGE: Controlled Text Generation With Future Discriminators by Yang et al. 2021

[7] Mix and Match: Learning-free Controllable Text Generation using Energy Language Models by Mireshghallah et al. 2022

[8] A DISTRIBUTIONAL APPROACH TO CONTROLLED TEXT GENERATION Khalifa et al. 2021

[9] On Learning Text Style Transfer with Direct Rewards by Liu et al 2021

[10] MASKGAN: BETTER TEXT GENERATION VIA FILLING IN THE ___ Fedus et al. 2018

Scaling Law Testing

Here are the LLM breakdowns for our experiments

• IMDB: GPT2-base (117M parameters, Causal model)
• CommonGen: T5-base (220M parameters, Seq2Seq model)
• TL;DR: GPTJ (6B parameters, Causal model)

We have the exact hugging-face model specifications in the implementation details in the appendix. Note that we aimed to try varying types of LLMs and sizes of LLMs across our experimentation. We agree that it would be interesting to see how the performance scales w.r.t. Policy model size, but in this work we hoped to focus on improvements beyond PPO when all design choices were held equal.

Output Perplexity

We computed Output Perplexity with GPT2-Large. We thank the reviewer for pointing this out and will make this clearer in the final version.

Hello, reviewer. Please review the author's response. Does the response address your concern about the experimental results?

This response, while clarifying some points, does not address my concerns. Let's consider each in turn.

• The benchmarks used are somewhat lacking. For text generation in 2023, most comparisons for state of the art methods are made by comparisons to other LLMs because fixed metrics have been found to be lacking. A comparison using a framework like Chatbot Arena (Zheng et al. 2023). would have made the results significantly more convincing. These often use GPT-4 as an evaluator, which has now been shown to be provisionally better than traditional metrics (Liu et al 2023, Min et al 2023, Zhou et al 2023, inter alia).

While the authors immediately mention that ChatBot Arena is for dialogue (fair enough, but I did not ask them to use ChatBot Arena, just a similar setup), the reality is there is still no direct qualitative study. Authors point out that they tried to use open-source models. Fair enough, but then human evaluations for a qualitative study of what these model outputs actually look like should be done. The scores given on these limited set of tasks are simply insufficient to conclude much.

• Even with these quite simple benchmarks, the resulting differences on metrics are small for the task where control is more significant, CommonGen. In IMDB the task-specific metric really only checks if the sentiment score is correct, which is very little information to be using to validate outputs.

Even with the extra results, the gains are small.

• The fact that perplexity goes up but output perplexity goes down on IMDB is worrying. I can understand why perplexity would go up: the model is likely collapsing onto a smaller subspace of possibilities, which may still be higher quality. However, the fact that the output perplexity also goes down is worrying: it indicates that after optimization the resultant model is more predictable to LMs which is usually not a good sign for tasks that LMs are bad at in the first place. The results on TL;DR are similarly quite small.

In their response, the authors describe how perplexity is a measurement of fluency (which is not exactly true, but it can be a decent proxy), and how this is not a problem. I disagree—we have seen almost no evidence that the resulting generations are actually desirable. Again, the lack of a qualitative comparison of what generations actually look like is the only way to find out whether models are over-optimizing certain metrics, which is precisely what the objective incentivizes. In their response, the authors justify the metrics by saying they are close to the training objectives. This is not the point—the question is whether the result of the objective is even desirable in the first place, for which there is still little evidence.

• I’m disappointed to see there’s no study of how this technique scales across model sizes: one could imagine doing this for smaller models, incurring relatively low compute expenses, but no such experiments were conducted. As it is, it is not clear how much these results will generalize to the ever-growing zoo of LLMs.

In response to my concerns about scale, the authors listed the specifications of the models used, suggesting that these details were in the appendix. I am aware of the models used, but using different models of slightly different sizes on different datasets does not tell us whether scale will impact the use of these techniques. Indeed it actually makes it impossible to deconflate whether the specific model being used is the cause of any gains achieved, rather than the technique itself.

My score remains unchanged.

Thank you for your review. We hope to address your concerns here

Motivation

The main motivation of this work is to improve on-policy reinforcement learning algorithms that have contributed to the success of ChatGPT. Despite their success, on-policy reinforcement learning algorithms can be very inefficient in exploration, and our goal is to address this issue. If we can address this issue then we can train policies that perform better. Our key observation, stated in our introduction, is that modern LLMs have impressive general language capabilities, which can help guide exploration for training models with reinforcement learning. Because there are numerous ways to provide guidance, we evaluate all combinations of guide paradigms (i.e., rollin-rollout interaction paradigms) to see which ones are better suited for text-generation.

The rollin-rollout scheme offers a paradigm to investigate various different possible ways that a guide policy can help with exploration. When rollin and rollout are with the learner policy then we reduce to normal PPO without any guidance. Rollin with a guide policy and rollout with a learner policy, means the guide policy provides more referent states where the learner policy can reset and perform optimization. These good reset states equate to the learner policy receiving high rewards much early during training. Seminal RL work like CPI [1] has shown that a good reset distribution can make RL to learn a higher quality policy. On the other hand, rollin with the learn policy and rollout with the guide policy means that the guide policy is providing feedback on the expected returns the learner partial generations will receive. Seminal imitation learning work like DAgger [2] shows that providing expert feedback on states generated by the learner is helpful for training a high quality policy. In this case, the guide policy is treated as an expert policy which the learner imitates and surpasses eventually.

[1] Approximately Optimal Approximate Reinforcement Learning, Kakade and Langford 2002

[2] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning, Ross et al. 2011

Theoretical Justification

Thank you for this critique. Due to the space limit, we mainly provided intuitive explanations of the benefits of using a guide policy. Please see the answer above where we provided more connections to the seminal works in the RL and IL literature. If that’s not helpful, could we ask the reviewer to provide more actionable details on how we could make the section more self-contained?

Performance Gain

Please see our updated results we posted along with the rebuttal for more results with TL;DR. For a quick summary, we show that when doing best-of-n sampling with our RL policies (PPO included), PPO++ shows greater performance improvement than PPO.

Ablation Mixing Parameter

Thank you for this recommendation. We did not find this parameter to be particularly sensitive in our experiments, but we agree that this would be a valuable addition to the paper. We will work to add this to the appendix in the final version.

Stronger Black Box Model such as LLama2/GPT4

We definitely agree with the reviewer that this would be very exciting future work since in principle, the use of Llama2 as $\pi_g$ is perfectly valid. Note that given the growing concern of data contamination in LLM evaluations [1], experimental design becomes more difficult when evaluating on popular benchmarks such as IMDB and TL;DR summarization which have not been explicitly defined as held-out datasets by models such as Llama2 [1].

[1] Llama 2: Open Foundation and Fine-Tuned Chat Models, Touvron 2023

Setting $\beta$=1

We would like to clear up a potential misunderstanding. When the mixing parameter is set to 1, we mean that all trajectories are mixed. That is every trajectory has some amount of $\pi_g$ tokens as well as $\pi_{\theta}$ tokens. For example in AggreVaTeD, this means that every trajectory has rollins from $\pi_{\theta}$ and rollouts from $\pi_g$. When doing the policy update, we would then update with the samples from $\pi_{\theta}$

Missing Results: D2LOLS TL;DR

Thank you for pointing this out. We will work to add these results to the appendix for the camera ready. We felt that investigating the individual components of D2LOLS (PPO++ and AggreVaTeD) was more valuable given our limited computational budget.

Missing Related Work

Thank you for pointing out this interesting work. We definitely missed this and will add a discussion for this in the related works. The main difference between STG and RLGF here is the different mixing strategies employed. In RLGF, we obtain entire partial sequence feedbacks while in STG, a mixed trajectory may involve a single token being replaced.

Hello, reviewer. Please review the author's response. Does the response address your concern about motivation and theoretical justification?

Thank you for your review. We hope to address your concerns here

Experimental Results

Please see our updated results we posted along with the rebuttal for more results with TL;DR. For a quick summary, we show that when doing best-of-n sampling with our RL policies (PPO included), PPO++ shows greater performance improvement than PPO.

Clarification of Guide and Learn Policy

Yes, you are correct. In our experiments, both our starting PPO policy $\pi_{\theta}$ and our guide policy $\pi_g$ are initialized with $\pi_{SFT}$. However, it's important to note that while $\pi_g$ remains fixed as a black box policy, providing PPO with additional states through the nucleus sampling decoding strategy, PPO itself is not fixed and gathers samples using the softmax sampling decoding strategy.

What LLMs were used?

• IMDB: GPT2-base
• CommonGen: T5-base
• TL;DR: GPTJ (6B)

We have the exact hugging-face model specifications in the implementation details in the appendix. Note that we aimed to try varying types and sizes of LLMs across our experimentation.

Output Perplexity

We computed Output Perplexity with GPT2-Large. We thank the reviewer for pointing this out and will make this clearer in the final version.

GPT4 Winrate

We did our best to use open-source models for the entirety of our work. Open-source models remain publicly available, guaranteeing the reproducibility of our experiments and results. Whereas closed source models’ accesses can be depreciated [1] at any time, making it impossible to reproduce our results. We will release our models to the huggingface hub for public usage and for full reproducibility of our results. In some preliminary experiments we found the relative winrate trends between LLama2 70B and LLama2 13B to be aligned. Working within our computational budget, we found that using Llama2 70B was difficult to execute for every experiment.

[1] https://openai.com/blog/gpt-4-api-general-availability

Hello, reviewer. Please review the authors' response. Does the response address your concern about performance?

Thank you for taking the time for a thorough review. We hope to address your concerns below:

Choice of Baselines

We agree with the reviewer that many of the cited works are relevant literature in the RLHF space. We’d like to emphasize two points:

1. Our work focuses on investigating direct algorithmic improvements to PPO when using a reward signal
2. Our work extends beyond preference data for reward signals. For example, we tested on CommonGen which uses an automatic metric CiDer-D and SPICE as the reward function for the task. We aimed to show that given any reward signal, whether it be a preference reward model in RLHF tasks or an automatic metric such as CiDer-D, RLGF can better optimize than PPO.

We agree that it would be interesting to explore combinations of RLGF with many of the other methods; however, we believe this is well beyond the scope of a single paper, especially given the reviewer’s observation that this investigation is already quite dense.

Paper Structure and Novelty

Thank you for the suggestions. We can definitely see the merit of focusing on one or two algorithms in detail. On the flip side, we felt that demonstrating the full combination of guide-policy and policy interactions in the policy gradient formulation would be a valuable contribution to the space of Deep RL algorithms for text generation. As the reviewer pointed out, many of these ideas were explored in different domains, but we would like to emphasize that most of them did not have a deep algorithm counterpart with this work being the first to unify all of these methods into a single, policy gradient implementation. Furthermore, by investigating all these combinations, we hope we convinced the readers of which rollin-rollout interaction paradigms are better suited for text-generation than others.

On a separate note, many of the works cited were concurrent work to ours, with all of them being archival preprints or just announced Neurips acceptances at the time of submission.

Hello, reviewer. Please review the author's response. Does the response address your concern about the choice of baseline?