Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

We thank the reviewer for their detailed feedback. To clarify, our primary goal is to effectively train language models by exclusively using existing language data and sequence-level language classifiers. Thus, we compare offline RL [22] methods that exclusively learn from a static dataset, to reduce training costs and improve efficiency.

Response to the weaknesses:

1. "a special case of TRPO/PPO in the bandit setting"
   We believe this comparison is an oversimplification. PPO/TRPO are online RL that assume the training trajectories are generated by constant interaction with the environment. Our method is offline and assumes access to only static data. Accordingly, value estimate, advantage, and the training process differs significantly from the online methods.
   Thank you for references to PPO in single action bandit setting; we will update Table 1 and the intro. Most modern implementations of PPO for LMs [12,13,14] use per-token action assumption with trajectory level sparse reward (only the last token gets reward). This makes the value estimation of intermediate tokens difficult. Instead of on-policy updates, we use a sequence-level value estimate of reference policy that we cheaply train from its validation performance. To our knowledge, such an easy-to-compute sequence-level value estimate has not been used before for offline RL training of LMs. We also show a combination of algorithmic modifications to improve training efficiency and performance (point 6).
2. "Eq. 5 is very similar to Eq. 4 in AWAC [3],"
   While it is true that Advantage-weighted Regression [3] is a well-studied method in offline RL, there are other differences from A-LoL. AWR maintains two identical models for value estimate and policy and it trains both models on samples from D. Here, D can collect additional off-policy trajectories. In contrast, A-LoL estimates sequence-level value on top of a frozen LM’s validation performance only once before target policy training. This allows A-LoL to discover a subset of positive advantages within the static training data, since negative advantage instances are detrimental to performance (point 6).
3. "suffer from exponentially large action space"
   Large action space is not an issue for A-LoL due to its use of offline data (that can be human-written). In contrast, both online on-policy and online off-policy methods use LM-generated data with high variance due to the large action space. While there exist a few studies with per-token [6] and phrase-level [15] reward functions, most prominent RLHF implementations use a sequence-level reward [1,2,12,13,14] and thus, we compare with the same setting.
4. “Weighted behavior cloning has been used in prior NLP papers”
   Thank you for sharing the related works! We use weighted behavior cloning as a reward-based offline RL baseline and will add a paragraph discussing these additional related works.
5. “reward for PPO is a good-or-bad classifier”
   Common RLHF benchmarks and PPO implementations [12, 13, 14] use sequence-level preference classifiers as reward models. For a fair comparison, we keep the same initial reference LM and reuse data, and reward models [21] from PRO’s [20] experimental setup (a preference-based offline RL baseline). In the PPO baseline, we use nucleus sampling (top-p=0.95) and also include entropy in the loss term to encourage diversity.
   All offline RL baselines use the same data, but vary in diversity. In particular, DPO [18] loses diversity, whereas all reward-based baselines and A-LoL methods show better diversity.
6. "exogenous components, such as (1) importance clipping, (2) discarding negative advantage datapoints, (3) prioritized sampling.”
   Thank you for pointing this out. We compare the impact (1) and (3) in the ablations in Appendix B.3 (Figure 6). We find (1) importance clipping is crucial for A-LoL; it degenerates to NaN losses without it. We find (3) priority sampling is notably better than random sampling. We added a new ablation for (2) and found that negative advantage is detrimental. For a fair comparison with reward-based baselines, we employed reward-based priority sampling (page 4), but do not discard the negative advantage data as they do not have an internal value estimate like A-LoL.

1. “sequence-level value estimate has been used in RL4LMs [14]. In fact, Line 2 of your Algorithm 1 is the same as Line 9 of Algorithm 1 of RL4LMs.”
   It is not correct that RL4LMs use a sequence-level value estimate. RL4LMs makes the per-token action assumption [12,13] and the sequence-level reward arrives at the end of the episode (Section 3.1 in RL4LMs[14], official code). The per-token reward definition (eq (1) in RL4LMs) also includes a per-token KL penalty. Subsequently, Line 9 in RL4LMs Algorithm 1 is a per-token value estimate from the samples drawn from the masked LM policy and is updated throughout the training. In contrast, our Line 2 in Algorithm 1 estimates a scalar value for the entire input (single-action assumption) on the validation performance of reference LM only once before the training. This reference LM value estimate allows the removal of negative advantage in offline training and enables prioritized sampling.
2. “algorithmic modifications are very standard in previous offline RL and NLP papers”
   We acknowledge that clipping and reward-based prioritized sampling are modifications suggested by two independent works and are respectively cited as inspiration in Section 2.2 (page 4). However, computing the cheap sequence-level value estimate in an offline setting and removal of negative advantages is novel in our technique. We are able to use advantage-based prioritized sampling because the reference LM advantages stay static. Overall, we tried multiple combinations of these optimizations and developed an offline RL training recipe for LMs that works very well in practice.
3. “comparison with AWAC … the main differences are with problem setting: offline v.s. online off-policy, and stepwise v.s. sequence-wise; rather than algorithmic differences.”
   There are key algorithmic distinctions in the value estimate of A-LoL and AWAC along with the other mentioned differences in the problem setting. Algorithm 1 lines 5,6 in AWAC [3] uses the samples from off-policy trajectories in $D$ to continually update both the value model and policy model which are two separate networks with identical architectures (Appendix C in AWAC). Instead, A-LoL reuses the bulk of frozen parameters of $\pi_{ref}$ and trains its sequence-level value estimate only once before target policy training and, therefore, is not only more computationally efficient than AWAC but is also able to remove negative advantage data and employ prioritized sampling. Our derivation also naturally yields importance weight in our main equation, which across all our experiments, has shown improvement in the performance and stability.
4. “how is the estimated $V(.)$ different from $R(.)$?”
   Concretely, we define a value layer (MhA + MLP) on top of the frozen reference LM which only uses the hidden states of any given input $V_{\pi_{ref}}(x)$ and emits a scalar value as output. To train the value estimate, we employ MSE loss on the rewards ($R(x, y’)$) obtained from one reference LM output ($y’ \sim \pi_{ref}(x)$) for all the inputs in the validation set $D_v$ (280 instances in HHA task). Our value function gets a crude estimate of how well the reference LM is expected to behave on any input. In our experiments, the MSE loss of this value estimate approaches 0 but is never equal to 0. Since, the reference LM cannot perfectly reproduce the performance of all the static trajectories in $D_{tr}$ and there is some delta between the rewards of the training set and the value estimate of the reference LM (line 3 of Algorithm 1 in our work).
5. “low advantage means $R - V < 0$. In other words, it means that the quality of those texts is "below average"”
   Instead of “below average”, we view the negative advantages as instances where reference LM is expected to generate better quality outputs than the ones in the static training set. Intuitively, there should be no benefit of training on these poor quality train trajectories and subsequently removing negative advantage instances yields benefits in all the tasks. We refer to these instances as suboptimal or noisy trajectories. In experiments with known good and bad data splits, A-LoL automatically finds a larger subset of negative advantages in the bad data split (48% of Reddit downvoted comments compared to 36% of Reddit upvoted comments).
6. “in Table 3, the best performing method in terms of Human evaluation is "A-LOL seq."”
   This is not correct. A-LoL seq. is considered more safe and helpful according to GPT-4, but humans rate A-LoL more helpful than A-LoL seq. (53.5% win vs 49.3% win) and A LoL seq. more safe than A-LoL (63.3% win vs 46.7% win). The discrepancy between GPT-4 and human eval is expected as we saw only 72% alignment in their labels. However, human evaluation should be considered more trustworthy.

We thank the reviewer for considering our responses to the previous concerns and improving the score. However, we would like to address the additional comments raised.

1. “removal of negative advantage comes from Eq. (3) of the CRR paper [24], except that you are dealing with the bandit setting and have added several other techniques.”
   The eq (3) in CRR uses a sign operator on target policy advantage as follows $\mathbb{1}[A_{\theta}(s,a) > 0]​$. The authors of CRR mention that "Intuitively, Eq. (3) entails BC on a filtered dataset where the filtering increases the average quality of the actions we learn from”. This is different from A-LoL, which uses reference policy advantage estimates and also naturally includes an importance weight term. Thus, the sum total of various components and training procedure of A-LoL is very different from CRR. Moreover, CRR is developed for continuous control tasks where reward signals and data assumptions are very different from the language domain. A-LoL is specifically designed for the large action space and sparse reward signal of language tasks and thus, creates a unique offline training recipe for LMs.
2. “I think the practical efficacy of this view is very likely tied to the specific datasets and tasks that you are working on”
   We didn’t apply any special criteria in our task selection that would make the removal of negative advantages particularly beneficial. In our experiments, we tested 4 different language generation tasks each with a different set of rewards and data sources - HHA uses an unknown 52B LLM generated responses (Sec 4), Reddit uses human-written internet comments (Sec 5), Commonsense reasoning uses GPT-3 generated data (Appendix C.1) and Knowledge-grounded dialog uses crowd-collected conversational data (Appendix C.2). Depending on the quality of the data source, reward distribution, its supervised reference LM, A-LoL finds 10% to 55% of the data as negative advantage.
3. “the success of A-LoL sequence may contradict the single-action assumption”
   We would like to reiterate that, although A-LoL seq. improves in diversity and safety, it is not the best-performing method in all the metrics (since humans rate A-LoL more helpful than A-LoL seq. in HHA evaluation Table 3). The net benefit of these different variants of importance weight (full-trajectory, per-token, and KL) is minor compared to the benefit brought by the entire suite of A-LoL methods: changing the reward term with reference LM advantage (as evidenced by all 4 task results in Tables 2, 4, 5, and 6).

We thank the reviewer for their feedback. We are encouraged that they find our discussion about online vs offline methods that use preference or rewards useful and our motivations and technical contributions clear.

Responses to the Weaknesses:

1. “For the “ref free” variant: Is it justified to use 1 as importance weights?”
   Yes. A-LoL reference free is effectively the supervised LM advantage multiplied with negative log-likelihood loss. Equation-wise, it is very similar to the weighted behavior cloning baseline which is the reward multiplied with the log-likelihood. Another way to look at reference-free is to use the clipping parameter $\epsilon = 0$. A-LoL reference free should be viewed as an ablation as we mainly compare with it to investigate the impact of importance weight.
2. “For the “sequence” variant: it’s essentially saying a1 * a2 * … * aT * (b1 + b2 + … + bT) = a1 * b1 + a2 * b2 + … + aT * bT… an explanation of why this is a good approximation will be helpful…”
   To clarify, the sequence variant importance weight is an approximation and not mathematically equal. Intuitively, it tries to reweight each token’s likelihood with its token importance weight while still using trajectory-level advantage values. Empirically, we find this approximation to improve diversity in all the tasks while also showing better training convergence than other A-LoL variants. We also find quantitative differences in how the sequence variant affects the importance weights (more in answer to question 2 below).
3. “did the authors train PPO methods for more training steps.”
   In order to compare the training efficiency of each method, we allocated a similar number of gradient updates to each baseline method including PPO. Our primary goal is to get the best possible policy from existing data while reducing computing costs. We ran for 2.6 times the training steps with PPO but didn’t notice any significant improvement.
4. “Phrasing of the main question in paragraph 1 … is already yes given the literature in the past few years… I’m also confused about what “similar to PPO” means”
   We thank you for sharing more related work. The main distinction with previous works is that A-LoL doesn’t need model-generated data during target policy training and can use arbitrary classifiers as rewards. Both cringe loss [1] and unlikelihood training [2] require contrastive trajectories from the learner model during its training loop. Director [3] needs to train the classifier alongside the target LM and cannot use arbitrary pre-existing classifiers as rewards. By “similar to PPO”, we mean the ability of PPO to incorporate arbitrary non-differentiable rewards while using pre-existing language data. We will update the text to highlight these distinctions.
5. “No on-policy RL performance on Reddit generation task”
   We thank you for this suggestion. We conducted additional experiments with PPO on the Reddit task with a sum of five classifiers as a reward and found that the best model achieved total reward of 3.06 (about 0.1 less than our best A-LoL method). However, their corpus diversity turned out to be very bad with most responses reading like “This is my favorite …” or “I don’t know …”. Overall, PPO gets stuck in a local maxima by optimizing three out of five rewards (fluency, safety, and tf-idf scores) and ignored the other two (engagement and upvote probability). We included this discussion with qualitative examples in Appendix C.3 (Tables 7 and 8).

We thank the reviewer for their feedback and are encouraged to see that they find our method more straightforward in language settings compared to modern PPO implementations for RLHF. We also appreciate that the reviewer found our method “different enough from other alternatives” and allows “flexibility in terms of optimizing for versatile rewards”.

Response to weaknesses:

1. “The paper seems to be suggesting that there is an issue of DPO reward hacking with the common reward function, is there a concrete example supporting this claim?”
   During our analysis of high-reward responses in the helpful subset of HHA task, we found that responses with list of items were assigned unusually high rewards. For example, even an empty list (“\n- \n- \n- …”) generated by DPO received 0.84 reward. Subsequently, we found DPO to generate a total of 294 responses with lists where A LoL seq. only generated 51 responses. In comparison, the test set only contained 24 responses with lists.
2. “... Is it possible to modify A-LoL such that it can continue to improve itself with online data generated by itself and labeled by the reward model?”
   Interesting question! In the future, we certainly want to investigate better exploration strategies than a random sampling of current RLHF implementations and eventually improve A-LoL further with high-quality on-policy generations. In this study, we wanted to benchmark and highlight the interesting properties of A-LoL in the offline setting.
3. “The single-action assumption might not always hold especially when the dialogue involves multi-step interaction with the users.”
   We agree with this issue and will include it in our limitations section in Appendix D.

Answers to questions:

1. “Why is this method called advantage leftover lunch RL?”
   A-LoL has the unique feature of using supervised finetuned LM’s value estimate to detect the negative advantage training instances. By exclusively training from positive advantage data, our method is robust to noisy instances (for instance, Reddit response generation from downvoted comments - Section 5). To capture this essence of selecting the leftover positive advantage data, we call our method Advantage Leftover Lunch.
2. “I'm curious if other contextual bandit algorithms (such as those from https://arxiv.org/abs/1802.04064) can work well too in the RLHF setting?”
   Thank you for this interesting suggestion. We will explore these methods in our future work.

We thank the reviewer for their feedback. We are encouraged that they find our discussion about online vs offline methods that use preference or rewards useful and our motivations and technical contributions clear.
Response to the weaknesses:

1. “why offline algorithms are more sample-efficient than online algorithms … online algorithms can work with small amounts of data and well-designed reward functions.”
   This is not always the case. Online RL is only sample efficient if exploration yields high-quality data, and the reward function is well-designed. Exploration with LMs is tricky due to exponentially large output space [2,3] and reward arrives at the end of the output [4]. Instead, offline RL operates on static training data. A-LoL can further filter low-quality instances using negative advantage. In the HHA task we allocate a similar number of gradient updates to all baselines and PPO shows slower growth (Fig 3) compared to offline RL methods (Fig 2).
2. “The paper relies on the assumption that each token is not an action but instead a sequence is an action, but it is unclear why this assumption matters”
   Most prominent implementations of PPO in RLHF use per-token as action and assign the full sequence reward to the last token [5,6,7]. This makes the value estimate of intermediate actions tricky and also affects the loss values at each token. Thus, concurrent works are investigating entire output as single-action in offline RL [1,8] for LMs. Using the single-action assumption, our method trains the reference LM value-estimate of the entire sequence and freeze it before training target policy, which allows for stable training (Fig 2).
3. “if you don't have good data coverage and a good initial policy, then A-LOL will fail”
   Thank you for highlighting this point. We agree that a bad initial policy or lack of good data coverage can be detrimental to the performance of A-LoL. We will clarify this in the limitations section (Appendix D).
4. “did not include PPO due to a seed collapsing, but there has been evidence in the literature that this does not happen[1].”
   We include the average PPO performance for the two out of three seeds (Table 2). We followed the experiment setup from the PRO baseline [10] which used a 1.4B parameter reward model [11] for training and a 6.9B parameter model for evaluation. The reward model used in the provided reference [1] is a 6B parameter and even the reference policy is different from what we used in our experiment, which could explain the difference in PPO’s performance.
   For hyperparameter tuning, we selected one seed and tested with PPO rollout batches in {16, 128} and learning rates in {2e-4, 2e-5, 5e-6, 5e-7} and found the best configuration be batch size=16 and lr=2e-5. Extending these hyperparameters to 3 seeds resulted in one of the seeds collapsing which highlights the instability in on-policy RLHF [1,2,3]. To verify the correctness of our PPO implementation, we also include the PPO checkpoint from an external source, but found our PPO implementation to work better.
5. “authors claim that their proposed approach is more data-efficient because they filtered out 33% … but the same procedure can be done for other techniques”
   The reward-based and preference-based offline RL baselines do not come with any principled method to filter data on their own. To filter data with rewards, one would need to know the reward landscape of each training dataset and also analyze the low reward instances to pick appropriate thresholds [9], which is out of the scope of this study. A-LoL is unique in its ability to automatically detect the subset of positive advantage in any training set using the sequence-level value estimate of the initial supervised LM. The amount of negative advantage varies for each task, initial LM, and data split, for example, 33% of the response in HHA task (Section 4), 48% of the downvoted, and 36% of upvoted reddit comments (Section 5) and 10%, 39% and 55% of different data splits of knowledge-grounded dialog (Appendix C.2).

Answers to the Questions:

1. “There seems to be a typo in equation (2) because in the $y \sim \pi_{\theta}$ in expectation.”
   The equation (2) does have $y \sim \pi_{\theta}$ but we rearrange the probabilities to convert it to $y \sim \pi_{ref}$ as follows:
   $\nabla_{\theta} J(\theta) = E_{x \sim d^{\pi_{\text{ref}}}} [\sum_{y \in \mathcal{Y}}R(x, y, \star) \nabla_{\theta}\pi_{\theta}(y|x)]$
   $= E_{x \sim d^{\pi_{\text{ref}}}}[\sum_{y \in \mathcal{Y}} \pi_{ref}(y|x)\frac{\pi_{\theta}(y|x)}{\pi_{ref}(y|x)} R(x, y, \star) \frac{\nabla_{\theta}\pi_{\theta}(y|x)}{\pi_{\theta}(y|x)}]$
   $= E_{x \sim d^{\pi_{\text{ref}}}, y \sim \pi_{\text{ref}}}[R(x, y, \star)\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}\nabla_{\theta}\ln \pi_{\theta}(y|x)]$