Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning

We would like to thank the reviewer for taking the time to read our work, and were pleased to hear that it “adeptly combines theoretical analysis with empirical findings” and is “clear and easily understandable”.

The reviewer states a single weakness of our work – that we do not compare to online algorithms like PPO with a learned reward model. We have run this baseline, but would like to additionally discuss our choice for not including it originally below.

1. In sequential settings, online methods like PPO require online access to the MDP, which can be impossible. Consider a multi-step version of the dialogue domain brought up and cited by the reviewer. At a minimum, sequential dialog data would be a sequence of at least two steps $s_1, a_1, s_2, a_2$ where $s$’s are the prompts and $a$’s are the LLMs completions. While the first prompt $s_1$ could be sampled from a fixed dataset, and the first action $a_1$ could be sampled from the LLM, the second prompt $s_2$ would need to be provided by a user as it depends on $a_1$! As PPO relies on the on-policy policy gradient, we would need humans to interact with the language model batch_size number of times for every gradient step in the sequential setting. Thus, using online methods like PPO in sequential domains can be infeasible. We would like to point out that the paper and RLHF domain the reviewer mentions is a contextual bandits setting (one step) and not a true sequential problem. In Appendix A we show that in this bandits setting CPL reduces to DPO, which can also outperform PPO on LLM training [1].

2. Online methods like PPO assume the cost of data collection is essentially zero. PPO requires rolling out the policy to estimate the gradient. In many applications, like robotics, this is very expensive and time consuming, making learning from offline data a way more attractive alternative. In fact, in that community, the offline learning approach has largely been adopted.

3. Many contemporary works on RLHF for robotics or control use only offline data [2,3,4].

Finally, we ran a PPO baseline with a KL-penalty on the reward as commonly used in RLHF [5]. Before discussing results we would like to note that this comparison is unfair to CPL as:

1. The PPO baseline has access to online samples from the MDP.

2. The PPO baseline uses 25x the number of state-action pairs as CPL for 2.5K Dense and 4x the amount for 20K sparse. This is because PPO uses 3.84 million additional online transitions plus the original 160,000 for CPL Dense and 1.28 million for CPL Sparse).

3. PPO required far more hyper-parameter tuning. We had to bound the learned rewards, bound the policy standard deviation, and aggressively tune the KL-divergence.

Despite all of this, we found PPO to be extremely unstable and often underperformed CPL in 4 of the environments. Note that we verified the performance of our PPO implementation on standard Benchmarks (Hopper-v2, exceeds performance here). Our full results are in Table 1 for two different values of the reward KL-penalty, 2 and 5, and learning curves for PPO are in Appendix C. We have copied them here for convenience:

2.5K Dense Results

20K Sparse Results

PPO faces a number of challenges in the sequential domain: the variance of policy gradients and value estimation increases with horizon, and instability of the KL-divergence penalty for continuous policies. Again, in 4/6 environments CPL attains better performance than PPO using 1/4 of the data.

As we have addressed the reviewers' only stated concern with the draft through this experiment, we would kindly request that the reviewer reconsider their score.

[1] Rafailov, Sharma, Mitchell, Direct Preference Optimization. NeurIPS 2023

[2] Kim et al. Preference Transformer. ICLR 2023.

Thanks for your response!

I am very grateful to the authors for conducting experiments on online reinforcement learning.

I requested such experiments because the authors extensively emphasize the advantages of CPL for RLHF. Without clarifying that the entire setting of the article is offline, I believe an online baseline is necessary.

For the experiments of PPO, it seems PPO-KL2 outperforms PPO-KL5 in almost all experiments. I understand that the authors may have time constraints to tune the hyper-parameters, and I believe that online algorithms also have great potential.

Overall, the author did address my concern, and I would like to increase my score.

We were excited to hear that the reviewer found our work to be “well-motivated”, “clear”, and that it “has the potential to make a significant impact in the community”. The reviewer's comments were largely based on a set of additional questions we seek to answer here.

The Regret Preference Model

The regret preference model states that human judgements are better captured by an optimal advantage function, and does not make any claims about what underlying reward functions humans use. Going back to the example in Section 2, we did not mean to claim that the human would intend to use a sparse reward function – a user providing preferences preferring movements towards a goal could have a number of different underlying reward functions, many of which could yield the same optimal policy and optimal advantage function.

Though the focus of our paper is not the validity of the regret preference model, for which we would refer the reader to [1], we do provide a number of additional insights on the regret preference model.

1. Theoretical: In Appendix A, we prove that the optimal advantage is just a shaped version of the reward function that results in the same optimal policy. The reviewer suggests that humans use dense rewards to make decisions, and Corollary 1 shows exactly that the advantage is a dense reward function.

2. Empirical: In the new Section 4.4, we show that CPL works on real human data, indicating that the regret-model holds on real human data.

Again we would encourage the reader to examine [1], as it contains a number of additional examples (like stochastic environments) and experimental results which support the validity of the regret-preference model.

Does CPL require more human preference data in comparison to two-phase PBRL

To demonstrate that CPL is effective with limited real-human preference data we perform additional evaluations on the PBRL benchmarks from [2]. These benchmarks use only 100 or 500 real human preferences for expert or replay datasets from D4RL respectively.

We showed in Section 4.3 that CPL performs better with dense preference data. To overcome this, we use a logistic regression model to predict which of two segments a user prefers from the preference data, and use it to relabel a larger offline RL dataset. Critically, this logistic regression model is not a reward or advantage model, as it predicts a single value for the entire segment.

We find that CPL performs similarly to baselines on 2/4 datasets, outperforms in 1, and performs worse in 1. Yet, CPL remains far simpler.

Pretraining

We have included results on Metaworld in state without any pre-training, and find that CPL performs similarly – no pretraining performs better in some cases and pretraining performs better in others. This ablation has been included in Appendix C and is copied below as well.

This ablation shows that the use of the conservative regularizer in CPL can help it learn effectively even without pre-training. Note that our results on D4RL (Table 3) also do not use any pre-training.

We did not use pre training for P-IQL for two reasons:

1. P-IQL uses a weighted BC objective ($\min \mathbb{E} [ e^{Q-V} \log \pi]$ and thus already has a loss function very similar to BC.
2. In IQL value learning is completely detached from policy learning, and thus pretraining $\pi$ does not improve estimates of $Q$ and $V$.

Does CPL lose the generalization power of RL

We are not 100% certain what the reviewer is referring to by the “generalization power of RL”.

To our best understanding, there are two types of generalization potentially at play in DeepRL: 1) the generalization of neural networks and 2) RLs ability to use the ground-truth reward function to extrapolate in new parts of the state space. For 1), CPL still has the generalization power of neural networks. For 2), we would like to point out that this does not necessarily apply in RLHF, as the accuracy of the reward model is dependent on the preference data. Thus, RLHF methods that use an explicit reward model at the end of the day, still rely on the generalization of supervised reward learning. CPL is just cutting out the middleman.

This approach has already shown to be effective for LLMs [3]. By removing the need for extra components, we believe CPL will help scale to larger datasets/models where better generalization is typically found.

continued...

[1] Knox et al. https://arxiv.org/abs/2206.02231

We would like to thank the reviewer for their time in reviewing our work. We are happy to hear that our algorithm was “novel and elegant”, and that it’s “motivation… is clear”.

As the reviewer has pointed out no weaknesses, we will just mention the following revisions to the draft:

1. Additional PPO baseline. We have modified Table 1 with an additional PPO baseline. Note that as PPO is an online RL method, it requires 4x the amount of state-action data as CPL plus online interactions with the MDP. Despite that, it still underperforms in many settings and exhibits low variance.

2. Additional Benchmark with Real Human Feedback. We have added additional experiments on the D4RL benchmark using only 100 to 500 real-human queries.

3. Additional Theoretical revisions. We have heavily modified the appendix and Section 3.3 including a new proposition on CPLs regularization with finite data.

4. Additional ablations on pretraining in Appendix C.

Thank you for taking the time to review our work. We were happy to hear that the reviewer found our work to be “generic”, “scalable”, and overcoming challenges in reward optimization. The reviewer had two primary questions about our work, first concerning our baselines and second concerning our benchmarks.

Baselines

The reviewer asked us to compare CPL to methods that learn a reward model.

1. We would graciously like to point out that P-IQL does indeed learn a reward model first, which could then be applied with any RL Algorithm. We chose to use IQL since it is generally a very high-performing offline RL method [1] and has been used as a standard in many prior RLHF for control works [2, 3].

2. We have additionally compared CPL to PPO with a learned reward function based on comments from Reviewer WpDS. We would like to point out that this comparison is unfair to CPL, as PPO requires additional online data to estimate the policy gradient. While in bandits settings (like LLMs) this can be done without access to the MDP, in sequential settings (like control, or 2+ steps of dialogue) we need to interact with the MDP to get full trajectories, which can be expensive (for ex running a robot in the real world). Despite the fact that PPO uses online data and 4x the number of state-action pairs, it is highly unstable in comparison to CPL and often underperforms it by a large margin. We have included full results in Table 1.

3. Our results are consistent with other recent works [3, 4] which have also shown that learning an explicit reward model is unnecessary to achieve good performance.

Benchmarks

1. The reviewer mentioned that RLHF domains with LLMs would be interesting to explore. We whole-heartedly agree that RLHF with LLMs is a great possible application of CPL. Our work, however is about settings with sequential data in particular, which in dialogue would constitute multiple turns of interaction with a human user. To our knowledge there were no such publicly available RLHF datasets with 2+ turns of dialogue at the time of submission. We believe such an investigation is a large undertaking, enough to warrant its own publication.

2. We would kindly like to push back against the notion that robotics domains are less interesting. We chose to evaluate in robotics domains precisely because they capture the sequential nature of CPL, and show that our method applies to general MDPs, instead of the simpler contextual bandit setting used in standard RLHF with LLMs.

3. We have included additional benchmark results on more robotics domains using only 100 or 500 queries of real-human feedback [2]. We hope this serves as another datapoint for the applicability of both CPL and the regret-preference model. Please see Table 3 in the paper for results, or the general response for results. We find that CPL is able to perform similarly to other methods while being vastly simpler as it requires no Q-learning. CPL also exhibits lower variance.

We hope the reviewer can consider these additional empirical results, as well as our revised theoretical contributions when assessing our work.

[1] Kosterikov et al. Implicit Q-Learning ICLR 2022.

[2] Kim et al. Preference Transformer. ICLR 2023.

[3] Hejna et al. Inverse Preference Learning. NeurIPS 2023

[4] Rafailov, Sharma, Mitchell, Direct Preference Optimization. NeurIPS 2023

We would like to thank the authors of OPPO for finding our work interesting, and note that we have already included a citation to their paper.

First, we are open to expanding the discussion of OPPO in our work. OPPO's objective works in a rather different paradigm than CPL's as it uses HIM rather than a theoretical derivation from RL principles. Currently, the popular paradigm for RLHF -- reward learning followed by (offline) RL -- more closely follows the latter and thus we focused our experimental efforts on comparisons with these methods which CPL seeks to replace. OPPO is a bit more complicated -- it first learns a preference model, estimates an optimal $z^*$, then learns a conditioned policy.

Second, we are open to including OPPO as baseline in Table 3, however we would request that the authors of OPPO give us sufficient time to ensure such a comparison is fair. OPPO uses a BERT-like preference model, a non-markovian transformer policy, and varies hyper-parameters across environments (Table 7 & Table 8 in Appendix). CPL and our baselines on the other hand use only simple MLP/CNN markovian policies and the same hyper parameters across domains. We are going to start work on an OPPO baseline using your offical codebase, but would like to first normalize all decision decisions and the evaluation procedure to ensure a fair comparison.

Thanks, CPL Authors

Dear Authors,

I appreciate the author's attention to the public comment and their prompt response. The reasons mentioned by the author seem reasonable to me.

It is encouraging to see further work being done in this direction, with the proposal and discussion of new methods. However, due to the constant emergence of new methods, I believe readers would prefer a comprehensive and objective understanding of the development path of relevant methods, as well as the specific similarities and differences of new methods. This would allow for a more essential understanding of the contribution of the paper.