Pairwise Proximal Policy Optimization: Language Model Alignment with Comparative RL

We appreciate the reviewer's feedback and recognize the importance of improving the overall presentation of our paper.

…overall paper writing/presentation is not very good…

We will provide a clearer explanation of the reward equivalence definition and its significance. We will include additional derivations and discussions of our algorithm in the appendix.

More intuitive explanations on why define the reward equivalence are needed.

The definition of reward equivalence is motivated by the observation that different prompts can yield significantly varying reward values. Suppose we have two prompts, one being relatively easy and the other more challenging. For the easy prompt, a standard response is likely to receive a high reward. In contrast, even a more sophisticated response to the hard prompt may result in a lower reward.

Methods that rely solely on absolute rewards, such as PPO, tend to favor increasing the likelihood of the easy prompt response while decreasing the likelihood for the hard one. This can be detrimental to the model's performance.

P3O circumvents this issue by utilizing comparative signals. By focusing on the relative preferences between responses, P3O is not sensitive to the absolute reward values associated with different prompts.

We will provide additional discussions to further clarify the motivation behind the reward equivalence definition and the advantages it brings to P3O.

…experiments can be further improved…

The bottleneck of tuning larger models (7b+) is on the infrastructure side, where there’s some conflict between ZeRO3 and generation. For the baseline, we only choose DPO as we think it's simple and well adopted. We’ll compare with SLiC or RSO if time permits.

…connection between your theory and experiments are not very strong…

The reward shift invariant issue pointed out by our theoretical analysis significantly impacts the empirical performance of PPO. Our observations reveal that adding a constant to the entire reward signal can dramatically alter the model's behavior after tuning with PPO. If the reward is generally negative, the model tends to generate excessively long responses, while a positive reward conversely leads to significantly shorter outputs. In practice, precisely balancing such constant shifts is crucial for PPO. In contrast, P3O consistently produces identical results regardless of constant shifts in the reward, thereby eliminating a significant source of instability inherent to PPO.

We appreciate the reviewer's insightful comments.

Novelty/Forward-outlook…

In the final draft, we will discuss preference-based RL and learning to rank literature. We note that P3O shares similarities with DPO and LiPO in that they all utilize comparative feedback. However, a key distinction lies in the fact that P3O is a policy gradient algorithm, benefiting from its strict improvement property. This sets P3O apart from ranking loss-based methods like DPO.

Depth/Technical impact…

We will provide a comprehensive discussion of the scale invariance issue, including its relevance to the learning-to-rank literature [2].

By explicitly studying the scale invariance problem and its impact on training stability and sample efficiency, we bridge the gap between the alignment community and insights from other fields.

Experimental evaluation…

Our choice of baselines, and evaluation metrics are standard and widely accepted in the field, following the precedent works [RAFT APA DPO]. Evaluation metrics: We have included the GPT-4 side-by-side evaluation, which has been shown to correlate strongly with human judgments [Rafailov et al., 2023]. We acknowledge the importance of human evaluation and will consider it for future work if resources allow. Baselines: We have compared our method with two strong baselines, DPO and PPO, which are established algorithms in the field. Regarding ablations, we have provided an analysis of the effects of clipping and KL constraint in the appendix.

...not convinced about the experimental fairness…

We performed extensive hyperparameter tuning, and we apologize if these details were not clear:

Hyperparameter analysis: We conducted a comprehensive analysis of hyperparameters for PPO, including discount factor, GAE lambda, and clipping threshold. We found that a discount factor 1 performed best, and adjustments to GAE lambda and clipping within wide ranges did not significantly impact the training curve.

lr and batch size: We identified lr and batch size as the most critical factors affecting PPO's performance. We observed that a smaller batch size improved convergence speed, while larger batch sizes stabilized training but at a slower pace. We also considered GPU memory limitations and potential gradient accumulation bugs and set the gpu-mini-batch size accordingly. Seed selection: We repeated each experiment three times.

We believe that this additional information will address the reviewer's concerns.

Thank you for your feedback. We appreciate the opportunity to enhance the clarity of our work.

...whether P3O with the ability to solve this inconsistency issue…

We will incorporate a training curve to highlight the instability observed in PPO. Furthermore, we will present a comparative analysis of the reward / KL as a function of training steps for different methods in the appendix for better understanding.

...the proposed method seems to use 2x generated samples to acquire a more stable estimation of gradients at each step compared with PPO…

To ensure a fair comparison between P3O and PPO, we have adopted a setting that potentially disadvantages P3O. Specifically, we use half the batch size for P3O compared to PPO. This adjustment guarantees that each gradient update across both methods is based on the same number of model-generated responses. However, it is important to note that under this setup, PPO benefits from processing a larger number of prompts per update, a factor generally considered advantageous for learning.

Regarding hyperparameter tuning, we have conducted extensive experiments on various parameters for PPO, including Generalized Advantage Estimation lambda, discount factor, batch size, and learning rate. We have identified the optimal configuration for PPO and adapted it to be compatible with P3O without conducting additional hyperparameter searches.

...add human evaluation to make the experimental results more convincing.

We chose GPT-4 evaluation for two reasons: its efficiency in approximating human preference and its reproducibility. Studies such as [Rafailov et al., 2023] have shown that GPT-4 judgments correlate strongly with human evaluations, with inter-annotator agreement often similar or higher than human agreement. Additionally, [Starling 7b] demonstrated significant improvement in an aligned model using GPT-4 judgments and RLHF on the live human evaluation platform like Chatbot Arena.

We recognize that GPT-4 evaluation has advantages in terms of efficiency and reproducibility, as labeller biases are difficult to avoid and document. The usage of GPT4 will allow us to directly compare the results obtained from GPT-4 evaluation and human evaluation, providing a more robust assessment of our proposed method's performance.

We will also provide more responses in the appendix. This will offer additional insights for curious readers and further validate the effectiveness of our proposed approach.

Thank you for your feedback. We appreciate your recognition of the novelty of our paper. We will endeavor to address your concerns in our revised draft.

They mainly use some relatively old and weak pretrained models…

The decision to employ these specific models was driven by considerations of flexibility and consistency. The Pythia series offers a wide range of model sizes, from 70M to 12B parameters. This flexibility in sizes is particularly valuable in research settings, where we’re able to initiate our experiments with smaller, more manageable 1B models on V100s and then scale up to larger 6B models on A100s. Furthermore, employing the pythia models ensures a consistent framework for evaluation across different model sizes, with minimal code changing.

...experiments can be strengthened by incorporating some reliable metrics…

We have incorporated GPT-4 evaluation into our methodology, primarily due to its cost-effectiveness and the strong correlation with human preferences they have demonstrated. For example, (Rafailov et al., 2023) provides compelling evidence of GPT-4 judgments' alignment with human evaluations, noting that agreement between GPT-4 and human annotators is often on par with or exceeds that among humans themselves. Furthermore, the study by (Starling-7b) highlights the effectiveness of using GPT-4 feedback for Reinforcement Learning from Human Feedback (RLHF), where models trained on GPT-4 judgments showed significant improvements in performance on the live human evaluation platform, Chatbot Arena. As a complementary, we will add more interesting responses and compare different methods in the appendix, which may be interesting for curious reviewers.

...importance sampling trick…

Importance sampling is a crucial technique in reinforcement learning, particularly when dealing with the challenge of updating a policy 𝜋 based on data collected from an older (hence different) policy 𝜋_old. This is common in RL as we usually collect a batch of samples and perform multiple gradient updates. The term "importance sampling" refers to this process of reweighting the contributions of each sample to reduce bias. Specifically, the importance sampling ratio π(a|s) / π_old(a|s) adjusts the influence of each experience based on how much more or less likely it is under the current policy compared to the old policy. This makes the estimator unbiased. In practice, we find the correction to be helpful when combined with clipping.