Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

We would like to thank you for your constructive comments! In the following, your comments are first started and then followed by our point-by-point responses.

W1: The adaptive scaling is defined on a per-instance basis, which necessitates significant compute costs and hinders real-world mini-batch learning.

Response to this concern is included in point 3 in the global rebuttal. The additional cost of our proposed method is negligible and it does not hinder mini-batch learning.

W2: Some of the derivation may be simplified, both to make the paper more succinct and leave room for algorithm box as well as other technical details.

Thank you for the suggestion. We will simplify some duplicate derivation in section 3 and include the algorithm box and technical details, such as a discussion on additional computational costs, in the main page.

W3: could you theoretically justify the incorporation of adaptive preference scaling into DPO. e.g., in the framework of KL-control as in the original DPO paper?

DPO start from the policy optimization objective: $\\max\_\\pi \\mathbb{E}\_{s,a\\sim\\pi (a|s)}[r(s,a)] - \\beta D\_{\\mathrm{KL}}[\\pi(a|s), \\pi\_{\\mathrm{ref}(a|s)}]$ and make use of the optimality condition given that the horizon is 1: $\pi_r(a|s) = \frac{1}{Z(s)} \pi_{\mathrm{ref}}(a|s) \exp \left( \frac{1}{\beta}r(s,a) \right).$ Then they reparameterizes the reward function as $r(s,a)=\beta\log(\pi_r(a|s)/\pi_{\mathrm{ref}}(a|s))+\beta\log Z(s)$ and employs the standard cross-entropy loss for reward learning. Our proposed Ada-DPO retains the same reparametrization and only modifies the reward learning loss to Equation (6). The derivation and analysis for this change in the reward loss are included in Sections 3.2 and 3.3.

We assume the reviewer is suggesting moving the adaptive scaling factors to the KL-constrained policy optimization objective. Unfortunately, we found this difficult due to the regularization on scaling factor. However, at a high level, as mentioned in Line 222, we can merge the regularization parameter $\beta$, which controls the KL term, with the scaling factors in the final objective. Thus, our method can be viewed as adapting a different KL-control term for each preference pair. For strong preference data, we learn a large scaling factor, corresponding to a smaller KL-control term, allowing the model to deviate more from the reference policy. In contrast, a larger KL-control term is used for ambiguous preference data.

Q1: "Prior work on this topic is very limited.": There are actually a vast pool of literature concerning loss functions for (RLHF's) reward modeling, a quick pointer is [1] for language tasks and [2] for image generation, as well as the literature review in them. I encourage the authors to add proper discussions and citations to make the claims more precise.

Thank you for pointing out the related works. We will include discussions and citations of these references in our next version. After carefully reviewing the mentioned works, we find that both focus on changing the reward function, but the used loss is still based on cross-entropy. This makes them complementary to our work, as we are improving the loss function. Specifically:

[1] introduces a token-level preference reward learning loss instead of the standard sequence-level objective, addressing the granularity mismatch between preferences and LM training losses. Their approach employs a more fine-grained reward function. However, the cross-entropy loss is still employed when there are two responses, which means their method can potentially be combined with ours.

[2] focuses on the preference alignment of text-to-image diffusion models and proposes temporal discounting DPO-style objectives that consider the sequential nature of the generation process. However, their method is specific to diffusion models and cannot be applied to general RLHF as ours can. Additionally, the used loss is still based on cross-entropy.

In summary, we consider these two works to be orthogonal and complementary to our work.

Q2: Could you elaborate this sentence more: "For some applications, it may lead to..."

Response to this concern is included in point 2 in the global rebuttal.

Q3: In Eq. (3), why do we need to have $\mathrm{KL}(p,1/2)$ in both the objective function and constraint?

Including the KL term in both the objective and the constraints is a common practice in constrained distributionally robust optimization (DRO) problems, as shown in Eq. (1) in [3]. Without the KL constraints, the loss becomes the cross entropy loss with temperature $\tau_0$, making the loss not be able to adapt to each pair of samples. Without the KL term in objective, the objective will not be smooth and hard to optimize.

L1: A discussion on limitations and/or potential negative societal impact seems missing.

Response to this concern is included in point 4 in the global rebuttal.

References

[1] Yang, Shentao, et al. 'Preference-grounded token-level guidance for language model fine-tuning.' Advances in Neural Information Processing Systems 36 (2023).

[2] Yang, Shentao, Tianqi Chen, and Mingyuan Zhou. 'A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.' Forty-first International Conference on Machine Learning. 2024.

[3] Qi Qi, Jiameng Lyu, Kung sik Chan, Er Wei Bai, Tianbao Yang. Stochastic Constrained DRO with a Complexity Independent of Sample Size. Transactions on Machine Learning Research, 2023.

Thank you for your thoughtful comments! We sincerely appreciate your time in reading the paper, and our point-to-point responses to your comments are given below.

W1: The increase in flexibility of the proposed reward function is not verified. The authors claim that one of the limitations of the BT model is the logit of the preference distribution scales linearly with the reward difference. However, according to Proposition 3.1, the limitation still exists. There is no empirical result that supports the claim.

Thank you for the suggestion. We provide additional clarification to better demonstrate the increased flexibility of the proposed method:

• We would like to clarify that the proposed method no longer maintains a restrictive linear relationship between the logit of the preference distribution and the reward difference. In Proposition 3.1, we show that the relationship between the optimal reward difference and the logit of the true preference distribution varies for strong preference pairs and ambiguous pairs, resulting in a more flexible non-linear relationship. Additionally, during training, the learned logit is often not optimal, meaning the scaling factor is not strictly on the bounds. This is illustrated in Figures 5a and 6, where the scaling factor in the middle can create a more complex correspondence. In contrast, the BT model always maintains the same linear relationship between the learned logit and the learned reward difference during training, restricting flexibility.

• Directly measuring the flexibility of the reward function empirically can be challenging since it's difficult to quantify. To better support our claim, we provide some supporting empirical evidence in the paper. For instance:

  • Figure 5(c) shows that Ada-Pref learns smaller reward differences for pairs with ambiguous preferences and larger reward differences for pairs with strong preferences, indicating a wider range of learned reward differences.
  • Figure 7 provides examples showing that Ada-DPO learns larger reward differences than DPO for clear pairs and smaller reward differences than DPO for ambiguous pairs. We also include more examples in the response to the second reviewer.

W2: Missing the ablation study of the regularization term in the proposed loss function.

The ablation study of the regularization term $\rho$ in the proposed loss function has been provided in Appendix D.3 due to space constraints. If we remove the regularization term, the proposed loss collapses to the standard DPO loss with a modified $\beta' = \beta \cdot \tau_0$, which performs significantly worse than the reported DPO results. Consequently, we did not include it in the main text. For reference, removing the regularization term achieves a win rate of 24.56 on the summarization task.

W3: Missing the RLHF baseline in the natural language generation task.

Thank you for pointing out the absence of an RLHF baseline in the natural language generation task. We understand the importance of experimenting with PPO instead of DPO to fully comprehend the impact of our method on RM's signal. Due to resource constraints, we prioritized experiments requiring PPO for robotic control tasks. Conducting PPO experiments for natural language generation tasks presents significant computational challenges, including the need to store and manage both reward and critique models (which are LLMs). PPO also involves more training steps and numerous hyperparameter tunings, making it difficult to implement within a resource-limited environment. Therefore, we opted for DPO as an alternative in our submission.

Due to the limited resources and the need to run other experiments, we are currently unable to provide PPO results for natural language generation tasks. Nonetheless, we will try our best to update the results before the discussion deadline.

W4: It seems that Ada-Pref's greater resistance to performance degradation is not true in the Hopper task?

This is true in the Hopper task as well. Comparing Tables 1 and 2, we can see that Ada-Pref drops by 32.9% in the Hopper task, while Pref drops by 43.7%.

Q1-A: I do not believe the proposed loss function is convex, can you provide the proof of Remark 3.1?

The proposed loss function is strictly convex with respect to each $\tau_i$ when $r(z_w) \neq r(z_l)$. For simplicity, we omit the index $i$ from $\tau_i$ and set $C = r(z_w) - r(z_l)$. Then the instance-level loss function is written as: $f(C, \tau) = -\tau \log \sigma(C/\tau) + \rho\tau.$ Since $\frac{\partial f}{\partial \tau} = \frac{C^2}{\tau^3} \sigma'(C/\tau) > 0$ for all $\tau \in [\tau_0, \tau_{\mathrm{max}}]$, $f$ is strictly convex with respect to $\tau$.

Q1-B: Why the reward models trained with the proposed method can better guide policy model selection considering the reward model performance is not improved?

We cannot directly evaluate the performance of the reward model solely based on preference prediction accuracy. This is because preference prediction accuracy does not fully capture the effectiveness of the reward model in the context of policy optimization. It only reflect the sign of reward difference but not the scale. Our reward function is designed to be more flexible, providing distinct rewards for clear pairs and comparable rewards for ambiguous pairs. This flexibility allows the reward function to generate a wider range of rewards, which is crucial for effective downstream policy optimization.

Q2: How do the authors obtain the true reward difference in Section 4.3?

PyBullet environments provide the ground truth reward for each pair of state and action, which are then used to obtain the true reward difference in Section 4.3.

Q3: The computation overhead introduced by learning the scaling parameter is not discussed. Response to this concern is included in point 3 in the global rebuttal.

I appreciate the authors' detailed reply. They have addressed most of my concerns.

However, there is no evidence (even without a toy example) demonstrating that the BT model would fail due to its linear relationship. This makes it unclear for researchers to determine when the proposed method, which seems hard to re-implement, should be used.

Therefore, I would keep my previous evaluation.

We would like to thank you for appreciating the feature of the proposed method and are grateful for the constructive comments! In the following, your comments are first stated and then followed by our point-by-point responses.

W1: It seems that only a single run was conducted for the experiments in the NLP tasks. More repeated runs would be more convincing.

Thank you for pointing that out. We have included additional results and analysis for more runs in point 1 of the global rebuttal. We can observe from the table that our method consistently achieves stable and significant improvements over the baselines in both tasks. Given the large margin of improvement, we believe p-value is not necessary.

Q1: I would like to see more examples like Figure 7, comparing the learned reward differences with different scaling factors.

We provide three examples with different learned scaling factors in the summarization task below. We do not include the complete prompt here for better readability and space limit.

---

Example 1:

• Original text: I need some advice. I've been talking with this girl for about 2 weeks now. We went out last weekend and it went great. We were working on setting up another date and she told me that she was concerned about distance ...

• Chosen summary: agreed to meetup for coffee but haven't heard from her since tuesday night. Want to know what i can do to make it happen again.

• Rejected summary: agreed to meetup but haven't heard from since. Tried texting a couple times just trying to understand whats going on.

This sample pair gets $\tau=0.325$ and the learned reward difference is 0.3071(Ada-DPO) vs 0.6419 (DPO). We can observe that the two responses in this example is similar, with the chosen summary being slightly more specific. Our method learn a small scaling factor and smaller reward difference since the gap between two responses is not that significant.

---

Example 2:

• Original text: I'm not sure if I can even do anything, and if the person in question wasn't an ...

• Chosen summary: store franchise owner is probably stealing from their store by under ordering groceries/charging less than customer pays. possibly unethical behavior by owner of store? not sure how to proceed/act. help pls reddit.

• Rejected summary: store franchise owner probably is hiding their grocery bill from the rest of the store staff and is getting some kind of unethical benefit out of it, not sure if I can do anything. Advice?

This sample pair gets $\tau=0.93$ and the learned reward difference is 0.8786(Ada-DPO) vs 1.066 (DPO). This example present a reasonable difference, with Chosen being more correct and better summarize the text. Our method and the DPO baseline get a similar reward difference and the scaling factor is around 1.

---

Example 3:

• Original text: So my dream is do stand up comedy, improv comedy, writing and/or sketch comedy full time...

• Chosen summary: I want to pursue stand up comedy full time but I am afraid of losing my brothers rent money and my family. Do I follow my dreams or play it safe? Any advice/criticism is greatly appreciated!

• Rejected summary: Follow your dreams or play it safe?

This sample pair gets $\tau=3.00$ and the learned reward difference is 2.832(Ada-DPO) vs 1.467 (DPO). Our method learn a large scaling factor and a larger reward difference for the last sample. As we can see from the two responses, the preference is obvious, with the rejected summary do not contain important information.

---

Q2: More discussion on the limitations of the proposed method and future directions can be added.

Response to this concern is included in point 4 of the global rebuttal.

We are grateful for the valuable feedback that you have provided! Please see our response per each of your concerns below:

W1: Weak experimental ablation. Why is the performance in summarization better in Figure 4 is not extremely clear?

The response to weak experimental ablation is included in the global rebuttal. In Figure 4, we select the best model based on the preference prediction accuracy and the improvement indicates that our method better aligns the learned reward function and policy optimization, while DPO baseline results in a model with high preference accuracy but low win rate. The gap may be due to that our Ada-DPO is more robust to overfitting, and that the learned reward is more flexible and better guide the policy optimization.

W2: The experimental evaluations lack important benchmarks and comparisons. For the Robotics environment, it's crucial to compare with Pebble, SURF, and PARL to understand the true efficacy of the proposed approach.

The proposed method is orthogonal to PEBBLE [1], SURF [2], or PARL [3] due to the distinct focus of our work. Specifically:

• PEBBLE aims to enhance both sample and feedback efficiency by integrating unsupervised pre-training with off-policy RL.
• SURF focuses on improving feedback efficiency by utilizing unlabeled data for reward learning.
• PARL tackles the issue of the alignment objective's dependence on data generated by the optimal policy by introducing a bilevel formulation.

In contrast, our method focuses on enhancing reward model training by considering the varying preference strength between each sample pair. This approach is distinct from the goals of improving feedback efficiency or resolving the entanglement between the alignment objective and the trajectory collecting policy.

Our proposed loss function can be readily adapted to enhance the performance of these methods (PEBBLE, SURF, and PARL), serving as a complementary approach rather than a direct comparison. As discussed in Remark 3.2 of the paper, Eq. (4) in PEBBLE [1], Eq. (3) in SURF [2], and Eq. (5) in PARL [3] show that these frameworks still optimize the standard cross-entropy loss for reward learning. Our proposed adaptive loss function can replace this standard cross-entropy loss to potentially enhance their performance.

Q1: Can the authors provide more concrete mathematical justifications regarding the issue with linear dependence with the logit of the preference distribution? Specifically, connecting with the relation to noisy preferences?

We want to clarify that the issue of the linear dependence of the BT model is not directly related to noisy preferences. Rather, it limits the flexibility of the learned reward difference, preventing the reward function from providing a wider range of rewards in downstream policy optimization. Our proposed method incorporates scaling factors to achieve a non-linear relationship, allowing the learned reward difference to be more flexible.

We provide a mathematical analysis in Section 3, demonstrating that our method increases the complexity of the reward model compared to the baseline. Specifically, we show that our method learns a smaller reward difference for ambiguous preference data and a larger reward difference for strong preference data. The experimental results further prove that this improved flexibility translates to better performance. We believe this sufficiently demonstrates the linear dependence is not flexible enough and hence can be suboptimal.

Q2: Included in global rebuttal.

Q3: Can the authors pls justify the statement and discuss why the proposed method is orthogonal to DRO?

While our proposed method borrows the technical concept from DRO, the aims are fundamentally different. DRO focuses on improving robustness against data distribution shifts between training and test sets, which does not necessarily translate to increased flexibility of the reward model that we want to address. In contrast, our method aims to improve flexibility by incorporating scaling parameters to help reward learning, but not targeting robustness against data distribution or noisy preferences. This distinct focus is why we consider our method orthogonal to DRO. More technical differences are discussed in Line 108.

Q4: Can the authors provide a more concrete definition of the best win rate and best prediction accuracy?

We use two different criteria to select the model among all those trained with different hyperparameter configurations:

• Best Win Rate: We identify the best policy based on its end performance, specifically the policy with the highest win rate, and then report the win rate and the preference prediction accuracy of the corresponding reward function (Figure 3).
• Best Prediction Accuracy: We identify the best policy as the one whose corresponding reward function achieves the highest preference prediction accuracy (Figure 4). We present this results because win rate evaluation involves costly proprietary models and tuning based on the preference accuracy could be much more efficient in practice.

Q5: It will be helpful if the authors can provide more details on how the trajectories are constructed for the robotics experiments.

Thank you for pointing that out. Due to the wide variety of experiments and the numerous details about the experimental settings, we were unable to include all the detailed explanations in the main text. However, a brief explanation of how the trajectories are constructed for the robotics experiments is provided in Appendix C.1.

Reference

[1] Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.

[2] SURF: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning.

[3] PARL: A unified framework for policy alignment in reinforcement learning