Robust Reinforcement Learning from Corrupted Human Feedback

We would like to thank you for your constructive comments! In the following, your comments are first started and then followed by our point-by-point responses.

W1, Q3, L1: The paper lacks an experiment for formal RLHF, such as Proximal Policy Optimization (PPO), in the context of text generation. Including such an experiment could provide a more comprehensive evaluation of the proposed method.

We agree that including experiments with PPO on NLP tasks can help fully comprehending the impact of our method. Please refer to point 4 in the global rebuttal.

W2: The use of $\delta$ as a regularization method may appear somewhat trivial for this approach. Conducting additional experiments on HH-RLHF or Ultrafeedback to provide further evidence and support for the effectiveness of this method would be beneficial.

Thanks for pointing this out! We conduct experiment on the Ultrafeedback dataset to further demonstrate the effectiveness of the proposed method. Results and details are included in point 1 in the global rebuttal.

Q1: It is unclear how far the Perturb Percentages can be extended before the model completely collapses. Further investigation and experimentation are needed to determine the limits and thresholds that lead to model failure.

Response is included in point 3 in the global rebuttal.

Q2: The difficulty level of optimizing $\delta$ is not specified. It would be helpful to elaborate on the challenges associated with optimizing this parameter and any potential limitations encountered during the optimization process.

We found that optimizing $\delta_i$ is not difficult, as $\delta_i$ has a closed-form solution in Eq. (3.4) for each iteration. Therefore, we can efficiently find the optimal $\delta_i$ without needing any optimization algorithms such as the proximal gradient descent for solving Eq. (3.3).

W1: Error bars are expected in Table 1.

Please refer to point 2 in the global rebuttal.

W2: The Anthropic Helpful and Harmless dialogue preferences dataset has a high disagreement rate, so that flipping a random portion of the training labels may not increase noise as expected, which could probably cause the unexpected results. The authors should probably filter the dataset or try a cleaner dataset to show the results on perturbation.

Since accurately filtering the dataset is challenging and there is no consensus on the best approach, we conducted the experiment on a cleaner, more modern dataset, Ultrafeedback. More details and analysis are included in Point 1 of the global rebuttal.

Q1: What’s the relationship and difference between Win Rate and Winning Score in your Table 1?

Win rate is defined as \dfrac{\text{\\# Win}}{\text{\\# Total comparisons}} and Winning score is defined as \dfrac{\text{\\# Win - \\# Lose}}{\text{\\# Total comparisons}} + 1. This means Winning Score additionally considers the "tie" cases compared to Win Rate. For example, two models with the same Win Rate can have different Winning Scores. The model with more tie cases would have a higher Winning Score. Therefore, we can view Win Rate as the primary criterion for evaluating models, and Winning Score as a secondary criterion that also accounts for tie cases.

Q2: Could you try to draw a fair comparison between Data Filtering and $R^3M$-DPO? Although your Data Filtering baseline doubles the training cost, it still indicates that filtering the dataset could be as effective as modeling the outliers. One thing you could provide is the performance of $R^3M$-DPO over the filtered dataset.

We believe that directly comparing Data Filtering with $R^3M$-DPO is fair and valid for noisy preference datasets. This is because our method improves the performance of standard DPO on noisy datasets, rather than on clean ones. With a noisy preference dataset, data filtering involves filtering the data first and then applying standard DPO to the filtered data. In contrast, our method does not require data filtering, as optimizing $\delta$ inherently performs a similar function.

L1: The paper does not discuss the limitation of the work. Please discuss the potential issue of applying your approach to real-world RLHF tasks.

Thank you for the suggestion! We provide our discussion as follows and will include it in our next version: Our approach does not introduce significant limitations compared to previous work. However, it does introduce an additional parameter, $\lambda$, in the regularization term, which may increase the tuning efforts in practical use. Developing more adaptive methods or conducting analysis on selecting this hyperparameter could be considered as future work.

Thank you for your valuable feedback! We provide a detailed response to your questions as follows:

Q1: In practical dataset, how do we justify the correctness of sparse perturbation assumption rather than stochastic noise in the original Bradley-Terry model?

As discussed in Section 6 and Remark 3.1, the sparse perturbation assumption in the preference data is a more challenging setting compared to the stochastic noise assumption. Therefore, our method can also improve robustness against other types of noise. Our experimental results illustrate that our method performs better across a wide range of perturbation types, including stochastic noise. While it is challenging to accurately verify assumptions in practical datasets, our method demonstrates better results due to its robustness.

Q2: What is the criterion for tuning the parameter $\lambda$ in the regularized loss?

We generally tuned $\lambda$ by performing a grid search within the range (0, 1). For the tuning metric, we used different criteria depending on the task: for the robotic control task, we optimized $\lambda$ based on the true episode return, while for the natural language generation task, we optimized $\lambda$ based on the win rate against a supervised fine-tuned model.

Q3: The TL;DR and HH dataset might not lead to significant improvement over preferences due to both responses being too much worse than current llama2 generation. Is it possible to run on modern preference dataset like HelpSteer2, Nectar or UltraFeedback, and report the Arena-Hard score as the evaluation?

Response included in point 1 in the global rebuttal.

Thank you for your comprehensive review and valuable feedback. We provide a detailed response to your comments as follows:

W1: The value of the $\delta$ offset could also be interpreted as by how much is the positive completion preferred over the rejected. This offset may be orthogonal to label-noise. Prior to this work, "Direct Preference Optimization with an Offset" explores DPO with an offset to model the difference in degree of preference (ODPO).

Thank you for your discussion and suggestions regarding related literature. We would like to clarify that our parameter, $\delta$, is fundamentally different from the margin parameter, $\Delta$, used in ODPO [1]. Specifically:

In our method, $\delta$ is jointly optimized with the reward model. By doing so, larger $\delta$ values are learned for corrupted preference samples to achieve smaller reward differences, compensating for the perturbations. In contrast, $\Delta$ in ODPO is not learnable but is a prefixed value proportional to the score difference between winning and losing responses. Optimizing the ODPO loss with larger $\Delta$ (i.e., pairs with strong preference strength) results in a larger learned difference, which is opposite to the effect of our $\delta$. When corrupted preferences (where the ranking of scores is flipped) are present, ODPO would exacerbate the label noise. Additionally, our $\delta$ parameter is sparse due to the use of an $\ell_1$ regularizer.

We further illustrate the relationship between $\delta$ and the learned reward difference in the Ant task by Figure 1 in the attached file in global rebuttal. We categorized the learned reward differences into five percentile bins and then binned $\delta$ accordingly to compute the average for each bin. From the figure, it is evident that larger $\delta$ values correspond to smaller learned reward differences.

W2: Given that the canonical framework for RLHF of in LLMs is using REINFORCE or PPO with a learned reward model (through logistic regression) , results in that setting are necessary given the framing / positioning of the paper.

Response included in point 4 in the global rebuttal.

Q1: I am a bit concerned about the results in Figure 3, from reading off the graph, the increase / decrease in Winning score when changing the perturb % is about the same for $R^3M$ and DPO for dialogue? It would be clearer if the authors simply plot the delta in the score as that's the metric underlying raw performance, that we care about in this context.

Thank you for your suggestion. Since both the HH dialogue and TL;DR summarization datasets contain over 20% noise labels [3] and the response quality in these datasets is considered low, evaluating our methods solely on these datasets could lead to unexpected results, as you pointed out.

To address this, we trained our method on the high-quality UltraFeedback dataset [3] with different levels of random perturbation. Due to resource limitations, we considered only 10% and 20% perturbations. The results are shown in Table 6 in the attached file. We evaluated the standard DPO and $R^3M$-DPO by their win rates against HH SFT. We define $\Delta$ as the win rate difference between two methods and present the value in the table. We will include the plot of delta in our next version.

Q2: More ablations along the extremes would help ground the experimental results further and increase the quality of the work. e.g. for $\lambda$, higher perturb % (100% perturbation would give a sense of how big the drops in performance are relative to their practical worst-case).

Response included in point 3 in the global rebuttal.

Q3: Reporting the ranking accuracy when training DPO (i.e. % of time $r_w-r_l>0$) with and without the robustification scheme would further strengthen the results.

**Rebuttal: ** Thank you for your suggestion. Including the ranking accuracy on the test set will provide additional support for our results. Table 5 in the attached file shows that training DPO with robustification scheme slightly improves the ranking accuracy compared to the one without robustification scheme. We would like to note that the accuracy can not completely reflect its quality towards policy optimization, since the accuracy only evaluate the sign of the reward difference, but not the scale, which could be important during policy optimization. Additionally, the test set may also be contaminated and hence the accuracy may not be accurate.

Reference

[1] Amini, Afra, Tim Vieira, and Ryan Cotterell. "Direct preference optimization with an offset." arXiv preprint arXiv:2402.10571 (2024).

[2] Gao, Yang, Dana Alon, and Donald Metzler. "Impact of preference noise on the alignment performance of generative language models." arXiv preprint arXiv:2404.09824 (2024).

[3] Cui, Ganqu, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. "Ultrafeedback: Boosting language models with high-quality feedback." arXiv preprint arXiv:2310.01377 (2023).

Please respond to the authors' rebuttal to acknowledge that you have read it, and let them know if they've successfully answered your questions and you are willing to change your score, or if you need additional clarification. The discussion period is ending tomorrow, so to give authors sufficient time to respond please try to respond to their rebuttal today.

Currently, while reviewers unanimously agree on accepting this paper (scores are 5,5,6,6), the scores are also low, which means it may not reach the bar for acceptance unless some reviewers increase their score.