Combating inherent noise for direct preference optimization

1. For Table2, I understand that the win rate of Adaptive-DPO vs. DPO* should increase as the noise rate increases, assuming Adaptive-DPO can resist the impact of noise better than DPO can. Could you explain why the indicators here do not show this trend?

2. Regarding "as the noise rate gets larger, our method can to some extent combat against the performance degradation resulted from more noisy data," I think you need an experiment comparing a model trained with DPO on synthetic data at different levels of noise rates with a model trained with DPO on real data to better support this point.

3. For the ablation study, I think it's missing the separate ablation results for W_{\theta}(x) and \Gamma_{\theta}(x), for both language models and text-to-image models, rather than just case study comparisons like in Figure 6.

4. Format Error: In Table 1, there is a wrong bold section of the PickScore column.

My concerns are as follows.

1. The claims that “the quality of preference data used in DPO training has been largely overlooked” (Lines 14-15) and “an often-overlooked aspect of the training process is actually the preference data used for training” (Lines 36-37) are incorrect. In fact, many existing studies [1,2,3] have noticed the prevalence of noise in the preference alignment and have proposed methods to address the challenges brought by noisy preferences.
2. Many important baselines for LLM alignment with noisy preferences are missing, such as C-DPO [2], R-DPO [3]. This point, together with the previous one, significantly hurts the professionalism and reliability of this work.
3. The motivation of the proposed Adaptive-DPO is based on a assumption---the model being trained tends to first learn the information in the clean label (as stated in Lines 404-406). However, this hypothesis requires solid theoretical or empirical support. PLEASE NOTE THAT I acknowledge that this phenomenon is common in some traditional classification tasks, but in the area of preference learning, this may not be true (i.e., DPO loss).
4. Theoretical or empirical analysis are needed to demonstrate the reliability of the proposed metric $u_\theta(x)$. As shown in Figure 3, when $0.1 \leq u_\theta(x) \leq 1$, the noise rate stably stays in the range of $(0.24, 0.30)$, which means that more than 70% samples are clean. Therefore, the metric seems to be ineffective in identifying noise.
5. AdaptiveDPO requires simultaneously training $M$ different models to compute the noise-aware metrics, i.e., bias and variance, which is expensive in real-world applications. Besides, the value of $M$ in experiments are not mentioned in Section 5.1.
6. The authors may want to follow existing studies [4,5,6,7] to use GPT-4 as the evaluator, which has already become a standard experimental setting in the preference alignment.
7. The authors may want to report the numbers of scalar metrics of model performance, such as ImageReward, PickScore, Aesthetic, and HPS, rather than just providing win rates under these metrics, as shown in Tables 2, 3, 5-6.
8. The authors may want to explain why the win rate of Adaptive-DPO vs DPO* (in Table 2) decreases as noise rate increases. This may indicate that Adaptive-DPO is ineffective under noisy preferences.
9. Many notations are used without explanation, such as the $\rho$ in Eq. (11), the $k_1, k_2, c_2$ in Eq. (15). Besides, what is the relationship between $c$ and $c_2$? Why do the authors use $\ast$ to denote multiplication operation? Although this is the last point, the weaknesses in notations are not minor. This is because preference alignment (whether RL-based or RL-free) is a direction based on theoretical derivation, and rigorous and clear mathematical presentation is very necessary.

[1] Impact of Preference Noise on the Alignment Performance of Generative Language Models

[2] A Note on DPO with Noisy Preferences & Relationship to IPO

[3] Provably Robust DPO: Aligning Language Models with Noisy Feedback

[4] SimPO: Simple Preference Optimization with a Reference-Free Reward

[5] Zephyr: Direct Distillation of LM Alignment

[6] Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators

[7] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

1. The paper needs more careful consideration of the problem of noisy labels in preference data. The authors mentioned that the noisy labels could due to the limited generalization ability of the machine annotators, inaccuracies from careless or malicious annotators, and the lack of standardized judgment criteria (line 038-043).

• For the model annotator case, the authors do not provide enough explanations, citations, or analyses.
• The human annotator case is a tricky problem. There could be two types of annotator mistakes: one is the annotator is not careful enough, and the other is the annotator has an individual understanding of the problem. In the first case, we should have algorithms to detect and correct the mistake during the training. In the second case, we should train the model to cope with or be aware of such differences. The authors should discuss the two cases and place their algorithm in more detailed context.
• In the lack of standardized judgment criteria case, more recent work tends to carefully curate the preference data and provide a reliable standardization [Llama2, Beaver]. Citation of recent work could help to make the problem more relevant and up-to-date.

2. The scope of the paper is confusing. In the introduction and preliminary, the authors mention that the paper focuses on the text and image generation tasks. However, exploration experiments, main results, and ablation are only on the image generation task.

3. The presentation of the results are confusing. It is not mentioned why the score-based metrics in table 1,2,3,4,5,6 are presented as win rate rather than the raw scores.

4. The comparison with other methods mentioned in line 669 help to understand the significance of the proposed method.

5. The proposed method is a bit confusing. Please see the question part.

[Llama2] Touvron et al. Llama 2 Open Foundation and Fine-Tuned Chat Model, 2023.
[Beaver] Dai et al. 2023 - Safe RLHF Safe Reinforcement Learning from Human, 2023.

• The definition and problem setup are questionable; The reviewer is not convincing about the existence of a 'clean' preference, since the preference is mostly somewhat subjective. Even for the same data pair, the preference might differ across the users so we can not call it 'noisy' or 'noisy label'. I think a model can only learn the overall human preferences, and it should not be perfect in some ambiguous cases. In Sec 3.1, the authors argue that the inconsistency among different annotators is evidence of the noisy labels, but this definition is questionable. In my opinion, this ambiguous case should be clearly distinguished from the noisy preference data, which may be adversarially annotated for very easy cases.
• Similarly, in Sec 3.2, only synthetic noisy preferences are used to evaluate the vulnerability of DPO. Is this a frequently occurring scenario? and in my view, the performance drop in Table 1 seems not significant.
• In the methodology section, do the roles of the reweighting and adaptive margin modules overlap, and how do they synergize with each other? or are they complementary?
• Several expressions are vague; 1) In line 54, the author says "if users aim to avoid copyright-protected symbols to evade costly penalties, such deviations can create serious problems". What serious problems? more detailed examples should be explained. 2) Prompts used to generate images are missing in Figure 4,5,6; Since most of the reward scores, including ImageReward, PickScore, and HPS, not only measure the image quality but also assess the text-image alignment, the exact prompts need to present together.