Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

We sincerely thank you for your valuable feedback and constructive comments. Below, we address each of your concerns in detail.

Q1: The discussion on pointwise noise and DRO lacks clarity.

A1: We appreciate the opportunity to clarify. The key point is that DPO fundamentally operates as an offline algorithm, where both DPO and Dr. DPO rely on pairwise data sampled from $\pi_{\text{ref}}$ (or $\pi_{\text{SFT}}$). Let us explain the sample acquisition process, particularly using datasets like "princeton-nlp/llama3-ultrafeedback":

1. For each prompt $x$, the SFT model generates five responses with a sampling temperature of 0.8.
2. These responses are scored using "llm-blender/PairRM," with the highest-scoring response selected as $y_w$ (winner) and the lowest as $y_l$ (loser).

Thus, the empirical sample distribution is inherently tied to $\pi_{\text{SFT}}$, and DPO consistently operates under this static distribution.

In contrast, PPO is an online algorithm. It dynamically updates the policy $\pi_\theta$ during training, requiring the generation of new samples aligned with the current $\pi_\theta$. This fundamental difference means that PPO explicitly interacts with its evolving policy, while DPO remains constrained to the fixed $\pi_{\text{SFT}}$.

To further elucidate, we have added a detailed connection between DRO and DPO in Appendix C.2, highlighting their shared optimization framework and differing motivations.

Q2: The model used in pairwise experiments is not explicitly stated in the main text. The Pythia 2.8B model is mentioned only in the appendix, which reduces confidence in the experimental results.

A2: Thank you for pointing this out. To address this concern, we have explicitly stated in the main text that all pairwise experiments are conducted on the Pythia 2.8B model unless a different base model is explicitly mentioned. This follows the experimental setup outlined in the original DPO work.

Q3: The authors have omitted relevant literature on applying Distributionally Robust Optimization to LLMs, specifically, Oren, Yonatan, et al.'s "Distributionally Robust Language Modeling."

A3: We appreciate the suggestion to include this important reference. We have incorporated a discussion of "Distributionally Robust Language Modeling" by Oren et al. into the related work section. This reference highlights the broader application of DRO principles to language modeling, providing additional context for our contributions.

Q4: Setting $\beta' = 1$results in $L\_{\text{DR.DPO}} = -\log \mathbb{E}[\exp(h\_{\text{DPO}})]$, which differs slightly from the original DPO formula. A more detailed interpretation of this form would be insightful.

A4: The primary difference between the original DPO formula and DrDPO lies in the introduction of the Log-Expectation-Exp structure:

This structure offers advantages for handling label-flipped noise. Examining the gradient formulation provides further insight:

This reweighting mechanism assigns higher weights to samples with larger $h\_{\text{DPO}}(x, y\_w, y\_l)$ when $\beta'$ is small. As $\beta'$ grows large, the gradient aligns with the original DPO formulation, reverting to a uniform distribution.

We empirically found $\beta' = 1$ to be a well-performing hyperparameter for our experiments.

Q5: In Figure 5 (left), Dr. DPO shows higher win and loss rates at 0 flips. Does this imply that incorporating Dr. DPO leads to a trade-off between performance and robustness?

A5: The 0% flipped case indicates that we did not introduce any label flips to the dataset, i.e., the original dataset. However, this does not imply that the dataset is devoid of label noise. Dr.DPO continues to outperform DPO and IPO in the 0% flipped case scenario, further substantiating the presence of some label flipping noise within existing datasets, which also serves as the motivation for Dr.DPO. This finding aligns with the results of rDPO [1], where a default flip rate of 0.1 was shown to achieve better performance on the HH dataset, as evidenced in Table 3 of their work.

References:
[1] Chowdhury, S. R., Kini, A., & Natarajan, N. (2024). Provably robust DPO: Aligning language models with noisy feedback. ICML 2024.

We hope these clarifications address your concerns comprehensively. Thank you once again for your constructive feedback and the opportunity to improve our work.

We sincerely thank you for your valuable feedback and constructive comments. Below, we address each of your concerns in detail.

Q6: If $\pi\_{\text{ref}}$ or $\pi\_{\text{SFT}}$ is the empirical distribution (relate to $Q$ in the above equation), then is $\pi\_{\theta}$ $Q'$?

A6: Your observation is correct that Equation (7) might lead to some misunderstanding. For a more precise formulation, the problem can be expressed as: $\max\_{\pi\_r} \mathbb{E}\_{x \sim \mathcal{O}, y \sim \pi\_r(y|x)}[r\_\phi(x, y)] \quad \text{s.t.} \mathbb{D}\_\phi(\pi\_r(y|x), \pi\_{\text{ref}}(y|x)) \leq \eta.$ In this context:

• $\pi\_{\text{ref}}$ (or $\pi\_{\text{SFT}}$) acts as the base empirical distribution, analogous to $Q$ in classical DRO.
• $\pi\_r$ serves as $Q'$, the alternative distribution we optimize over to identify the optimal reward function under the ambiguity constraint.

Q7: "The estimation of $\theta$ should be conducted in the second equation above, after obtaining $L\_{\text{DRO}}(\theta)$ by maximizing the loss around distributions around the empirical distribution."

A7: The standard DRO framework typically consists of two optimization processes:

1. Inner Maximization: Identify the distribution $Q'$ within the ambiguity set, which maximizes the expected loss: $Q' = \arg\max\_{Q': \mathcal{D}(Q', Q) \leq \eta} \mathbb{E}\_{x \sim Q'}[L(x, \theta)].$
2. Outer Minimization: Minimize the obtained loss with respect to the model parameters $\theta$: $\hat{\theta} = \arg\min\_{\theta} \mathbb{E}\_{x \sim Q'}[L(x, \theta)].$

In our method, we achieve an analogous structure but with a focus on the preference-based framework:

1. Inner Maximization: We solve for the policy $\pi\_r(y|x)$ within the ambiguity set around $\pi\_{\text{ref}}(y|x)$: $\pi\_r(y|x) = \arg\max\_{\pi\_r} \mathbb{E}\_{x \sim \mathcal{O}, y \sim \pi\_r(y|x)}[r\_\phi(x, y)] \quad s.t. \mathbb{D}\_\phi(\pi\_r(y|x), \pi\_{\text{ref}}(y|x)) \leq \eta.$ This ensures that the reward function $r\_\phi(x, y)$ and the policy $\pi\_r$ are optimized within the divergence constraint.
2. Outer Minimization: The parameters $\theta$ are then optimized by minimizing the loss function derived from the Bradley-Terry (BT) model: $\theta = \arg\min\_{\theta} -\mathbb{E}\_{(x, y\_w, y\_l) \sim \mathcal{O}} \left[\log \sigma \left(r\_\phi(x, y\_w) - r\_\phi(x, y\_l)\right)\right].$ Here, $r\_\phi(x, y)$ represents the reward function obtained from the inner maximization process. Thus, while our approach does not explicitly solve a separate $\min\max$ structure, it achieves a similar two-level optimization.

Note:

• In RLHF, the policy $\pi\_\theta$ is updated through RL (PPO), and $r\_\phi$ is updated through Reward Modeling (BT model). In contrast, DPO integrates these two steps. It performs an inner maximization over $\pi_r(y|x)$, allowing $\pi_r$ to represent $r_\phi$. This is followed by an outer minimization with respect to $\theta$, parameterizing $\pi_r$ as $\pi_\theta$.
• Our aim is not to force a connection between DRO and DPO, but to provide a DRO perspective to explain the role of $\beta$ in DPO.

Why Not the Traditional $\min \max$ Form?

There are several reasons why the traditional $\min\max$ form is not explicitly used in our method:

1. Efficient Representation Through Analytical Maximization:

   • The traditional DRO formulation involves a costly $\max$ operation over an arbitrary distribution $Q'$ within the ambiguity set. However, in our framework, we leverage the problem's structure (e.g., convexity) to analytically solve the inner maximization.
   • By incorporating the results of the inner maximization into the loss function, we eliminate the need to explicitly perform the $\min\max$ optimization.

2. Alignment with Preference Optimization:

   • Our method is closely tied to preference learning models (e.g., DPO), where the primary goal is to optimize a preference-based loss rather than explicitly adjust distributions. The $\min\max$ form is reinterpreted in terms of optimizing rewards ($r\_\phi(x, y)$) under divergence constraints, which is more computationally feasible.

3. Theoretical Contributions Focus on Inner Maximization:

   • As an example, consider the classic paper [1] on DRO with KL divergence. In its Section 2.1, titled "Solving the Inner Maximization Problem," the analysis focuses on addressing the inner maximization challenge. Similarly, our Theorem 3.1 and Lemma 3.2 are derived based on the Inner Maximization Problem without additional assumptions, allowing independent optimization.

Beyond the analysis above, our contribution includes Dr. DPO: we offer an enhanced version of DPO that additionally achieves pairwise robustness, broadening its applicability and robustness attributes.

[1] Hu Z, Hong L J. Kullback-Leibler divergence constrained distributionally robust optimization[J]. Available at Optimization Online, 2013, 1(2): 9.

We greatly appreciate your thoughtful feedback and insightful suggestions. Below, we address your concerns in detail, aiming to clarify and improve our paper.

Q1: The assumptions about pointwise and pairwise noise in Subsection 3.1 and Subsection 4.1 are not quite comparable, which may affect the conclusion in the paper that DPO is robust to pointwise noise but not robust to pairwise noise.

A1: We define pointwise and pairwise noise based on practical considerations during fine-tuning large models:

1. Pointwise Noise refers to low-quality data points containing irrelevant or incoherent information. This type of noise can undermine the reliability of the initialization model $\pi_{\text{SFT}}$ during the DPO phase.
2. Pairwise Noise arises from erroneous associations between data pairs, leading to misjudged preference rankings. This type of noise is likely to be amplified during the DPO phase, thereby causing overfitting issues related to label flipping.

Furthermore, we emphasize that pointwise and pairwise noise are inherent to different types of noise and are not meant to be directly comparable. Our statement that "DPO is robust to pointwise noise" is grounded in its alignment with the DRO objective (Section 3.2). While pointwise noise may lead to suboptimal initialization of $\pi_{\theta,0} = \pi_{\text{SFT}}$, adjusting $\beta$ enables better optimization toward a reliable $\pi_\theta$. Conversely, pairwise noise directly impacts DPO's objective function (Section 4.3), which Dr. DPO aims to address.

Q2: There is a lack of ablation experiments regarding the batch size in the paper.

A2: To examine the effect of batch size on optimization results, we conducted additional experiments using Llama3-8B-Instruct as the base model. Preference optimization was performed on the UltraFeedback dataset, and evaluations were conducted on the AlpacaEval2 benchmark. The results are presented below:

Notes:

• The "batch size" in the tables refers to the number of samples used in a single computation of $\mathbb{E}_O$. Gradient accumulation was applied with gradient_accumulation_steps = 16, so the effective batch sizes are 64 (4×16), 128 (8×16), and 256 (16×16).
• Observations:
  1. Batch size significantly impacts DPO performance. In noise-free scenarios, larger batch sizes tend to reduce performance, which might be attributed to characteristics of Llama3-8B-Instruct (refer to Issue).
  2. Dr. DPO consistently demonstrates improved performance across all batch sizes, particularly in noisy scenarios, where the performance gains are more pronounced.

Q3: Some declarations in the paper are not clear, making it a little hard to follow.

A3: Thank you for highlighting this concern. We provide clarification for the following points:

1. "DPO is Implicitly a Pointwise DRO"

   • This statement is derived from the theoretical equivalence between DRO and DPO established in Theorem 3.1. In Equation 7, the RM-DRO objective addresses the robust optimization goals of reward modeling, and Theorem 3.1 connects this objective to DPO. Therefore, we state that "DPO is implicitly a Pointwise DRO."
   • To further elucidate, we have added a detailed connection between DRO and DPO in Appendix B.2 and Appendix C.2, highlighting their shared optimization framework and differing motivations.

2. "Why DPO is Robust to Pointwise Noise"

   • By definition, DRO mitigates pointwise noise through robust optimization (Section 3.2). Since Theorem 3.1 demonstrates the equivalence between DPO and DRO, we conclude that "DPO is robust to pointwise noise."

   • The part of B. The Optimal Value of $\beta$ Reflects the Noise Level within the SFT Model. in Section 3.2, offers an alternative perspective on why DPO is robust to pointwise noise. This interpretation is supported by its theoretical relationship with the robustness radius and corroborated by experimental evidence.

We hope these additions address your concerns.

Q4: In Figure 4 (Left), there seem to be similar trends of trading off between Reward and KL. If a small KL value is required under the scenario of RLHF (e.g., less than 2), are similar rewards gained from different rates of flipped pairs?

A4: Figure 4 (Left) illustrates the trade-off between reward and KL under different label-flipping scenarios. The trends suggest that for similar KL values, corresponding rewards are also similar.

In the context of RLHF, if we focus on the single-step training process where the policy model samples responses and the reward model assigns scores, potentially causing label flipping, we posit that its optimization behavior aligns with that of DPO, and the overall trend appears consistent. However, since RLHF involves a multi-step reinforcement learning process, we believe this observed trend may not generalize across iterations. We plan to explore this issue further in future research.

Q5: The toy example in Section 4.3 is a little hard to follow. The function $h$ has been clearly defined in the previous text, but an inconsistent combination of parameters is passed in, which is confusing.

A5: We sincerely apologize for this oversight. The inconsistency is due to a typographical error, which has been corrected in the revised manuscript. The updated and more precise formulation is as follows:

This correction ensures consistency with the earlier definitions and clarifies the example.

We hope our responses address your concerns comprehensively. Thank you once again for your constructive feedback and the opportunity to improve our work.

We sincerely thank you for your valuable feedback and thoughtful suggestions. Below, we provide detailed responses to the concerns and questions raised.

Q1: The evaluation could be compared with more baselines, please refer to arXiv:2409.02795.

A1: We appreciate the suggestion to expand the scope of baselines. However, it is important to emphasize that the primary goal of Dr. DPO is not to develop a method that simply excels on benchmarks but to address the issue of pairwise noise. For this reason, we chose baselines that specifically target similar challenges, such as cDPO [1] and rDPO [2].

Additionally, we extended the concept of Dr. DPO to SimPO[3] (a SOTA variant of DPO). Below, we provide results comparing SimPO and Dr. SimPO under noise-free and noisy conditions (Noise Ratio = 0.2), using Llama3-8B-Instruct as the base model and the UltraFeedback dataset for preference optimization. And we evaluate their performance on the AlpacaEval2 benchmark. The results are as follows:

These results demonstrate that the DRO concept underlying Dr. DPO can be effectively extended to DPO-like methods, with particularly significant performance improvements in noisy scenarios. We hope this additional comparison alleviates your concern regarding the choice of baselines.

References:
[1] Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2023). A note on DPO with noisy preferences and relationship to IPO. https://ericmitchell.ai/cdpo.pdf.

[2] Chowdhury, S. R., Kini, A., & Natarajan, N. (2024). Provably robust DPO: Aligning language models with noisy feedback. In ICML 2024.

[3] Yu Meng, Mengzhou Xia, Danqi Chen (2024): SimPO: Simple Preference Optimization with a Reference-Free Reward. In NeurIPS 2024.

Q2: These evaluation tasks are too simple; some methods may be effective on benchmarks like IMDB but may not generalize well to more complex tasks.

A2: Thank you for your suggestion. To evaluate the impact of preference optimization methods on downstream tasks, we conducted experiments on a range of more challenging benchmarks listed on the HuggingFace Open Leaderboard. The results, using Llama3-8B-Instruct as the base model, and the UltraFeedback dataset for preference optimization. The results are as follows:

Additionally, we tested Dr. DPO on the AlpacaEval2 benchmark for further validation:

Across diverse tasks and benchmarks, Dr. DPO consistently improves performance without introducing significant complexity. These results validate the generalizability and robustness of our method.

Q3: Although the authors provide a beautiful theoretical proof, the objective of Eq. (12) seems to be in contradiction with IPO.

A3: Thank you for raising this concern. It is important to clarify that IPO and Dr. DPO address different challenges:

• IPO aims to mitigate overfitting to data by learning $h(x, y\_w, y\_l)$ to a specific value, focusing on regularization.
• Dr. DPO addresses the sensitivity of DPO to label-flipped noise by reweighting pairwise data within a batch, reducing the amplification of noisy labels.

Thus, the motivations and application scenarios of these methods differ fundamentally. To further support this distinction, we compared the performance of IPO and Dr. DPO using the UltraFeedback dataset with Llama3-8B-Instruct as the base model:

Note: In the aforementioned experimental setting, the performance of DPO and IPO closely aligns with the results reported in SimPO.

These results demonstrate that IPO does not achieve optimal performance under noise-free settings, whereas Dr. DPO consistently improves over DPO by addressing label-flipping issues. We hope this comparison clarifies the complementary nature of these methods.

Q4: Despite how funny this question might sound, can I understand Dr. DPO as simply adding an exponential function ($e^{\cdot}$) to the objective function of DPO?

A4: A more precise formulation is that Dr. DPO modifies the DPO loss function from $-\mathbb{E}[h(x, y\_w, y\_l)]$ to $-\beta' \log \mathbb{E}\left[\exp\left(\frac{h(x, y\_w, y\_l)}{\beta'}\right)\right]$, introducing a Log-Expectation-Exp structure. This structure enables the method to address label-flipping issues effectively.

The gradient formulation highlights the difference:
$\frac{\partial}{\partial h} \left(-\beta' \log \mathbb{E}\_{\mathcal{O}}\left[\exp\left(\frac{h}{\beta'}\right)\right]\right) = -\frac{\exp\left(\frac{h}{\beta'}\right)}{\mathbb{E}\_{\mathcal{O}}\left[\exp\left(\frac{h}{\beta'}\right)\right]}.$

This reweighting mechanism assigns higher weights to samples with larger $h(x, y\_w, y\_l)$ when $\beta'$ is small, while converging to a uniform distribution as $\beta'$ increases. Therefore, Dr. DPO generalizes DPO by incorporating this adaptive weighting mechanism.

We hope these responses address your concerns comprehensively. Thank you again for your thoughtful comments and the opportunity to improve our work.

We sincerely thank you for acknowledging our contributions and for providing valuable feedback. Below, we address each of your comments in detail.

Q1: Which implementation is used in the paper? How does the batch size affect the optimization results if using the former one?

A1:
In the paper, Dr. DPO computes the estimation of $\mathbb{E}\_O$ within a batch. To examine the effect of batch size on optimization results, we conducted additional experiments using Llama3-8B-Instruct as the base model. Preference optimization was performed on the UltraFeedback dataset, and evaluations were conducted on the AlpacaEval2 benchmark. The results are presented below:

Notes:

• The "batch size" in the tables refers to the number of samples used in a single computation of $\mathbb{E}\_O$. Gradient accumulation was applied with gradient_accumulation_steps = 16, so the effective batch sizes are 64 (4×16), 128 (8×16), and 256 (16×16).
• Observations:
  1. Batch size significantly impacts DPO performance. In noise-free scenarios, larger batch sizes tend to reduce performance, which might be attributed to characteristics of Llama3-8B-Instruct (refer to Issue).
  2. Dr. DPO consistently demonstrates improved performance across all batch sizes, particularly in noisy scenarios, where the performance gains are more pronounced.

Q2: Experiments on more models and sizes will make the results more convincing.

A2: In Table 3 of the original manuscript, we presented results using Llama2-13B. In response to your comment, we have included experiments with the Llama3-8B-Instruct model under different batch sizes (detailed in Q1). Additionally, we have extended our evaluation to Mistral-Instruct-7B, using AlpacaEval2 as the benchmark. The results are as follows:

These results highlight the consistent performance gains of Dr. DPO across both model types (Llama and Mistral) and varying model sizes (7B, 8B, 13B).

Q3: There are several papers discussing reweighting DPO, such as WPO (arXiv:2406.11827v1), which could be discussed in related work.

A3:
Thank you for this suggestion. In the revised manuscript, we have incorporated a discussion of reweighting DPO methods, including WPO and related work. We appreciate your recommendation, which has helped us strengthen the related work section.

Q4: Typo: line 194: \pi_\theta

A4:
We greatly appreciate your meticulous review. The typographical error in line 194 (\pi_\theta) has been corrected in the revised manuscript.

We hope that our responses and the additional experiments adequately address your concerns. Thank you once again for your constructive feedback and the opportunity to improve our work.

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address your comments in detail.

Q1: It is better to provide more datasets like TruthfulQA, GSM8k, and other datasets.

A1: We appreciate your insightful suggestion. To evaluate the impact of preference optimization methods on downstream task performance, we conducted additional experiments using Llama3-8B-Instruct as the base model. Preference optimization was performed on the UltraFeedback dataset, and evaluations were conducted on the different tasks such as ARC-Challenge, TruthfulQA-MC2, and GSM8k. The results are summarized below:

These results demonstrate that Dr. DPO achieves consistent performance improvements across various tasks with just a single additional line of code.

Due to time constraints, we primarily explored the noise-free and noise ratio = 0.2 settings. However, the results validate that Dr. DPO is robust to noise and offers stable performance gains across diverse datasets.

Q2: It is better for the authors to conduct experiments on more open-source LLMs to further verify the effectiveness of their algorithms.

A2: Thank you for this suggestion. We extended our evaluation to include both Llama3-8B-Instruct and Mistral-Instruct-7B as the base models. Preference optimization was performed on the UltraFeedback dataset, and evaluations were conducted on the AlpacaEval2 benchmark. The results are as follows:

These results confirm that DrDPO consistently improves performance across different model families (Llama and Mistral) and model sizes (7B and 8B), further validating the robustness and generalizability of the proposed approach.

Q3: The authors can provide some more case studies to better understand the performance of their method.

A3: Thank you for this suggestion. To better illustrate the qualitative impact of our method, we have added several case studies to the Appendix E of the revised manuscript. These examples highlight scenarios where DrDPO demonstrates improved text quality and response accuracy compared to baseline methods. We hope these additions provide a clearer understanding of the practical benefits of our approach.

We hope these additional results and clarifications address your concerns comprehensively. Thank you again for your thoughtful review and the opportunity to improve our work.