What Matters in Data for DPO?

1. Whether $\pi(y_w|x)$ is small

We apologize if we misunderstood your original comment and have done our best to respond based on our current understanding. We had difficulty locating the exact sentence “The quality of the rejected responses may not always be critical when $\pi(y_w|x)$ is small.” The closest related statement we could identify appears around line 190, where we discuss the roles of the distributions $\pi_w(y|x)$ and $\pi_l(y|x)$, rather than the specific probability $\pi(y_w|x)$ of generating a particular response $y_w$. Our intended point was the following: if two rejected response distributions, $\pi'_l(y|x)$ and $\pi''_l(y|x)$, differ only in regions where $\pi_w(y|x)$ is small, then the corresponding likelihood ratios $\pi_w(y|x)/\pi'_l(y|x)$ and $\pi_w(y|x)/\pi''_l(y|x)$ may still be similar. As a result, the model may exhibit similar behavior or performance under both distributions, despite the differences in rejected samples.

That said, following your suggestion, we carefully reviewed all the experiments we conducted. In most of our DPO runs, the probability assigned to the chosen response either fluctuates or decreases over training, which is consistent with observations reported in recent studies (e.g., [1], [2]). Therefore, we do not necessarily expect $\pi(y_w|x)$ to increase during training.

Furthermore, determining the optimal number of training epochs for DPO is nontrivial. As far as we are aware, the number of epochs varies widely across implementations. In our experiments, we find that training for two epochs generally yields better performance than just one. Additionally, widely-used methods such as classic online DPO and DPO with curriculum learning typically involve multiple epochs of training. We believe this variability is closely tied to both the characteristics of the data and the compatibility of the base model.

2. The increase of the chosen-rejected margin

We fully agree that the chosen–rejected margin tends to increase as DPO training progresses. However, we do not think that this observation contradicts the fact that DPO can converge to the SFT solution under certain conditions. Convergence to the SFT solution inherently involves increasing the logits (or likelihood) of the positive samples. This, in turn, implicitly reduces the relative likelihood of responses not used for instruction, thereby increasing the chosen–rejected margin as well. Furthermore, in our numerical experiments under the online DPO setting, we also observe that the probability of generating the chosen response increases over training, which closely resembles the behavior typically associated with SFT.

3. The collapse of online DPO

We fully agree with the reviewer that online DPO can be prone to collapse. We would like to respond from two perspectives:

First, the form of online DPO we study in Theorem 4.5 is a special case in which the chosen responses are of very high quality (e.g., written by human experts) and remain fixed throughout training, while only the rejected responses are updated and generated by the model. In contrast, standard online DPO typically generates both the chosen and rejected responses from the model itself, making it more likely to overfit and collapse. In our case, since the chosen responses are externally sourced and of consistently high quality, the risk of collapse is mitigated. We acknowledge that this distinction makes our setting narrower in scope, and we agree that your concern is highly relevant to the standard formulation of online DPO.

Second, we would like to point out that similar instability can also occur in SFT. There are numerous community reports indicating that SFT checkpoints sometimes perform worse than their pre-trained counterparts, even producing nonsensical or repetitive outputs. This issue has been observed in widely used models such as LLaMA and Gemma, including their most recent versions. These observations highlight that SFT, too, is vulnerable to overfitting and collapse under certain conditions.

4. Numerical results

We sincerely appreciate your great comments on the numerical results. First, we present the results of SFT on the chosen response, and compare them with the online DPO we studied in Theorem 4.5.

The table demonstrates that our online DPO setup yields performance comparable to that of SFT, consistent with the prediction of Theorem 4.5. Furthermore, by comparing the SFT results with those in the original Table 1, we observe that incorporating rejected responses does provide some performance benefit, though the improvement is relatively modest. We hope these additional results help address your concerns.

Also, we agree with your observation that, based on Table 2, comparing LG-HQ with SG-HQ (and similarly, LG-LQ with SG-LQ) may suggest that larger gap sizes lead to better performance. However, we would like to highlight that when comparing LG-HQ and SG-HQ, the difference is not solely in the gap size—the quality of the chosen responses in LG-HQ is also higher than that in SG-HQ.

To clarify this point, we present the revised Table 2 below. In the table, “Avg.Chs” and “Avg.Diff” denote the average quality score of the chosen responses and the average quality gap between the chosen and rejected responses, respectively. Δ Score represents the performance difference between two rows on the LC-AE2 benchmark.

• Part 1 of the table illustrates the comparisons we consider most informative—those that hold the gap approximately constant while varying the quality of the chosen response.

• Part 2 corresponds to the comparisons you suggested, where both the gap and the chosen response quality differ.

For example, the Avg.Chs values for LG-HQ and SG-HQ are –2.98 and –3.56, respectively, showing a notable difference in chosen response quality. This indicates that the observed performance differences may stem from both the chosen response quality and the gap size. As a result, we believe it is not safe to attribute the performance gains solely to the gap size based on LG-HQ vs. SG-HQ or LG-LQ vs. SG-LQ.

To further disentangle these effects, we constructed two additional datasets:

• LG-HQ-inverse, which retains the high chosen response quality of LG-HQ but reduces the chosen–rejected gap.

• SG-HQ-inverse, which maintains the chosen response quality of SG-HQ while increasing the gap.

As shown in Part 3 of the table, increasing the gap alone does not significantly affect DPO performance, thereby supporting our claim that the quality of the chosen response plays a more dominant role.

We hope these additional numerical results help clarify our conclusions and address your concerns.

Reference:

[1] Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., & Hanin, B. (2025). Unintentional unalignment: Likelihood displacement in direct preference optimization. ICLR 2025

[2] Ren, Y., & Sutherland, D. J. (2025). Learning dynamics of llm finetuning. ICLR2025

1. More comprehensive numerical experiments

Thank you very much for your constructive suggestion to conduct more comprehensive numerical experiments. Following your recommendation, we paired each prompt in both the OpenAssistant 2 and UltraFeedback datasets with five responses of varying quality, labeled as best, high, medium, low, and worst. For each dataset, we conducted two sets of experiments: (1) fixing the best response as the chosen one and varying the rejected response among the remaining four, and (2) fixing the worst response as the rejected one and varying the chosen response.

Open Assistant 2: fix the best as the chosen and vary the quality of the rejected.

The performance is actually quite similar across all five tasks, suggesting that the quality of the rejected response may not be a critical factor. This also implies that the chosen–rejected quality gap may not be fundamentally important. For instance, on GSM8K and TruthfulQA, using the medium-quality response as the rejected one yields the best performance. Similarly, on IFEval, using low-quality responses as the rejected ones leads to slightly better results.

Open Assistant 2: fix the worst as the rejected and vary the quality of the chosen.

When the chosen response is fixed to be of the best quality, the model consistently achieves the highest performance across all tasks. This highlights the critical role that the quality of the chosen response plays in driving learning.

UltraFeedback: fix the best as the chosen and vary the quality of the rejected.

Similarly, the performance does not exhibit a monotonic trend as the quality of the rejected response increases or decreases. This confirms that the quality of the rejected sample alone may not be a reliable indicator of DPO performance.

UltraFeedback: fix the worst as the rejected response and vary the quality of the chosen response.

The set of experiments show that increasing the quality of the chosen response leads to a consistent, monotonic improvement in DPO training performance.

However, when we fix the worst-quality response as the rejected one, increasing the quality of the chosen response not only improves the quality of the chosen sample itself but also enlarges the gap between the chosen and rejected responses. To further disentangle the effect of this quality gap from the absolute quality of the responses, we design an additional set of experiments. Specifically, we construct four new datasets—LG-HQ, LG-LQ, SG-HQ, and SG-LQ. The LG-HQ and LG-LQ datasets have similarly large chosen–rejected gaps but differ in the absolute quality of their responses. In contrast, SG-HQ and SG-LQ have similarly small gaps but also differ in response quality.

“Avg.Chs” and “Avg.Diff” refer to the average quality score of the chosen responses and the average quality gap between the chosen and rejected responses, respectively. The Δ Score is the performance difference between the HQ and the LQ on LC-AE2. Across all four tasks, we observe consistently better performance when the chosen responses are of higher quality. Moreover, the magnitude of improvement is comparable across both large-gap and small-gap settings (8.7 vs. 7.1, 3.0 vs. 1.4, 3.1 vs. 1.9, 0.7 vs. 1.2), demonstrating the generalizability of this observation regardless of the gap size.

To double check our findings on the non-essential effect of the quality gap, we construct two additional datasets. The first, LG-HQ-inverse, retains the high chosen response quality of LG-HQ but reduces the chosen–rejected gap. The second, SG-HQ-inverse, preserves the high chosen response quality of SG-HQ while increasing the gap.

Expanding the gap does lead to some performance improvement, but the gains are notably limited compared to those achieved by improving the quality of the responses. In fact, on GSM8K, we even observe a slight decrease in performance when the gap is increased.

We hope these additional results address your concerns.

2. The numerical performance of SFT and online DPO

Here, we present the numerical results supporting Theorem 4.5, including evaluations on two datasets. We find that continual SFT and online DPO (with a fixed chosen response) exhibit similar performance across all five tasks. We sincerely thank you for this insightful comment, which has helped us improve the comprehensiveness of our experimental evaluation.

3. Terminologies

Thank you for your great catches. We will carefully update them in our next version.

Thanks for the additional experiments!

Based on the UltraFeedback fix the best experiment, it seems that as the quality of reject response decreases, the performance monotonically increases besides just 2 exceptions? (Medium/Worst GSM8K & IFEval). And for fix worst, I am not surprised of the results as a new finding because the preferred response intuitively serves as the upper bound of the performance.

For the SFT & DPO comparison experiment, it seems that DPO appears to perform noticeably better than continual SFT. What’s your take on that? So this does seem that DPO is doing something better than SFT?

One final question is, since the paper is centered on “what matters in data for DPO,” what message does it convey for practitioners constructing DPO datasets? In practice, we samples responses from the model without knowing their absolute qualities, and we query human to annotated the preference dataset. In some sense there is no control of the qualities generated by the model. For example, we cannot force the model to generate high quality response, and if the model is able to generate high quality response, there is no reason for us to not use the high quality response but rather use some low quality response as the chosen one. Plus, it is hard to evaluate about the absolute scores, which is why people use preference learning methods such as DPO. If absolute scores are given, you should use RL then. Thus, the practical limitations would be my biggest concern about the impact of the paper.

1. More details on the empirical section

We sincerely thank the reviewer for the great comments and suggestions. We will make sure that in the revised version, we will include the exact metric of each benchmark used in our experiments. And the following is for your reference.

Caption: Overview of Evaluation Benchmarks and Metrics. All the evaluations are run with only 1 attempt, i.e., under the pass@1 setting.

2. Concrete examples of $\pi_w$ and $\pi_l$

We greatly appreciate your valuable comments and will follow your excellent example in the revised version. We also recently found similar ideas of your example in prior work [1], which we will cite and discuss in the updated manuscript.

3. Example in line 198-204

We sincerely apologize for any confusion our presentation may have caused. Our intention was to use the example in lines 198–204 as numerical support, rather than very rigor theoretical justification, for the claim that "the quality of rejected responses may not always be critical." We fully agree that the likelihood ratios $\pi_w(y)/\pi_l(y)$ between best vs. worst and best vs. random can indeed differ. However, despite our efforts, deriving analytical results to characterize these differences seems to be challenging. We will highlight this as an important direction for future work.

Reference:

[1] Feng, Y., Kwiatkowski, A., Zheng, K., Kempe, J., & Duan, Y. (2025). Pilaf: Optimal human preference sampling for reward modeling. arXiv preprint arXiv:2502.04270.

We sincerely appreciate your effort and your time in reviewing our paper. Please see the below for our responses.

No new model/algorithm/methodological innovation

We sincerely appreciate your encouragement to translate our observations into new methods or algorithms. However, we respectfully disagree with the premise that contributions must involve algorithmic novelty to be considered innovative. A core contribution of our paper lies in theoretical clarity and novel insights into the role of data in the behavior of DPO, a widely adopted yet not well understood method. Our work characterizes the impact of various preference data structures on DPO’s performance—results that were previously unknown and that address pressing open questions in the community. Foundational analysis has a long-standing tradition of driving progress by revealing when, why, and how existing methods work. We believe our contribution fits firmly within this tradition.

Specific to DPO

We fully acknowledge that our analysis focuses specifically on DPO. However, we believe this focus is well-justified for several reasons. First, DPO is one of the most foundational and widely adopted paradigms in preference-based fine-tuning. Many recent alignment methods are either inspired by or closely related to DPO. While our results are centered on DPO, they have broader implications for understanding and improving these related methods. Second, our work provides a clear and extensible framework—particularly in the numerical analysis—that can serve as a starting point for investigating the role of data in other alignment approaches. To further support this, we will release our codebase, which can be easily adapted to analyze or benchmark other alignment methods beyond DPO.

Practical challenges in collecting high-quality data

Thank you for raising this important and practically relevant point. Our paper is primarily analytical in nature, and its goal is to explain the impact of data quality on the performance of DPO. While implementation-level data curation strategies are important, they might be outside the scope of this work. However, we do identify concrete properties—such as the informativeness of chosen–rejected margins and the effect of suboptimal preference scores—that guide future data collection. Rather than offering ad-hoc heuristics, our framework provides principled criteria for evaluating whether a dataset is suitable for effective DPO training.