Preference Learning Algorithms Do Not Learn Preference Rankings

We thank the reviewer for their detailed review and greatly appreciate their support of the acceptance of our paper. We address the feedback below:

"First, while it is fair to say that DPO is not good at optimizing the ranking policy in terms of the non-regularized target, it is not immediately clear that the non-regularized ranking policy is a better metric."

We are not entirely sure we understood your question: what is the “non-regularized ranking policy,” and what would it mean for this policy to be a better metric? Thank you!

"In most of the cases, the ability of maximizing the reward/ranking accuracy, while maintaining a moderate KL divergence is more suitable to evaluate the algorithm performance in my viewpoint. Therefore, the ranking accuracy alone is also not a good learning target, which is also evidenced by figure 4 of this work."

We agree that optimizing for ranking accuracy alone is not necessarily a good learning target; to clarify, we are not suggesting that one should be optimizing ranking accuracy. Rather, we use ranking accuracy as a lens to better understand what models learn during DPO training, as well as when it correlates with win rate.

"However, in [SimPO’s] experiments, as the win rates evaluated on the standard benchmark significantly increase, we also observe a significant drop in some academic benchmarks, particularly those on the mathematical problems solving… I feel that the paper can still be largely improved if a more comprehensive evaluations can be done, like the trade-off between the performance on instruction-following benchmarks (alpaca-eval, mt-bench, gpt4 win rate) and other academic benchmarks"

If we understand correctly, the reviewer seems to be suggesting that, in addition to assessing win rate on the query distribution of interest, we should evaluate the models in the paper on benchmarks for other domains. We believe this is out of scope of this paper, which focuses on the fact that DPO does not even yield correct rankings on the training data. We do agree broadly with the reviewer’s sentiment, though, that it is useful to assess top-performing models on a wide variety of benchmarks.

The ``non-regularized ranking policy'' refers to equation (1), which does not consist of the regularization of the reference model in the original DPO reward.

My main point is that I do not think dropping the reference is a good idea. This is because even with the strong KL regularization, the DPO algorithm (and its variants) already suffers from considerable alignment tax. Therefore, chasing for a high ranking policy is too aggressive in my viewpoint, because it typically leads to performance regression in the out-of-distribution tasks. These have been widely observed in almost all the practices with DPO algorithm and has been mentioned in many original technical report of PPO-based framework (e..g instruct-gpt), as well as the more recent kto and simpo [1].

[1] The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism

We thank the reviewer for recognizing the “clear implications” of our theoretical and empirical analysis. We answer additional questions below.

"Specifically, the authors could benefit from a deeper analysis of the implications of using ranking accuracy versus win rate as measures of model performance, including potential discrepancies and how they might be reconciled."

Thank you for this suggestion. We agree that it would be interesting to further study when and to what extent offline metrics like ranking accuracy are indicative of generative behaviors from the model. Our experiments in Section 5 initiate a study of this, and we find that when the model is close to the reference model, the win rate and the ranking accuracy trend together. We hope that future work can study how different design choices, such as on-policy data (lines 269-275), affect this relationship.

“The paper identifies a significant determinism of ranking accuracy by reference model likelihoods, which is a crucial observation. It would be advantageous for the authors to expand on this point by discussing potential techniques or algorithmic adjustments that could alleviate the heavy reliance on the reference model.”

We agree that it is important to find better ways to regularize toward the reference model. Our formulation of the $L_\text{DPO}^\gamma$ objective in Equation 4 is one possibility, and we discuss several others in lines 269-275. We furthermore note that several prior/concurrent works, including CPO and SimPO, remove the reference model entirely and instead modify the objective and dataset to prevent the generative behavior from degrading during preference learning.

Thank you to the reviewer for your thoughtful comments and feedback. We provide responses to your concerns below:

"The motivation for studying ranking accuracy is weak.”

We study ranking accuracy because DPO directly optimizes for ranking accuracy. For example, the authors of the DPO paper state that "the DPO update increases the relative log probability of preferred to dispreferred response," which corresponds exactly to increasing the ranking accuracy. Our Corollary 3.3 + idealized ranking accuracy results (Table 1) also formalize this. Nevertheless, our paper shows that DPO fails to improve ranking accuracy even though it induces desirable behaviors such as win rate improvement, motivating the need for further investigation. Our analyses help us understand better what the model actually learns during DPO training (if not ranking), and can inform future improvements to preference learning algorithms.

"The analysis of PPO in terms of ranking accuracy is missing, which somehow makes the conclusion regarding on-policy and off-policy weak."

Figure 1b and Table 1 contain the ranking accuracies of multiple RLHF-PPO models. Theorem 3.1 and Corollary 3.3 also apply to both RLHF-PPO and DPO.

“The ranking accuracy, especially when defined on the offline dataset, usually does not correlate with the quality of generation behaviors."

We agree, and Section 5 goes further to plot these correlations explicitly. We find that when the model is close to the reference model, the win rate and the ranking accuracy trend together, but as training goes on, they become anti-correlated. Our paper does not argue that ranking accuracy should replace the win rate metric. Instead, we demonstrate that DPO does not accomplish what it was designed to do (i.e. optimize ranking accuracy), despite its success at improving win rate.

"Let us split the response set into two parts, R_good and R_bad. In practice, we care that sum_{y} P( y in R_good) > sum_{y'} P(y' in R_bad)... This says that ranking accuracy measures the worst-case behavior of generative models. Thus, we do not need to bother ourselves to care about the ranking accuracy"

The case brought up by the reviewer concerns generalization, while our analysis is concerned with whether DPO even achieves its own training objective, i.e., whether the model achieves high ranking accuracy on its own training dataset. This is a much easier case.

"The analysis in terms of ranking accuracy is somehow superficial. In fact, it is the algorithmic bias that DPO introduces the KL-regularized optimization, and as such, it cannot recover the preference-generating distribution."

We respectfully disagree. We agree with the reviewer that even the target distribution of DPO and RLHF-PPO may not have perfect ranking accuracy even on the training data (see discussion on idealized models in Section 3.2, and note that the Table 1 results under $\mathcal{R}^*$ are often less than 100%). However, our results show that there is a gap even between the idealized target distributions and models in practice (see Table 1). In other words, the reviewer’s analysis is not sufficient to explain the results of models in practice; to fill in this gap, we investigate what happens throughout training (Section 4 and 5).

Xiao et al. provides an interesting and complementary perspective on our work, but their proposed regularizer appears to promote higher perplexity in general (see Table 2 in Xiao et al.). Rather than resolving the "algorithmic bias" described by Xiao et al., we investigate the gap between on- and off-policy behavior.

We thank the reviewer for their valuable feedback and suggestions. We appreciate your recognition of the thoroughness of our work. We have included our responses to your questions below and new results in the PDF attached to the global rebuttal.

"One concern is that the ranking accuracy considered here is the hard accuracy (accuracy of the Bayesian optimal classifier). I wonder if the "soft" ranking accuracy will exhibit more noticable improvement after preference fine-tuning"

Thank you for this suggestion – we provide plots of the soft ranking accuracy during training for GPT2, Pythia 2.8B, and Llama 2 7B in Fig. 1 of the PDF attached to the global rebuttal. While there is some gradual improvement for the "incorrect->correct" group, there is little to no improvement for the examples that stay incorrectly ranked (i.e. "incorrect->incorrect" group).

"While it is already argued in the paper that ranking accuracy is practically more meaningful than the reward accuracy, it is still questionable whether totally discarding $\pi_\text{Ref}$ is reasonable. Specifically, the paper should have included experiment results on whether there is also a gap in reward accuracy between the learnt and ideal policy."

The reason we consider ranking accuracy instead of reward accuracy is, as mentioned in lines 133-134, "we ultimately sample from $\pi_\theta$ rather than $\frac{\pi_\theta}{\pi_\text{ref}}$." Moreover, the reward accuracy of many models can be quite high even if the models have terrible ranking accuracy. For example, consider an example $(x,y_w,y_l)$ where $\pi_\text{Ref}(y_w|x)=0.1,\pi_\text{Ref}(y_l|x)=0.8,\pi_\theta(y_w|x)=0.11,\pi_\theta(y_l|x)=0.72$. Then the ranking accuracy is 0 on this example because $\pi_\theta(y_w|x)=0.11<0.72=\pi_\theta(y_l|x)$ but the reward accuracy is 1 because $\log(\pi_\theta(y_w|x)/\pi_\text{Ref}(y_w|x))=\log(1.1)>\log(0.9)=\log(\pi_\theta(y_l|x)/\pi_\text{Ref}(y_l|x))$. Thus, reward accuracy is a much less useful metric to consider.

"It is not clear to me what is the main failing part in DPO/RLHF training for high ranking accuracy. In Theorem 3.1, it seems that the perfect RLHF policy should encode the information from $\pi_\text{Ref}$ and the dataset perfectly at the same time. Does the DPO/RLHF model fail to capture $\pi_\text{Ref}$ or the preference dataset?”

First, we would like to clarify that it is not always possible to encode both the reference model and the dataset perfectly, since there may be conflicts between the two. Indeed, the takeaway from Theorem 3.1 is that the optimal policy encodes the dataset preferences scaled by the reference model, so it is not always possible for the model to recover the true preference probability $\alpha$ if $\beta$ is not set correctly (see Table 1). We will be sure to clarify this in our next revision.

Section 4 studies why models fail to achieve high ranking accuracy in practice. Section 4.2 in particular demonstrates that when the reference model has an incorrect ranking on a datapoint, the DPO loss must be reduced to a very small value to correct the ranking (Theorem 4.1 and Figure 3). This value is so small that DPO typically overfits before it reduces the loss to this value (as demonstrated by Figures 2a and 3.) Thus, in practice, it is difficult to optimize the model to achieve the idealized ranking accuracy described in Section 3.

"Have the authors tried to plot the quantity $\frac{\pi_\text{DPO}(y_w)\pi_\text{ref}(y_l)}{\pi_\text{DPO}(y_w)\pi_\text{ref}(y_l)}$ and compare it to $(\frac{\alpha}{1−\alpha})^\beta$ (perhaps a scatter plot)? This can illustrate more clearly what DPO/RLHF failed in reaching the perfect policy."

Thank you for this suggestion. We agree that if we had access to the true proportions of raters who preferred one response over the other, denoted $\alpha$, this analysis would be insightful. Unfortunately, most preference training datasets report only binary preference values, as we discuss in lines 65-76. However, our analyses in Figures 2 and 3 provide a clear diagnosis of the failing – DPO encourages the model to continually increase the reward margins, and flipping the actual ranking is difficult (Theorem 4.1).