Preference learning made easy: Everything should be understood through win rate

Thank you for your review. We answer your questions (paraphrased for brevity) below:

1. [datasets and model of Figure 2?]

• The reviewer is correct that the model is Pythia 2.8B. The dataset is Open Assistant.

2. [Table 1 shows different h win rate methods, but did not show a clear winner, and thus it's hard to draw a useful conclusion there]

• We'd like to emphasize that the lack of a clear winner is the primary conclusion of Table 1. This result suggests that design choices for WRO which affect the optimal solution do not seem to have a first-order systematic effect on performance in practice; optimization success does though, as a show later in Figure 3.

3. [Figure 3 shows the correspondence between train loss and win rate, but I'm curious about the sudden drop of loss on the left subfigure.]

• Great question. Each point on either plot in figure 3 is a different final model from a training run of a given WRO variant. The sudden drop of loss denotes a subset of models which achieve final loss past a certain threshold (around -0.35). These models achieve much better win rates than models which are not able to reach such a loss value, suggesting that simply being able to optimize past a certain threshold is strongly indicative of test win rate performance.

4. [The experiments, while well intended, look a bit small in scale and not very strong. It would be helpful to conduct higher scale experiments, and show better correspondence between win rates and train loss, which is the main benefit of the methodology part.]

• We'd like to emphasize that the current experiments consist of 72 distinct RL fine-tuning runs on a several billion parameter model. Unfortunately, limited resources prevent us from significantly scaling up past the current experiments during the response period, but if the reviewer has specific suggestions, we'd be happy to incorporate them into the final version of the paper.
• It is also worth noting that the level of correspondence we are already seeing between train loss and test win rate is extremely striking, especially given the fact that different WRO variants do not necessarily even share the same scale for loss (due to choice of $\beta$ and $h$). In other words, we would not expect, a priori, such significant rank correlations, much less a better correspondence. Instead, the fact that the correlation exists with train loss, even when it doesn't with other design choices we hypothesized a priority to be relevant, suggests that optimization success is the first order consideration for WRO.

5. [Is it possible to extend the proposed pairwise metrics to listwise?]

• Yes! In the case of preference for a set of $k$ responses, the binary preference classifier would become a k-class preference classifier, and h-win rate would become a multi-way win rate over $k-1$ anchor distributions, which could optionally be the same distribution. Additional assumptions also generalize, such as Plackett-Luce for Bradley-Terry. We’ve added this discussion to the appendix.

Thank you for your review. Please let us know if you have any additional questions or concerns; otherwise, we would greatly appreciate it if you would consider raising your score.

Thank you for the thoughtful review! Questions (paraphrased) and responses below:

1. [Undesirable that optimal solution depends on anchor, not true for reward maximization]

• This dependence is a limitation of the information contained in the distribution of pairwise comparisons itself. As mentioned in Prop 4.1, we agree that the issue can be bypassed with additional assumptions (e.g., under BT and finite rewards, optimizing any ℎ-win rate over any anchor optimizes for all ℎ-win rates over all anchors). We have added more discussion.

2. [Better motivations for properties, prevalence-consistency for generator rules out non-RL]

• We want preference-consistency or else a model can be deemed good even if it generates dispreferred responses. We want prevalence-consistency wrt generator or else we are not evaluating the model’s generation behavior. Enforcing prevalence-consistency means objectives include an expectation over the model, but this could be approximated with off-policy samples and importance weighting for instance.

3. [I don’t think you need BT to prove Prop 4.1]

• Good point. Prop 4.1 is meant to show what BT gives us (see 1), but we have generalized the proof.

4. [Two anchor/competitor distributions is confusing]

• Could you clarify what you mean? If you mean $p_{anchor}$ vs. $p_{ref}$, we distinguish them to highlight that the anchor we are optimizing against can be different from the reference we are regularizing to (but please let us know if we are misunderstanding!)

5. [Highlight that DPO is win rate-consistent]

• DPO’s optimal solution matches that of a regularized WRO objective; thus we can say that it satisfies regularized win rate consistency. We mention this in appendix A and have added a line in the main paper, but it is also the motivation for DPO, i.e., an objective whose solution matches that of RLHF.

6. [Is win-rate correspondence a big deal? Can you clarify the intuition for DPO?]

• Failing win rate correspondence makes training and model selection difficult (i.e., why reduce loss if it doesn’t help with the goal, how to select checkpoint when best loss is not best win rate).
• Intuition: DPO performs MLE of the preference classifier, where this classifier is a function of the policy model. If loss goes down, this implicit preference classifier is getting closer to the true preference classifier (in expectation over the offline distribution). However, this does not mean that the corresponding policy model is getting better at win rate under the true preference classifier. First, prevalence mismatch between offline data and model means the preference classifier could get worse for responses that are relevant under the model even as it gets closer to ground truth for the offline data. Second, DPO breaks preference consistency as the implicit preference classifier is a function of the policy model itself and thus changes throughout training as the policy model changes.

7. [Elaborate on three cases for $p_{\ell}$]

• Oracle is a model used to label the preference data and evaluate the win rate of the trained models. BT = True is a reward model trained on the oracle-labeled preference data. BT = False is a preference classifier trained on the oracle judge-labeled preference data without BT (lines 377-384).

8. [SFT > DPO?]

• Summarizing footer 4, we conclude SFT > DPO with respect to ease of optimization but based on works which show optimization difficulties of DPO, e.g., failure to increase the probability of the chosen response or improve the rankings.

8. [I suspect some win rate correspondence violations of DPO are due to KL, Fig 1 is extreme]

• To formalize, we add a new property called regularized win rate correspondence, i.e., improvement in loss implies improvement in win rate or divergence to reference. This is very inclusive, as even a change that results in a large decrease in win rate will still meet regularized win rate correspondence if divergence to reference decreases. Any regularized WRO objective satisfies this. DPO (off- or online) does not: loss can improve even though win rate and divergence get worse (as DPO does not directly optimize for either).
• Note that Fig 1 is less extreme than DPO as setting is online and infinite data. Fig 5 is even less extreme with a uniform starting model and equally spaced rewards, and Fig 6 further decreases $\beta$ in the least extreme setting in Fig 5. These figures show that DPO still does not satisfy win rate correspondence even as we continue to make the setting more and more favorable (and at some point, smaller $\beta$ doesn't help). Even so, given that the Fig 1 is quite specific, we've moved it to the appendix and have replaced it with a discussion on regularized win rate correspondence.

10. [Additional related work]

• Thank you, they were a pleasure to read, and we have added them to related work.

Thank you again for your questions. If we have answered them to your satisfaction, we hope you will consider raising your score.

Thank you very much for your review. We answer your questions (paraphrased for brevity) below:

1. [Theory in the paper is similar to IPO. Proposition 4.1 can be proven using equation 7 in IPO with $\tau$ going to zero. More empirical analysis can distinguish the paper.]

• We agree that one can use equation 7 in the IPO paper to gain intuition for proposition 4.1, but equation 7 in the IPO paper is the target distribution specifically for KL-regularized WRO objectives and does not include the additional analysis required to take the limit of $\tau$ to zero (e.g., proving existence of a limit, connection between limit of solution and solution of limit`) nor the specific implication of the Bradley Terry assumption. In contrast, Proposition 4.1 in this work is meant to emphasize a consequence of the Bradley Terry assumption on the dependence of WRO on choice of anchor distribution: Namely, it proves that under the Bradley-Terry assumption and finite rewards, the resulting solution optimizes win rate over all possible anchors regardless of the choice of anchor in the objective. In other words, the Bradley-Terry assumption effectively gives us the ability to optimize over all anchors, whereas without that assumption we choose the anchor. We have updated the text to emphasize this point.

2. [Expected versus observed win rates; the authors suggest there might be other factors at play, such as optimization etc., but these are not supported by any empirical evidence.]

• Section 6.3 / Figure 3 provides empirical evidence of optimization being an important factor for the misalignment between expected versus observed win rate patterns. Namely, the train loss achieved by a given run is significantly predictive of the test win rate, even as other factors which dictate the target distribution of the objective are not (Table 1). This result is especially striking given that the losses for different objectives do not necessarily share the same scale.

3. [It is interesting to me that using oracle $\hat{p}_\ell$ doesn't exhibit the best performance]

• We agree. This result suggests that there is a more important first order factor, which Figure 3 (a scatterplot of train loss versus test win rate for selected models across different WRO runs), and the corresponding significance test suggests is optimization success.

4. [The paper uses only Pythia-2.8b model to train preference models as well as to judge results. I think this limits the analysis. Using the same base model for both preference learning and judging can introduce biases. It would help if other similar sized models, such as Gemma2-2b, Qwen2-1.5b, are used as well.]

• Due to resource constraints, we are unfortunately unable to run the same experiments on additional models at this time (requires 7 offline finetuning jobs and 36 online finetuning jobs for just a single dataset with no hyperparameter sweep), but we take the reviewer's point that it would be helpful to add more models to the final paper and are working to do so. And just to clarify, we are using separately trained models for the judge and policy model even if they are based on the same base model.

5. [I think proposition 3.3 and definition 3.2 can be practically useful to check whether a preference learning method respects the win-rate objective. The alternative, comparing the objective directly to win-rate objective, can be more challenging.]

• Thank you for highlighting the practical usefulness of proposition 3.3 and definition 3.2. We agree!

6. [Section 4.2 notation]

• Thanks for the feedback, fixed!

We hope we have addressed your concerns, and we hope you will consider raising your score. Thank you!

Thanks for the review. Responding to your concerns below:

1. [Overly idealized assumptions (in-distribution assumption, unreliable preference models) diminish practical significance of conclusion]

• Our framework is not focused on idealized assumptions per se but rather what can be learned from the preference data distribution alone, specifically to clarify the role of assumptions vs. information inherent in preference data itself. In fact, our analysis is meant to make it easier to consider challenges such as those posed by the reviewer: Namely, our work clarifies what is optimal when nothing else is considered besides the information contained in preference data alone. This understanding of what is in preference data makes it possible to disentangle anything extra in a given method as an additional assumption or strategy. We are not arguing that anything extra is bad if the resulting objective is not pure WRO; instead, win rate optimization should be seen as the starting point, with any addition or modification understood modularly based on the underlying assumption being encoded.
• To drive that point home, we've added the following lines to the discussion:
  • …this work offers a simplifying insight: win rate is the only evaluation that can matter based on preference data alone, and thus all of preference learning should be understood in relation to it—both how well a given method optimizes for win rate as well as the role of additional assumptions that move a method beyond pure WRO.

2. [Limited practical guidance for real world preference learning]

• We respectfully disagree; some immediately implementable practical guidance that comes out of our analysis includes 1. given how important and finicky optimization success is for WRO objectives (including RLHF), it could be helpful to kick off multiple seeds to find one that optimizes best; 2. given that DPO loss fails win rate correspondence on multiple fronts, one should consider performing model selection with a metric other than validation loss; 3. greater generation diversity as well as alternative filtering strategies can improve the win rate limits of supervised fine tuning on preferred samples; 4. studying optimization strategies (e.g., contemporaneous paper [1]) would be a high leverage direction of inquiry, given the importance of optimization success in practical performance of preference learning algorithms.

Based on our response, we hope you will consider raising your score. Thank you!

[1] https://arxiv.org/pdf/2503.14286