Win Rate is All that Can Matter from Preference Data Alone

Thank you for your review and acknowledging our “general framework” that can “motivate further studies on finding improved optimization objectives.” We address your specific questions and concerns below:

1. [The framework does not appear to account for unpaired/ranked preference methods.]

• We specifically consider the setting of pairwise comparison data in order to formalize what is possible with this form of data in particular. We believe that the community would benefit from the clarity of considering the distinct information offered by different forms of data / feedback / annotations and the implications for the methods that operate on such data. For instance, methods which use direct positive / negative annotations rather than preference pairs can take advantage of the knowledge that a particular sequence is considered globally bad / good and should not / should be generated; this offers different possibilities for evaluation unavailable in the preference pair case (e.g. probability positive sequences should be maximized). We believe future work that conducts an analogous inquiry to ours for other forms of data would be valuable for guiding the field of alignment / post-training methods.

2. [The judge model used seems weak / Stronger judge models should be evaluated for comparison.]

• We agree that the judge model is not a very accurate representation of the preferences encoded in the open-source datasets, but since we treat the resulting judge as our oracle preference classifier and relabel all data with it, its accuracy should not affect the results of our analysis, which focus on comparing different methods including in the ideal setting of having access to the oracle preferences.

3. [Diverse models could have been employed for more comprehensive results.]

• Unfortunately, we were unable to run the same experiments with additional models at this time (the updated experiments add up to over 80 runs, most of which are RL finetuning runs utilizing two A100s for 48 hours each), but the theoretical results hold regardless of model size.

4. [The framework does not address biases in preference learning, such as length bias.]

• Under the framework, we can say the following about length bias: under perfect optimization, the result of an objective such as RLHF or DPO is simply a function of the starting model and the underlying preference classifier for the environment $p(\ell=1|x, y_0, y_1)$. If neither the starting model nor preference classifier has a systematic preference of longer outputs over shorter ones, the framework predicts that the expected solution of existing objectives should not incur a length bias, meaning that any observed length bias is not a fault of the theoretical ideal implied by the objective but rather a result of estimation error (i.e., not reaching the target distribution). If either the starting model or the preference environment has a preference for longer inputs, it is possible to estimate how the preference propagates to the expected target distribution of a DWRO-KL objective like RLHF or DPO; then, a preference for length beyond what is expected can also be attributed to estimation error.

5. [The paper focuses solely on the DWRO-KL variant without exploring other possible variants.]

• Thanks for the feedback. We have added in the discussion of DWRO that the regularization need not be reverse-KL in particular; the chi-squared divergence is another example that could be optimized in the same way.

6. [Why do you fine-tune on dispreferred outputs? Which step is that in the framework?]

• Good question. The reason we finetune on all outputs is to make our initial model approximate the distribution of the preference data samples, thus simulating the case where we improve the model through preference annotations alone.

7. [Could you clarify what the anchor distribution is? It is used interchangeably as p or q, which adds to the confusion. Is it solely based on y_0 as defined in Definition 2?]

• The anchor distribution is the distribution we compare against when measuring win rate. Often, it is the initial start model, as this is generally the one model available other than the current model being trained, but it theoretically could be any model. We’ve updated the notation to be consistent (i.e., always p(y_0|x)).

(continued in part 2)

Thank you for your review and for appreciating the proposed framework as a “useful lens for interpreting past and future work in this domain.” And thank you for your insightful questions and comments; they have definitely helped us strengthen the paper. We address each below:

1. [The paper feels lacking in actionable insights. While the authors offer a useful framework for understanding different approaches to learning from preferences, leading to a range of alternative objectives beyond the standard RLHF objective, RLHF still appears to be the top-performing method. This raises questions about the practical value of the additional flexibility provided by the proposed framework.]

• Thanks for the feedback. We’ve incorporated the following three actionable insights to the paper discussion:
  • The (expanded and updated) comparisons of DWRO objectives show that optimization success is more indicative of performance than the spectrum of design choices around DWRO, suggesting that perhaps the highest leverage strategies for improving DWRO methods (including RLHF) is improving optimization rather than proposing a new variant of such an objective.
  • The fact that RLHF > DPO > SFT based on closeness to DWRO but SFT > DPO > RLHF based on ease of optimization points to the potential benefit of mixing and matching methods, a strategy that is already used (e.g. SFT then RLHF, DPO + SFT) but could possibly be leveraged further.
  • Overall, the framework suggests the following directions for future work: 1. Improving the optimization of an objective that already directly optimizes for win rate, or 2. developing easier-to-optimize objectives that are closer to direct win optimization than existing alternatives.

2. [Additionally, although the finding that SFT improves win-rate is intriguing, I would appreciate more clarity on why this insight is valuable for the community.]

• Thanks for the feedback. The primary insight comes from the analytical expression for the win rate improvement itself. We’ve added the following implications to the discussion following this theorem:
  • The win rate improvement from preference annotations alone is a function of the starting model.
  • This improvement is bounded for SFT based on properties of this starting model. In other words, SFT on a model’s preferred samples has limits in terms of win rate improvement, no matter how many annotations of preference pairs are collected. This is also true for RLHF or DPO with a fixed, non-zero beta.
  • The result characterizes a particular form of diversity (namely, in preference) that is important for self-improvement. On one extreme, if a model only ever outputs responses that are equally preferred to each other, no improvement is possible. The more diverse in preference the starting model, the more improvement possible.

3. [Furthermore, Figure 1 could be clearer. This figure would be significantly improved with guidance on how to interpret it and by highlighting any specific conclusions drawn from it. I was surprised that Figure 1 was not discussed in the results section; perhaps added explanation there would be helpful.]

• Good point. Figure 1 is meant to show in a toy example how the target distributions of different objectives vary. We’ve added more discussion around the figure, but to make room for other more important text (e.g., more sign-posting around takeaways), we’ve decided to move it and the relevant discussion to the appendix.

4. [What practical insights can we draw from the provided closed-form solutions for win-rate improvement for the DWRO-KL objectives?]

• Thanks for the question. The primary insight is that the introduction of the KL regularization means that the win rate possible must be a function of the support of the starting model; we’ve formalized this as Corollary 1 in the updated paper. Concretely, the best these objectives can do theoretically is defined by how well the best response under the original model can do.

Thanks again for your feedback and questions. If you believe we have sufficiently addressed them, we would really appreciate it if you considered raising your score.

Thank you for your review. We wish to highlight that our paper is not a methods paper but rather an analysis paper. We have updated the text to further emphasize that point, and we hope you will reconsider your review in light of this. Responding to your individual comments below:

1. [The major contribution, DWRO, is not new, as it has already been proposed (see Equation 6, the preference optimization objective, in [1]). The function Ψ serves the same role as h. Additionally, the relationship between the objective and RLHF has also been discussed in [1].]

• DWRO is not the major contribution of this paper, nor is its relationship to RLHF. As mentioned in the related work, we agree with the reviewer that DWRO-KL is equivalent to $\Psi$-PO from [1]. We also cite [1] for the derivation of the relationship between RLHF and DWRO-KL in the RLHF analysis section. Instead, the primary contribution of this work is to provide a mathematical justification for win rate as the central object in preference learning and re-characterize existing methods from this vantage point.

2. [Section 3.3, which discusses SFT, is unclear. In line 313, the authors state that “SFT is analogous to estimating the distribution $p(y_1∣x, \ell=1)$ but do not provide justification. The quantity $p(y_1∣x, \ell=1)$ is confusing, as it’s unclear how another response, $y_0$, is sampled. Is it sampled from a fixed reference distribution? More details are needed on how SFT is performed on preferred samples. Furthermore, the justification for why Eq. 16 is the target distribution for SFT is lacking. The authors show equivalence between Eq. 15 and Eq. 16 but do not explain the derivation of Eq. 16.]

• Given a joint distribution over $x, y_0, y_1, \ell$ where $\ell=1$ denotes that $y_1$ is preferred, the distribution over the preferred sample in a pair is by definition the conditional distribution $p(y_1|x, \ell=1)$. $y_0$ is sampled from the distribution $p(y_0|x)$ that is part of the joint sampling distribution of $x, y_0, y_1, \ell$. Performing SFT on the preferred sample in a pair targets $p(y_1|x, \ell=1)$ via maximum likelihood estimation. Eq 16 (now Eq 12) follows directly from the RHS of the numerator in Eq 14 (now Eq 13).

3. [Experimental details are also insufficient. There is no explanation of how the reference model is trained—does it use the same preference dataset as the models with DWRO variants?]

• The reference model is the starting point for all the DWRO runs. Thus, the win rate is computed between the model before and after DWRO.

4. [In addition, details on constructing the preference pairs for training are missing.]

• Appendix G describes how the datasets are processed to be preference pairs.

5. [The purpose of the experiments is also unclear. If the goal is to compare different DWRO objectives, it would be more meaningful to have them compete directly with each other, rather than against a fixed reference model.]

• We are interested in evaluating the methods based on the objective they seek to approximate. In this case, that objective is maximizing win rate over their starting point, which is why we evaluate the methods this way.

6. [Furthermore, the judge model, Pythia-2.8b, is lightweight, whereas higher-standard models like GPT-4 are typically used as judges for evaluating win rates.]

• Our analysis is applicable for any preference environment defined by any arbitrary preference classifier $p(\ell = 1 | x, y_0 , y_1)$. We agree that GPT-4 is typically a better judge for approximating the average human preference, but it is only a proxy. We train our own judge model and make it the oracle preference so that we have easy access to the truth for analysis. As mentioned in section 4.1, “To simulate a preference environment, we train an oracle judge model per dataset to estimate $p(\ell = 1 | x, y_0 , y_1)$ and relabel the preference annotations in the dataset using this judge model.”

7. [The overall presentation requires improvement. Section 2.2 all talks about the evaluation function $\phi$, but it does not appear in later sections. Additionally, the notation for the anchor distribution is inconsistent, with q used in some expressions and p in others, which makes the paper difficult to follow.]

• We do not use $\phi$ in later sections because we have no need to discuss an evaluation function in general once we’ve determined that the evaluation function must be win rate. We have updated the notation for the anchor distribution to be consistent–thanks for the feedback!

Thank you for your comments, which helped us realize that there were parts of the paper we could clarify to avoid the possibility of confusion. We hope that, after reading this response and looking over the updated paper, you will reconsider your review.

Thank you for your review and for recognizing the formal proof about win rate as a strength of the paper. As we mentioned in the overall response, however, we believe that there may have been some confusion over what the paper claims as its core contributions. We hope our response to each of your points below, as well as our updated paper, will provide more clarity:

1. [The writing is poor. Especially for the contribution is poor, first, the authors should use mathematical formulas to accurately show their novelty instead of confusing words; second, the contribution should be split into fewer points, like 3 to 4. Now 7 points are too many.]

• Thanks for the feedback. We have updated the introduction to include only the primary contributions of the paper, and we have made sure to avoid any imprecise language.

2. [The novelty of this work is little. Although the authors claim that they are the first to consider the win rate, the win rate is basically the same as the general preference, which is already been studied by a series work of Nash learning, such as Munos 484 et al. (2023), [1], [2], [3].]

• First, we do not claim that we are the first to consider win rate. As mentioned in the related work: “While win rate is already a central evaluation in preference learning (Li et al., 2023; Zheng et al., 2024), in our work we underscore that it is the only evaluation grounded in the sampling distribution itself.” Moreover, we already cite Munos et al in the related work and have added the other contemporary works as well. More fundamentally, however, our paper is not a methods paper; instead, the novelty of our work lies in our win-rate centric analysis and its resulting takeaways.

3. [Based on the theory of Munos 484 et al. (2023) and [3], fixing one response and optimizing the other may not be efficient and achieve the best performance since one iteration will not lead to the Nash equilibrium. Hence, the authors are suggested to compare the performance of their method to Munos 484 et al. (2023), and [3].]

• The point of our paper is not to propose a new method, but rather to understand the preference learning landscape through the lens of directly optimizing for win rate. We do not suggest that one should choose one variant of direct win rate optimization over another–in fact, our experiments (updated to consider additional settings) show that the particular variant matters less than the success of optimization for a given run. The insight that preference learning could benefit from improved strategies for optimization is generally applicable, including for the NLHF method proposed in Munos et al 2023.

4. [Line 124: If h is a function, what is its' mapping: from what space to what space?]

• $h$ maps from [0, 1] to $\mathbb{R}$.

5. [Line 354: the notation for the variance is confusing and I cannot find a clarification. To which variance the authors are taking variance?]

• The variance is with respect to the distribution $p(y_1|x)$, as denoted in the subscript. This variance term can be understood procedurally as follows: for each response $y_1$ from $p(y_1|x)$, we compute its average preference over $p(y_0|x)$ by computing preference probabilities for each pair and averaging by the prevalence of $y_0$ under the initial model. Then, we take the variance across all $y_1$s of these average preferences. In short, it is the variance in average preference across responses.

6. [Some typos]

• Thanks, fixed!

We appreciate your feedback, which highlighted parts of the paper that we could clarify. In light of our response and paper update, we hope you will consider raising your score.