Self-Play Preference Optimization for Language Model Alignment

We would like to thank the reviewers for their thoughtful feedback and valuable suggestions. Below, we address each of the raised questions in detail.

---

Q1: Missing Analysis of Hyperparameter η (β) in [1]): [2] demonstrated a clear relationship between the KL divergence with the original policy and the quality of responses. Comparing algorithms with different KL divergences is incorrect, as one may have lower response quality due to a lesser KL divergence with the original policy. Quality can be improved by adjusting the hyperparameter η(β). The results would be more convincing if presented as a Pareto front, as done by Rame et al. [3].

A1: Thanks for your suggestion, we will add discussion on [3]. We have uploaded a new figure in the updated version (Section D.4). We plot the AE2.0 LC win rate vs. the KL divergence to Mistral-7B-Instruct-v0.2, for each model checkpoint we reported in the paper, as well as Mistral-SPPO with different learning rate(1e-1, 1e-2, 1e-4; 1e-3 reported in the original submission). Notably, SPPO has the following results (for more results please see Section D.4):

Our understanding is that the reviewer may suspect that the high alignment metric of SPPO comes from a larger KL budget. The plot (and table here) shows that this is not the case. Compared with other methods, the three iterations of SPPO do not incur more KL divergence but have shown a steady win rate increase. For the last iteration, it has a higher alignment metric, but less KL divergence compared with the iterative DPO by the Snorkel team. From these experiments, we also find that $\beta=1e-2$ gives a better trade-off between KL and win rate (almost half the KL but close win rate).

---

Q2: Absence of Comparison with Other Game-Theory-Based Methods: Since the work relies on Nash equilibrium and cites other methods grounded in the same theory, it is logical to compare the proposed method with the Nash-MD [4] algorithm, as well as Nash-EMA [4] and DNO [5] (For Nash-MD and Nash-EMA the convergence was proven by the last iteration rather than on average).

A2: Thanks for your suggestion. We have added more discussion on the comparison with Nash-MD/Nash-EMA and DNO in the updated version (Section A). Here are some key points:

• DNO: DNO [5] requires a gold response generated directly from GPT-4 and asks GPT-4 as a judge to assign a numerical score to each response for preference pair construction, which is a stronger signal than the preference feedback used in our work. This is quite a different task setting: access to a stronger language model’s generation motivates model distillation rather than model alignment, where only preference feedback is available. It would be hard to attribute whether the preference signal or the gold response helped alignment.

  If we examine DNO without a gold response, then the practical version of DNO uses the DPO loss and the algorithm effectively becomes iterative DPO. It is therefore difficult to make a fair comparison between DNO and SPPO in experiments.

• Nash-MD/Nash-EMA: We agree that these methods are among the first to consider alignment toward Nash equilibrium. The main difficulty in experimental comparison is that they may require calling $K^2$ PairRM comparisons to estimate the win rate as a reward on the fly. Unlike directly calling a reward model for each trajectory (prompt + response), the on-the-fly $K^2$ PairRM calls are computationally much heavier and may stall the training process due to their huge compute overhead. We are not aware of any open-sourced implementation that can perform real-time on-policy sampling and win rate calculation for 7B models.

  We have emphasized the contribution of Nash-MD/EMA in the updated version and discussed their advantages/disadvantages. While Nash-MD enjoys last-iterate convergence, the theorem requires exact sampling from the mixture distribution $\pi_t(y|x)^{1-\alpha} \pi_{\text{ref}}^{\alpha}(y|x)$. This can only be done exactly in the bandit setting. For sequential generation, this mixture distribution is difficult to sample from and is approximated by token-level distribution mixing.

We would like to thank the reviewers for their thoughtful feedback and valuable suggestions. Below, we address each of the raised questions in detail.

---

Q1: The paper does not suggest any analysis of how SPPO-trained models are aligned to PairPM.

A1: Thank you for your suggestion. In fact, we have reported the PairRM win rates from different iterations in Appendix D.3 (Figure 3). These results show that the PairRM win rate against the initial reference model (Mistral-7b-instruct-v0.2) consistently increases throughout the three iterations, demonstrating the effectiveness of SPPO.

---

Q2: The choice of total training epochs of 18.

A2: We appreciate the question. We have updated the description in the paper for clarity. The 18 epochs were not fully utilized in actual training. We observed that stopping after the first iteration yielded the best performance, likely due to avoiding overfitting. The choice of 18 epochs was primarily for defining the linear-decay learning rate schedule.

More specifically, the learning rate at step $t$ is computed as:

$$\eta(t) = \bigg(1- \frac{t}{T} \bigg) \eta_0.$$

Here $T=N*E$ is the planned total number of steps, where we have $N$ steps per epoch and $E$ epochs. We set $E=18$ but stop after the 1st epoch, and $18$ is only used when calculating the learning rate.

---

Q3: Terminology choices and hyperparameter statements.

A3: Thank you for your feedback. We have revised the paper to improve clarity:

• "Iteration" and "round" are used interchangeably and refer to the high-level updates indexed by $t = 1, 2, \dots, T$, meaning there are $T$ total rounds/iterations of approximating the exponential weight update rule. To avoid confusion, we have standardized the terminology to "iteration" throughout the paper.

• "Epoch" describes the number of times each data sample is used within a single round of training. For example, 2 epochs indicate that each sample is utilized twice.

• Regarding training hyperparameters, we provide the following details for DPO and IPO experiments:

  • Iterative DPO:
    • Rounds 1, 2, 3: $\beta = 0.1$, learning rate = $5e-7$, trained for 2 epochs per round.
  • Iterative IPO:
    • Rounds 1, 2, 3: $\beta = 0.01$, learning rate = $5e-7$, trained for 1 epochs per round.

These details will also be updated in the appendix.

---

Q4: Would SPPO also be prone to the length bias?

A4: We believe SPPO is less susceptible to length bias compared to other methods due to its loss function design, which constrains how far the log ratio can deviate. According to the proposed SPPO objective, $\log \frac{\pi_{\theta}(y_w|x)}{\pi_{\text{ref}}(y_w|x)}$ will at most increase to $\eta/2$, so the length will not increase arbitrarily. In contrast, translation-invariant loss functions such as DPO and IPO cannot control the changes in the log ratio effectively but only the gaps between the winner and loser. Therefore, even when length bias exists in the dataset or preference oracle, SPPO’s loss formulation helps mitigate its impact.

---

Q5: How would better reward models/preference models impact SPPO’s performance?

A5: SPPO is designed to work with any arbitrary preference model, including cyclic preference relations or Bradley-Terry (BT) models. Under the BT model, SPPO aims to maximize the win rate as defined by the BT model. Specifically, the win rate between two responses $y_i$ and $y_j$ can be computed as:

$$P(y_i \succ y_j | x) = \text{sigmoid}(r(x, y_i) - r(x, y_j))$$

Similarly, $P(y \succ \pi_t | x)$ can be estimated in the same way proposed in the paper. In theory, if the preference model follows the BT assumption, SPPO will converge to a policy that concentrates on responses with the highest BT reward.

A stronger preference oracle—whether BT-based or not—will naturally enable SPPO to obtain stronger policies. For example, using a stronger Llama-3-8B BT reward model from [1] (91.1% accuracy on Reward-Bench) instead of PairRM led to better results:

Specifically, Llama-3-8B-Instruct with the 8B BT reward model achieved a 40.55% length-controlled win rate on AlpacaEval 2.0, outperforming the same model guided by PairRM (38.77%).

---

References:
[1] General Preference Modeling with Preference Representations for Aligning Language Models (https://arxiv.org/abs/2410.02197)

We would like to thank the reviewers for their detailed feedback and valuable suggestions. Below, we address each of the raised questions.

---

Q1: The annotation step in line 4 of Algorithm 1 requires K^2 calls for preference model, which should be time-consuming.

A1: We agree that $O(K^2)$ calls are required. But calculating the preference score actually only requires one-step inference (rather than sequential generation) and can be performed in a batched and parallel way. So the time consumption can be alleviated given more computing resources. On the other hand, there are new techniques developed recently so that calculating $K^2$ pairwise preferences only requires $O(K)$ compute and can be as fast as a BT reward model (see [2]).

We want to reiterate that if we assume there are nuances in human preference relations (such as non-transitivity), it is statistically inevitable to query the full preference matrix and pay $O(K^2)$ comparisons (see [1]).

---

Q2: The preference model should be also updated together with policy model in order to capture the probability distribution of evolving policy.

A2: We would like to thank the reviewer for their insightful suggestion that the preference model can evolve jointly with the LLM policy. We believe this is an important further direction. In this work, the preference model is assumed to be the ground truth we aim to align with. Ideally, the pairwise preference should come from the human annotators, which is considered static for any given pair, so it does not have to evolve along with the LLM policy.

In practice, we have to compromise, replacing humans with LLM-based annotators such as GPT-4, or PairRM in this case. We used PairRM mainly because it was state-of-the-art when the experiment was initially conducted, and our baseline Snorkel used PairRM and DPO with the same training data. For a fair comparison, we use PairRM as the ground truth.

---

Q3: I believe the preference model, here is PairRM, is critical for the final performance of SPPO, it will be good if the authors also run experiments with other models.

A3: Thanks for your suggestion. We agree that it is worth examining SPPO with stronger reward models or preference models. We run SPPO with a stronger BT reward model and a general preference model from [2]:

• A strong Llama-3-8B BT reward model with 91.1% accuracy on Reward-Bench.
• A stronger Llama-3-8B general preference model with 92.2% accuracy on Reward-Bench (ranked 4th within 8B models, best 8B model is 93.1%).

The results are:

With these different reward/preference models, SPPO has shown steady improvement and high performance on benchmarks. Other conditions unchanged, Llama-3-8B-Instruct + 8B BT reward model can achieve a 40.55% length-controlled win rate on AlpacaEval 2.0, while Llama-3-8B-Instruct + PairRM has 38.77%.

---

Q4: The annotation step in line 4 of Algorithm 1 requires K^2 calls for preference model, then, the authors select the highest and lowest PairRM scores and winning and losing responses. This setting is used also for DPO and IPO? If so, the only differences come from the loss function definition?

A4: That is right. Selecting the winner and loser to form a pair is adopted by the baseline method Snorkel. This setting is also used in our own implementation of SPPO, iterative DPO and IPO for a fair comparison. The only difference is the loss function definition, and for SPPO we utilize the numerical win rate.

We would like to thank the reviewers for their thoughtful feedback and valuable suggestions. Below, we address each of the raised questions in detail.

---

Q1: The paper does not include a direct comparison with conventional RLHF methods, such as using a BT-formulated reward model (RM) and PPO training.

A1: Thanks for the suggestion. We have conducted the following PPO+BT RM baseline:

Policy: Mistral-7B-Instruct-v0.2 Value function: Mistral-7B-Instruct-v0.2 Reward Model: weqweasdas/RM-Mistral-7B (80.4% accuracy on Reward-Bench) Prompt Dataset: Ultrafeedback prompt.

We use the standard implementation from the Huggingface trl library. The results are:

It is difficult to establish a fair comparison between SPPO (paired with a preference model) and PPO (paired with a reward model) because the quality of the preference or reward feedback significantly affects the performance of the learned LLM policy. For example, PPO here used a 7B Mistral-based reward model and SPPO used a 0.4B sequential classification model PairRM.

We would also like to highlight the findings from [1], which demonstrate that certain mechanisms used in PPO—such as gradient clipping, value functions, and token reward redistribution—are unnecessary for LLM alignment. Instead, REINFORCE suffices. As discussed in Section 3.3, SPPO can be viewed as a multi-round, semi-online REINFORCE method. Within each round, SPPO essentially follows the policy gradient of REINFORCE. The key difference lies in that the reward (win rate) used by SPPO changes dynamically across rounds. We will add discussion on the connection with and difference from the conventional RLHF.

[1] Admadian et al., Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (2024).

---

Q2: Lack of comparison with similar methods, such as Nash Preference Optimization (NPO). The paper lacks a comparison with conceptually similar work, such as Nash Preference Optimization (NPO). Without this comparison, it is difficult to assess the practical or theoretical advantages offered by SPPO.

A2: We are not sure which specific paper is being referred to here. If you are referring to [2], our paper and DNO [2] are actually concurrent works, and we have discussed it in detail in Appendix A. The practical version of DNO employed the DPO loss, and the algorithm can be seen as iterative DPO. Furthermore, DNO requires a gold response directly generated by GPT-4, which provides a stronger signal compared to the preference feedback used in our work. DNO will eventually become iterative DPO in our setting for a fair comparison.

[2] Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences.

---