$\alpha$-DPO: Adaptive Reward Margin is What Direct Preference Optimization Needs

3. A Theoretical Justification via Reward Difference Estimation

The original DPO objective maximizes the probability $p^*(y\_{\mathrm{win}} \succ y\_{\mathrm{lose}} \mid x)$, which, under the Bradley-Terry (BT) model, can be formulated as maximizing the following log-likelihood: $\max \log \sigma(r\_\theta(x, y\_{\mathrm{win}}) - r\_\theta(x, y\_{\mathrm{lose}})),$ where $\sigma$ is the sigmoid function. Defining the reward difference $\delta = r\_\theta(x, y\_{\mathrm{win}}) - r\_\theta(x, y\_{\mathrm{lose}})$, the optimization objective simplifies to: $\max \log \sigma(\delta) := \mathcal{T}(\delta).$ Suppose we start with an initial $\delta\_0$. A possible improved estimate $\delta\_1$ that maximizes $\mathcal{T}(\delta)$ is given by:

$\delta\_1 = \delta\_0 + \alpha \nabla \mathcal{T}(\delta\_0) = \delta\_0 + \alpha G(\sigma(-\delta\_0)),$ where $G$ is a monotonic function preserving the direction of $\sigma(-\delta\_0)$. This implies that for sufficient samll$\alpha$, $\delta\_1$yields higher likelihood and can be viewed as a more optimistic estimation of the reward difference than $\delta\_0$.
In the case of DPO, the reward difference $\delta$ can be reformulated as: $\delta\_0 = \frac{\log \pi\_\theta(y\_{\mathrm{win}} \mid x)}{\log \pi\_{\text{ref}}(y\_{\mathrm{win}} \mid x)} - \frac{\log \pi\_\theta(y\_{\mathrm{lose}} \mid x)}{\log \pi\_{\text{ref}}(y\_{\mathrm{lose}} \mid x)}.$ Updating $\delta$ using $\delta\_1$ leads to:

\delta\_1 &= \frac{\log \pi\_{\theta}(y\_{\mathrm{win}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{win}}|x)} - \frac{\log \pi\_{\theta}(y\_{\mathrm{lose}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{lose}}|x)} + \alpha G\left(\sigma\left(- \frac{\log \pi\_{\theta}(y\_{\mathrm{win}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{win}}|x)} + \frac{\log \pi\_{\theta}(y\_{\mathrm{lose}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{lose}}|x)}\right) \right)\notag \\\\
& = \frac{\log \pi\_{\theta}(y\_{\mathrm{win}}|x)}{\log \pi\_{\theta}(y\_{\mathrm{lose}}|x)} + \left(\gamma - \frac{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{win}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{lose}}|x)}\right) -\left[\gamma -\alpha G\left(\sigma\left(-\frac{\log \pi\_{\theta}(y\_{\mathrm{win}}|x)\log \pi\_{\mathrm{ref}}(y\_{\mathrm{lose}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{win}}|x)\log \pi\_{\theta}(y\_{\mathrm{lose}}|x)}\right)\right)\right]

If we take the assumption that $\gamma \approx \log \pi\_{\mathrm{ref}}(y\_{\mathrm{win}}|x) - \log \pi\_{\mathrm{ref}}(y\_{\mathrm{lose}}|x)$ holds per uniform assumption, then above equation implies $\delta\_1 \approx \frac{\log \pi\_{\theta}(y\_{\mathrm{win}}|x)}{\log \pi\_{\theta}(y\_{\mathrm{lose}}|x)} -\left[\gamma -\alpha G\left(\sigma\left(-\frac{\log \pi\_{\theta}(y\_{\mathrm{win}}|x)\log \pi\_{\mathrm{ref}}(y\_{\mathrm{lose}}|x)}{\log \pi\_{\mathrm{ref}}(y\_{\mathrm{win}}|x)\log \pi\_{\theta}(y\_{\mathrm{lose}}|x)}\right)\right)\right]$ The remaining part follows by choosing$G(\cdot)\propto \sigma^{-1}(\cdot)$while maintaining monotonicity.

In conclusion, our proposed method can be viewed as a new reward difference estimator (i.e.,$\delta\_1$ instead $\delta\_0$), which is more optimistic than its vanilla counterpart under the uniform assumption.

Q5: I wonder when the algorithm becomes DPO?

A5: Currently, the $\alpha$-DPO algorithm cannot be transformed into DPO merely through parameter adjustments, similar to how SimPO cannot be converted to DPO by altering $\gamma$. However, I believe this topic presents significant promise, allowing us to propose a more generalized formulation: $\hat{\pi}\_{\text{ref}}(\cdot|x)= U(\cdot|x) \pi\_\theta^{\alpha\_1}(\cdot|x) \pi\_{\text{ref}}^{\alpha\_2}(\cdot|x)$ This expression encompasses several specific cases:

• Setting $\alpha\_1=0$ and $\alpha\_2=1$ results in the DPO formulation with $\gamma$.
• Setting $\alpha\_1=\alpha\_2=0$ recovers the SimPO expression.
• Setting $\alpha\_1=\alpha, \alpha\_2=-\alpha$ yields the $\alpha$-DPO formulation.

We hope these additional clarifications address your concerns comprehensively. Thank you again for your thoughtful review and the opportunity to improve our work.

Q4: The transition between eq(15) and eq(16) is not convincing.

A4: To elucidate the rationale behind this transition, we offer the following explanations.

1. Necessity of the Transformation:

When the condition $\pi\_{\theta\_{\text{old}}}(y\_l|x) > \pi\_{\text{ref}}(y\_l|x)$ arises—potentially due to the overrepresentation of less preferred responses in $\pi\_{\theta\_{\text{old}}}$—the importance weight $w(y\_w, y\_l | x)$ becomes disproportionately large. This disproportionate weight amplifies the impact of less preferred responses within the expectation, which could consequently result in suboptimal updates to $\pi\_\theta$. To mitigate this issue, an adjustment is made by placing $\pi\_{\theta\_{\text{old}}}(y\_l|x)$ in the denominator, thereby inversely scaling the weight associated with the overrepresented less preferred responses $y\_l$. This adjustment serves to diminish the undue influence of these responses that have higher probabilities under $\pi\_{\theta\_{\text{old}}}$ compared to $\pi\_{\text{ref}}$.

2. Supporting the Transition Between Equations (15) and (16) Through Simulation of Online DPO Training Steps:

Consider the scenario at step $t$ of the optimization process, where we refine $(x,y\_w,y\_l)$. In this scenario, the updated model $\pi\_\theta^{t+1}$ assigns a higher score to $y\_w$ and a lower score to $y\_l$. Consequently, when new samples are generated using $\pi\_\theta^{t+1}$, the probability of generating $y\_w$ in response to the same prompt $x$ increases, while the probability of generating $y\_l$ decreases. This aligns with the corrected importance weight direction, given by: $w\_{\text{corr}}(y\_w, y\_l|x) = \frac{\pi\_{\theta\_{\text{old}}}(y\_w|x)}{\pi\_{\text{ref}}(y\_w|x)} \cdot \frac{\pi\_{\text{ref}}(y\_l|x)}{\pi\_{\theta\_{\text{old}}}(y\_l|x)}$ We attribute the perceived inconsistency to a lack of rigorous symbol usage, which can be clarified by considering the following conditional distributions as mentioned in A1:

• For the response with the highest score $y\_w$, it adheres to the conditional distribution: $\pi(y \mid x, \text{score}(y) = \max(\{\text{score}(y\_1), \ldots, \text{score}(y\_5)\}))\approx\pi(win|x),$
• For the response with the lowest score $y\_l$, it adheres to the conditional distribution: $\pi(y \mid x, \text{score}(y) = \min(\{\text{score}(y\_1), \ldots, \text{score}(y\_5)\}))\approx \pi(lose|x)$

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address your comments in detail.

Q1: It seems to me that this method is just a one-step extension to the SimPO to make the reference model a tunable mixture of the uniform distribution and original reference model.

A1: While $\alpha$-DPO has a mixture form, we do not view this as a weakness. Recent work, such as [1], highlights the importance of the reference model and its impact on performance, with SimPO introducing the concept of being reference-free. Our work aims to explore the optimal reference model needed for offline preference learning without additional gold RM models. We hope to provide new insights, for example, by integrating online algorithms and KL Divergence control.

[1] Liu et al. 2024. Understanding Reference Policies in Direct Preference Optimization. arXiv preprint arXiv:2407.13709

Q2: In online DPO, another oracle needs to be called to compare between $y\_w$ and $y\_l$ and generate the preference label.

A2:Thank you for your suggestion. We acknowledge the issue of inconsistency with the notation, and we will revise this section in the draft for clarity. The corrected description of the online process is as follows:

Let's revisit the acquisition method for these samples (e.g., using princeton-nlp/llama3-ultrafeedback dataset): For each prompt $x$, we generate 5 responses using the SFT model with a sampling temperature of 0.8. We then score these responses with llm-blender/PairRM and select the highest-scoring one as $y\_w$ and the lowest-scoring one as $y\_l$. For the highest-scoring sample $y\_w$, it follows a new conditional distribution:

$\pi(y \mid x, \text{score}(y) = \max(\{\text{score}(y\_1), \ldots, \text{score}(y\_5)\}))\approx\pi(win|x),$ Where the last part uses the fact that for a given x the generative model has a uniform chance to generate all possible y. Similarly, for the lowest-scoring sample $y\_l$, it follows: $\pi(y \mid x, \text{score}(y) = \min(\{\text{score}(y\_1), \ldots, \text{score}(y\_5)\}))\approx \pi(lose|x)$ We have ensured that the notation is now consistent and precise across the descriptions.

For online DPO, ideally, we would utilize an oracle reward model to evaluate each sampled instance. However, this approach incurs significant computational costs. Therefore, we aim to approximate this effect through importance sampling, as described in Equation 14.

Q3: An online DPO justification might not be convincing enough to justify the offline DPO methods

A3: It is well known that preference optimization with online ingredients, particularly methods like online AI feedback (OAIF) [1] and self-play [2,3], enhances model alignment by generating new data during training. However, due to the computation cost sample regeneration and training stability issues, the pure online type preference optimization methods are not well applied in industrial practice. Based on this observation, we thus proposed to study an interesting research topic, i.e., can we mimic the online feature in the classic offline setting? Our theoretical analysis is trying to close the gap with the importance sampling trick.

In particular, the underlying logic here first defines the expression for online SimPO, which is characterized by the continuous updating of sampled data during training. In this process, the data transitions from the offline set $(y\_w, y\_l) \in D$ to the online set $(y\_w, y\_l) \in \pi\_\theta$. This operation can be viewed as an important sampling method. Interestingly, this key technique of importance sampling aligns with the optimization direction of $\alpha$-DPO loss, with both approaches converging in the scenario where $\alpha \to 0$. Consequently, Lemma 4.2 aims to establish a connection between $\alpha$-DPO and the Online SimPO loss, thereby facilitating the integration of these concepts.

[1] Guo et al. Direct Language Model Alignment from Online AI Feedback. CoRR abs/2402.04792 (2024)

[2] Chen et al. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. ICML 2024.

[3] Wu et al. Self-Play Preference Optimization for Language Model Alignment. CoRR abs/2405.00675 (2024)

We greatly appreciate the opportunity to engage in a substantive dialogue with you, aimed at further improving the quality and clarity of our work. Below, we provide a detailed summary addressing your key concerns. We hope that this explanation will sufficiently address your questions and merit your endorsement.

1. The proposed reference model is a tunable mixture of the uniform distribution and orginal refernece model.

In our response A1, we clarify that this formulation is neither a weakness nor merely a trivial extension achieved by "tuning the parameter in a superset."

First, let us clarify that the primary motivation for proposing the reference policy in Eq. (8) arises from the limitations of the conventional $\pi\_{\text{ref}} = \pi\_{\text{SFT}}$ setup, which often exhibits significant randomness and errors. Addressing this limitation is a core objective of our work, a perspective that is similarly reflected in SimPO [1] (Section 3.2 and Figure 4b) and Liu et al. [2] (Section 5).

Our goal is thus to design a more robust and effective reference policy, as expressed in Eq. (8). To provide additional context, we elaborate on the following key points:

• Alignment with online SimPO optimization. Preference optimization with online elements (e.g., online AI feedback [3] and self-play [4][5]) is widely acknowledged to improve model alignment by generating new data during training. However, the high computational cost and instability of purely online methods limit their practicality in industrial applications. Our work explores whether online-like behavior can be emulated in a classical offline setting. Theoretical analysis based on importance sampling is provided to bridge this gap.

• Structural advantages of $r(x, y\_w) - r(x, y\_l) - \delta$. A major contribution of token-level DPO is the introduction of the form $r(x, y\_w) - r(x, y\_l) - \delta$, which is similar to DPO with an offset and helps control the KL divergence. Our $\alpha$-DPO framework leverages the form $r(x, y\_w) - r(x, y\_l) - M$, which enhances performance by using $M$ instead of $\delta$. Appendix Table 6 provides empirical evidence, showing the performance advantages of $\alpha$-DPO over TDPO.

• Mitigating the impact of label flipping noise. Within the SimPO framework, the gradient is expressed as:

  $$\nabla\_\theta \mathcal{L}\_{\mathrm{SimPO}}(\pi\_\theta) = -\beta \mathbb{E}\_{(x, y\_w, y\_l) \sim \mathcal{D}} \left[ s\_\theta \left( \frac{1}{|y\_w|} \nabla\_\theta \log \pi\_\theta(y\_w|x) - \frac{1}{|y\_l|} \nabla\_\theta \log \pi\_\theta(y\_l|x) \right) \right],$$

  where

  $$s\_\theta = \sigma \left( \frac{\beta}{|y\_l|} \log \pi\_\theta(y\_l|x) - \frac{\beta}{|y\_w|} \log \pi\_\theta(y\_w|x) + \gamma \right).$$

  This formulation amplifies weights when the reward estimate is inaccurate (e.g., $\log \pi\_\theta(y\_l|x) - \log \pi\_\theta(y\_w|x) > 0$), increasing noise in gradients. In contrast, $\alpha$-DPO introduces an additional term:

  $$s\_\theta = \sigma \left( \frac{\beta}{|y\_l|} \log \pi\_\theta(y\_l|x) - \frac{\beta}{|y\_w|} \log \pi\_\theta(y\_w|x) + \gamma + \alpha M(x, y\_w, y\_l) \right).$$

  This term increases the weight when the reward estimate is accurate and decreases it when the reward estimate is inaccurate, mitigating noise amplification.

• Improving reference policy quality. This work directly addresses the unreliability of reference policies (Section 3.1). By integrating the policy model into the reference model's design, we enhance the quality of the reference model and improve fine-tuning performance. Similar ideas have been explored in recent works [2][6], highlighting the broader relevance of this approach.

References:
[1] Meng et al. (2024): Simpo: Simple preference optimization with a reference-free reward. NeurIPS 2024.
[2] Liu et al. (2024): Understanding Reference Policies in Direct Preference Optimization. arXiv preprint arXiv:2407.13709.
[3] Guo et al. (2024): Direct Language Model Alignment from Online AI Feedback. CoRR abs/2402.04792.
[4] Chen et al. (2024): Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. ICML 2024.
[5] Wu et al. (2024): Self-Play Preference Optimization for Language Model Alignment. CoRR abs/2405.00675.
[6] Gorbatovski et al. (2024): Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656.

2. The significance of theoretical guarantees

In our response A2, we revisited the distributions of $y\_w$ and $y\_l$ and revised the original text to ensure consistency throughout.

In our response A3, we explained the motivation for establishing connections with online algorithms. The core rationale is that online algorithms remain the most effective choices due to their superior performance. However, purely online preference optimization methods are often impractical for industrial applications due to computational cost, sample regeneration, and training stability issues. As a compromise, importance sampling is frequently employed, and similar approaches are widely adopted in the reinforcement learning literature [1][2][3][4].

Finally, we would like to emphasize that SimPO has already demonstrated its effectiveness in various experimental settings. Our method extends SimPO to personalized scenarios, with the parameter $\alpha$ acting as a key control factor. Specifically:

• The choice of $\alpha$ balances two competing objectives. When $\alpha = 0$, the method reduces to SimPO, where all training samples are assigned the same target margin.
• As $\alpha$ increases, the model places greater emphasis on samples with accurate reward estimates.
• However, excessively large values of $\alpha$ cause the model to focus solely on a few samples with the most accurate reward estimates, neglecting the contributions of other samples, which can hinder the training process.

This trade-off highlights the importance of selecting an appropriate $\alpha$ to achieve a balance between conservativeness and aggressiveness.

[1] Sergey Levine et al.: "Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems," CoRR abs/2005.01643 (2020).
[2] Alberto Maria Metelli et al.: "Policy Optimization via Importance Sampling," NeurIPS 2018.
[3] Tengyang Xie et al.: "Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling," NeurIPS 2019.
[4] Philip J. Ball et al.: "Efficient Online Reinforcement Learning with Offline Data," ICML 2023.

3. The transition between eq(15) and eq(16) is not convincing.

In our response A4, we provided a detailed analysis of the necessity and rationale behind this transition from multiple perspectives.

Your constructive feedback has been invaluable, and we are committed to leveraging this discussion to improve our work. Thank you again for the opportunity to address your questions and for considering our responses.

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address your comments in detail.

Q1: If LC and raw WR values come from different α values, then the results are slightly misleading since it's not just one model being used for comparison against the benchmarks.

A1: Thank you for your insightful suggestion. We would like to clarify that all LC and raw WR values reported in our experiments were obtained using a fixed $\alpha$ value across all benchmarks.

We observed that introducing $\alpha$-DPO leads to performance improvements across all benchmarks, which maintains the state-of-the-art performance, albeit with varying degrees of relative improvement. We attribute this observational phenomenon to the current limitations in benchmark evaluation methods, possibly due to the inconsistencies introduced by GPT-4's judgment as a metric.

This trend is not unique to $\alpha$-DPO; it also occurs with other methods, meaning each method has different rankings depending on the benchmark. We believe that developing a truly rigorous and effective benchmark represents an intriguing future research direction.

We hope these additional clarifications address your concerns comprehensively. Thank you again for your thoughtful review and the opportunity to improve our work.

Q4: First, relating the objective to the online SimPO loss is not meaningful, as the online SimPO loss itself lacks theoretical guarantees.

A4: It is well known that preference optimization with online ingredients, particularly methods like online AI feedback (OAIF) [1] and self-play [2,3], enhances model alignment by generating new data during training. However, due to the computation cost sample regeneration and training stability issues, the pure online type preference optimization methods are not well applied in industrial practice. Based on this observation, we thus proposed to study an interesting research topic, i.e., can we mimic the online feature in the classic offline setting? Our theoretical analysis is trying to close the gap with the importance sampling trick.

In particular, the underlying logic here first defines the expression for online SimPO, which is characterized by the continuous updating of sampled data during training. In this process, the data transitions from the offline set $(y\_w, y\_l) \in D$ to the online set $(y\_w, y\_l) \in \pi\_\theta$. This operation can be viewed as an important sampling method. Interestingly, this key technique of importance sampling aligns with the optimization direction of $\alpha$-DPO loss, with both approaches converging in the scenario where $\alpha \to 0$. Consequently, Lemma 4.2 aims to establish a connection between $\alpha$-DPO and the Online SimPO loss, thereby facilitating the integration of these concepts.

[1] Guo et al. Direct Language Model Alignment from Online AI Feedback. CoRR abs/2402.04792 (2024)

[2] Chen et al. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. ICML 2024.

[3] Wu et al. Self-Play Preference Optimization for Language Model Alignment. CoRR abs/2405.00675 (2024)

Q5: However, minimizing a lower bound is questionable since the gap between the true value and the lower bound is unknown. Minimizing an upper bound would be more meaningful.

A5: We apologize for any confusion. A more precise statement is needed: For any policy model $\pi\_\theta$ and reference model $\pi\_{\text{ref}}$, there exists a sufficiently small $\alpha$ such that the following inequalities hold: $|\mathcal{L}\_{\text{SimPO}}^{\text{online}}(\pi\_\theta, \pi\_{\text{ref}}) -\mathcal{L}\_{\text{$\alpha$-DPO}}(\pi\_\theta, \pi\_{\text{ref}})|\leq \varepsilon(\alpha),$ where $\varepsilon(\alpha) = \mathbb{E}\_{\pi\_{\text{ref}}}\left[ \alpha |B| \left| \log \sigma(A) - \sigma(A) + 1 \right| \right].$

This result demonstrates that optimizing $\alpha$-DPO provides a tight approximation of online-SimPO, which is crucial for achieving performance improvements. In response to feedback, we have revised Lemma 4.2 and included the supporting proofs. Thank you for your valuable suggestion.

Q6: The experimental improvement is marginal, typically less than 1.5%.

A6: We believe $\alpha$-DPO's improvements are significant. For example, AlpacaEval2 (LC) shows improvements near 6% on most models. Arena-Hard benchmarks also demonstrate over 5% gains, except for gemma2-9b. We conduct a comparison of the relevant results from Table 1, focusing on SimPO and $\alpha$-DPO. Detailed results are provided:

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address your comments in detail.

Q1: First, Equation 8 does not have a normalization factor, and tuning the hyperparameter $\alpha$ very likely results in an invalid distribution.

A1: We recognize that the expression lacks rigor. A more precise formulation is:

$\hat{\pi}\_{\text{ref}}(y|x) \propto U(y|x)\left(\frac{\pi\_\theta(y|x)}{\pi\_{\text{ref}}(y|x)}\right)^\alpha.$ This expression combines the benefits of DPO and SimPO. We have revised the manuscript to reflect this clarification.

Q2: Additionally, it’s unclear why this implicit reference model is necessary instead of using a standard SFT model.

A2: The assumption that the SFT has large errors serves as the motivation for proposing $\alpha$-DPO. This assumption is discussed in Section 3.1, where we highlight the unreliability of the reference policy in DPO.

Q3: The authors consider a special case with $\alpha=1$, incorporating the ratio between the policy model and the reference model, but there is no clear rationale for why such a ratio is required.

A3: The motivation for the proposed reference policy $\hat{\pi}\_{\text{ref}}(y|x)$ can be clarified as follows:

1. Utility Theory Perspective: The proposed $\hat{\pi}\_{\text{ref}}(y|x)$ is designed with the uniform distribution $U(y|x)$ as a baseline. The term $\left(\frac{\pi\_\theta(y|x)}{\pi\_{\text{ref}}(y|x)}\right)^\alpha$ dynamically adjusts the reward margin by balancing contributions from the policy and reference models. This mechanism can be interpreted through the lens of utility theory as relative attractiveness, enabling adaptive instance-specific reward modeling.

2. Gradient Perspective: By introducing $\hat{\pi}\_{\text{ref}}(y|x)$, the framework mitigates the label flipping issues found in DPO or SimPO. In the SimPO framework, the gradient is expressed as: $\nabla\_\theta\mathcal{L}\_{\mathrm{SimPO}}(\pi\_\theta)=-\beta\mathbb{E}\_{(x,y\_w,y\_l)\sim\mathcal{D}}\left[s\_\theta\left(\frac1{|y\_w|}\nabla\_\theta\log\pi\_\theta(y\_w|x)-\frac1{|y\_l|}\nabla\_\theta\log\pi\_\theta(y\_l|x)\right)\right],$ where $s\_\theta=\sigma\left(\frac\beta{|y\_l|}\log\pi\_\theta(y\_l|x)-\frac\beta{|y\_w|}\log\pi\_\theta(y\_w|x)+\gamma\right)$. This formulation may amplify weights when the reward estimate is incorrect. By contrast, under $\alpha$-DPO: $s\_\theta=\sigma\left(\frac\beta{|y\_l|}\log\pi\_\theta(y\_l|x)-\frac\beta{|y\_w|}\log\pi\_\theta(y\_w|x)+\gamma + \alpha M(x,y\_w,y\_l)\right),$ the additional $\alpha M(x,y\_w,y\_l)$ component increases weight when the reward estimate is accurate, ensuring a more robust reward signal.

3. Motivational Core: The central goal of the proposed $\alpha$-DPO is to address the unreliability of the reference policy, as outlined in Section 3.1. By integrating the policy model into the reference model design, the quality of the reference model is enhanced, improving fine-tuning performance. Similar concepts have been explored in recent works [1][2]:

We have also provided a theoretical justification through reward difference estimation. For more details, please refer to our response (A4) to Reviewer 8xwP.

[1] Gorbatovski et al. (2024): Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656.
[2] Liu et al. (2024): Understanding Reference Policies in Direct Preference Optimization. arXiv preprint arXiv:2407.13709.

Q7: Furthermore, the authors evaluate performance on only two benchmarks, which is limited, particularly in LLM alignment experiments. More evaluations on academic benchmarks like MT-Bench, MMLU, GSM8K, and TruthfulQA are required. Testing on models without RLHF, such as Llama-3-8B, would be necessary to confirm that the proposed algorithm does not rely on pre-existing alignment.

A7: Thank you for your suggestion. We have incorporated a new benchmark into our study. Please note that all experiments have been conducted using the Llama-3-8B-BASE model to address your concerns regarding pre-existing alignment.

These results confirm stable performance improvements, and the additional experiments will be included in the paper.

We hope these additional clarifications address your concerns comprehensively. Thank you again for your thoughtful review and the opportunity to improve our work.

2. The significance of theoretical guarantees

In our response A11, we emphasized that importance sampling is a widely adopted strategy in the literature on preference optimization and reinforcement learning.

In our response A12, we analyzed the advantages of $\alpha$-DPO from a gradient perspective, highlighting its alignment with the optimization direction of online SimPO.

Finally, we would like to emphasize that SimPO has already demonstrated its strengths in various experiments. Our method extends SimPO to personalized scenarios, with the parameter $\alpha$ serving as a critical control factor. Specifically:

• The choice of $\alpha$ strikes a balance between two competing objectives. When $\alpha = 0$, the method simplifies to SimPO, where all training samples share the same target margin.
• As $\alpha$ increases, the model places greater weight on samples with accurate reward estimates.
• However, excessively large values of $\alpha$ cause the model to focus solely on a few samples with the most accurate reward estimates, neglecting the contributions of other samples, which can hinder the training process.

This trade-off underscores the necessity of selecting an appropriate $\alpha$ to balance conservativeness and aggressiveness effectively.

3. The improvements are not significant.

In our response A14, we first highlighted that AlpacaEval2 and Arena-Hard are two of the most widely used benchmarks. These benchmarks have been validated using GPT-4 Turbo, achieving over 98% agreement with human evaluations. Therefore, we believe the scores derived from these benchmarks are sufficiently reliable.

Furthermore, considering the development trends of current methodologies—from DPO to various approaches such as IPO, KTO, CPO, and the state-of-the-art SimPO introduced this year—our improvements over SimPO are both substantial and stable. We encourage the reviewers to re-examine the progress in existing techniques and recognize the significance of our contributions within this context.

Your constructive feedback has been invaluable, and we are committed to leveraging this discussion to improve our work. Thank you again for the opportunity to address your questions and for considering our responses.

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address your comments in detail.

Q1: My major concern is that the motivation of the proposed reference policy Eq.(8) is not so clear.

A1: The motivation for the proposed reference policy $\hat{\pi}\_{\text{ref}}(y|x)$ can be clarified as follows:

1. Utility Theory Perspective: The proposed $\hat{\pi}\_{\text{ref}}(y|x)$ is designed with the uniform distribution $U(y|x)$ as a baseline. The term $\left(\frac{\pi\_\theta(y|x)}{\pi\_{\text{ref}}(y|x)}\right)^\alpha$ dynamically adjusts the reward margin by balancing contributions from the policy and reference models. This mechanism can be interpreted through the lens of utility theory as relative attractiveness, enabling adaptive instance-specific reward modeling.

2. Gradient Perspective: By introducing $\hat{\pi}\_{\text{ref}}(y|x)$, the framework mitigates the label flipping issues found in DPO or SimPO. In the SimPO framework, the gradient is expressed as: $\nabla\_\theta\mathcal{L}\_{\mathrm{SimPO}}(\pi\_\theta)=-\beta\mathbb{E}\_{(x,y\_w,y\_l)\sim\mathcal{D}}\left[s\_\theta\left(\frac1{|y\_w|}\nabla\_\theta\log\pi\_\theta(y\_w|x)-\frac1{|y\_l|}\nabla\_\theta\log\pi\_\theta(y\_l|x)\right)\right],$ where $s\_\theta=\sigma\left(\frac\beta{|y\_l|}\log\pi\_\theta(y\_l|x)-\frac\beta{|y\_w|}\log\pi\_\theta(y\_w|x)+\gamma\right)$. This formulation may amplify weights when the reward estimate is incorrect. By contrast, under $\alpha$-DPO: $s\_\theta=\sigma\left(\frac\beta{|y\_l|}\log\pi\_\theta(y\_l|x)-\frac\beta{|y\_w|}\log\pi\_\theta(y\_w|x)+\gamma + \alpha M(x,y\_w,y\_l)\right),$ the additional $\alpha M(x,y\_w,y\_l)$ component increases weight when the reward estimate is accurate, ensuring a more robust reward signal.

3. Motivational Core: The central goal of the proposed $\alpha$-DPO is to address the unreliability of the reference policy, as outlined in Section 3.1. By integrating the policy model into the reference model design, the quality of the reference model is enhanced, improving fine-tuning performance. Similar concepts have been explored in recent works [1][2]:

We have also provided a theoretical justification through reward difference estimation. For more details, please refer to our response (A4) to Reviewer 8xwP.

[1] Gorbatovski et al. (2024): Learn your reference model for real good alignment. arXiv preprint arXiv:2404.09656.
[2] Liu et al. (2024): Understanding Reference Policies in Direct Preference Optimization. arXiv preprint arXiv:2407.13709.

Q2: The authors use the Taylor expansion w.r.t (\log(A-\alpha B)) on (\alpha=0). This makes the lower bound only established around zero, which limits the theoretical contribution.

A2: Thank you for pointing this out. We apologize for the imprecise explanation in the original submission. The correct statement is:

By expanding $\log\sigma(A-\alpha B)$ around $A$ using the first-order Taylor expansion, we obtain: ...

• The expansion is expanded around $A$, not $\alpha=0$.
• Using the first-order Taylor expansion for $\log\sigma(A-\alpha B)$ at $A$, we derive:

$\log\sigma(A-\alpha B)\approx \log\sigma(A)+(\log\sigma(A))^{'}(-\alpha B)=\log\sigma(A)-\alpha B (1-\sigma(A)),$

where the derivative of $\log\sigma(x)$, $(\log\sigma(x))'=1-\sigma(x)$, is used. This derivation ensures mathematical validity and is not limited to the neighborhood of $\alpha=0$. We have update the manuscript to reflect this clarification.

Q3: Moreover, in lines 913 and 914, the authors state that "under the assumption that the reference policy has large errors", but it seems that I do not find it in the lemma. The authors should mention it directly in the lemma and discuss why this assumption is mild.

A3: Thank you for pointing this out. The assumption that the reference policy has large errors serves as the motivation for proposing $\alpha$-DPO. This assumption is discussed in Section 3.1, where we highlight the unreliability of the reference policy in DPO. To make this point clearer, we have revised the manuscript to include this assumption explicitly in the lemma and expanded the discussion to emphasize its practical relevance and mildness. Specifically:

• The unreliability of the reference policy arises due to the static nature of its design, which may not generalize well across diverse datasets or instances.
• Incorporating the policy model into the reference policy design mitigates these limitations and improves the alignment between the reward model and policy fine-tuning objectives.

This revision will ensure that the lemma is self-contained and directly addresses the context of the assumption.

We hope these additional clarifications address your concerns comprehensively. Thank you again for your thoughtful review and the opportunity to improve our work.

Thank you for your comments. Below are our responses to your concerns:

Q4: However, when the reward estimate is inaccurate, could this hinder training? Is there any empirical evidence demonstrating that the reward estimate is consistently accurate (or accurate most of the time)?

A4: When the reward estimate is inaccurate, it does not hinder training. Specifically, when the reward signal is incorrect (e.g., $M(x, y_w, y_l) < 0$), the weight decreases as defined by

This reduces the influence of incorrect signals, aligning with the design motivation of $\alpha$-DPO.

Although we could not provide additional experiments within the rebuttal period due to time constraints, we have strong reasons to believe that the reward estimate is consistently accurate. Specifically, the quality of the reference model improves with the fine-tuned $\pi_\theta$, as shown in our theoretical and empirical analyses.

Q5: I understand that $\alpha$ can be non-zero, but lines 891 and 841 suggest that $\alpha$ still needs to be very small. This makes $\alpha$-DPO still need to be quite similar to SimPO to make the theory establish.

A5: Based on both theoretical considerations (to maintain similarity to SimPO and support the theoretical framework) and empirical results (validated by the parameter selections in Appendix Table 3 and Table 4), it appears that $\alpha$ is effective when kept small. Our findings suggest that using a small $\alpha$, such as the default value of $1\text{e-}2$, provides stable and significant performance improvements, aligning well with theoretical expectations and experimental observations.

The choice of $\alpha$ reflects a trade-off. When $\alpha$ is set to 0, the approach reduces to SimPO. As $\alpha$ increases, the model places greater emphasis on samples where the reward estimate is correct. However, if $\alpha$ becomes too large, the model may become overly conservative, which can negatively affect training by limiting its ability to explore diverse solutions. This trade-off underscores the importance of selecting an appropriate $\alpha$ to balance alignment and adaptability.

We thank the reviewer for their thoughtful feedback and constructive suggestions. Below, we address your comments in detail.

Q1: However, as uniform policy assigns equal probabilities to all responses, this directly implies $\gamma=0$.

A1: We first want to jusitfy that the uniform distribution assumption is proper, and the uniform distribution does not lead to $\gamma = 0$. The uniform distribution assumption on a LLM describes this model as not having any alignment w.r.t human preference. It is commonly happening in the LLM without enough high equality instruction tuning. The positive/negative samples are generated via reject sampling procedures on LLM and the corresponding distributions are different.

Let's revisit the acquisition method for these samples (e.g., using princeton-nlp/llama3-ultrafeedback dataset): For each prompt $x$, we generate 5 responses using the SFT model with a sampling temperature of 0.8. We then score these responses with llm-blender/PairRM and select the highest-scoring one as $y\_w$ and the lowest-scoring one as $y\_l$.

For the highest-scoring sample $y\_w$, it follows a new conditional distribution:

$\pi(y \mid x, \text{score}(y) = \max(\{\text{score}(y\_1), \ldots, \text{score}(y\_5)\}))\approx\pi(win|x),$

where the last part uses the assumption that for a given x the generative model has a uniform chance to generate all possible y. Similarly, for the lowest-scoring sample $y\_l$, it follows:

$\pi(y \mid x, \text{score}(y) = \min(\{\text{score}(y\_1), \ldots, \text{score}(y\_5)\}))\approx \pi(lose|x)$

Each response is generated from the same conditional distribution $\pi\_{\text{SFT}}$, implying the same distribution during generation. However, by scoring and selecting using llm-blender/PairRM, there is an inherent selection bias. In the above setting, the $\gamma$ value is $\gamma \approx \log \pi (win|x) - \log \pi(lose|x)$.

By further assuming this situation uniformly happens over all x, we reduce to the SimPO configuration. Thus, the statement "uniform policy assigns equal probabilities to all responses" is inaccurate.

Q2: They claim that when it is 0, they get back the SimPO loss.

A2: Please note that according to the discussion in the answer to Q1, the uniform assumption doesn't lead to $\gamma = 0$. Based on the definition in Eq (8), when $\alpha = 0$, $\hat{\pi}\_{\textrm{ref}}$ becomes the uniform distribution. Similarly to A1, the uniform distribution can mitigate the unreliability of $\pi\_{\text{ref}}$, but it still lacks sample personalization and remains suboptimal. Our $\alpha$-DPO objective is designed to incorporate the strengths of both DPO and SimPO. We thank the reviewer for pointing it out and will add the discussion in the Section 3.1 to further clarify it.

Q3: Their proposed loss in Eqn.12 combines SimPO and DPO loss with an extra stop gradient on the DPO loss.

A3: Indeed, from a formal perspective, the $\alpha$-DPO appears as DPO with an additional stop gradient on the Margin term. However, this result demonstrates that SimPO with an offset ($r(x,y\_w)-r(x,y\_l)-M$) can further enhance performance, presenting theoretical benefits (c.f. Section 4) and relationships that, although concise, are effective.

We have also provided a theoretical justification through reward difference estimation. For more details, please refer to our response (A4) to Reviewer 8xwP.

Q4: They introduce an importance sampling correction term for the online SimPO loss. How is that relevant to Lemma 4.2 or any other part of the paper?

A4: It is well known that preference optimization with online ingredients, particularly methods like online AI feedback (OAIF) [1] and self-play [2,3], enhances model alignment by generating new data during training. However, due to the computation cost sample regeneration and training stability issues, the pure online type preference optimization methods are not well applied in industrial practice. Based on this observation, we thus proposed to study an interesting research topic, i.e., can we mimic the online feature in the classic offline setting? Our theoretical analysis is trying to close the gap with the importance sampling trick.

In particular, the underlying logic here first defines the expression for online SimPO, which is characterized by the continuous updating of sampled data during training. In this process, the data transitions from the offline set $(y\_w, y\_l) \in D$ to the online set $(y\_w, y\_l) \in \pi\_\theta$. This operation can be viewed as an important sampling method. Interestingly, this key technique of importance sampling aligns with the optimization direction of $\alpha$-DPO loss, with both approaches converging in the scenario where $\alpha \to 0$. Consequently, Lemma 4.2 aims to establish a connection between $\alpha$-DPO and the Online SimPO loss, thereby facilitating the integration of these concepts.

[1] Guo et al. Direct Language Model Alignment from Online AI Feedback. CoRR abs/2402.04792 (2024)

[2] Chen et al. Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. ICML 2024.

[3] Wu et al. Self-Play Preference Optimization for Language Model Alignment. CoRR abs/2405.00675 (2024)

Q5: I am not sure how Lemma 4.3 contributes. Further, their claim that they improve upon token-level DPO is not correct.

A5: There seems to be a misconception that needs clarification. The core contribution of token-level DPO is the introduction of the form $r(x,y\_w)-r(x,y\_l)-\delta$, similar to DPO with an offset, enabling control over KL divergence. The $\alpha$-DPO follows the form SimPO with an offset $r(x,y\_w)-r(x,y\_l)-M$, providing performance enhancement and showing that $M$ is more effective than $\delta$. Appendix Table 6 supports this with performance comparisons between TDPO and $\alpha$-DPO. These findings illustrate:

1. Adding an offset to DPO and its variants is a successful strategy, applicable to both token-level DPO and $\alpha$-DPO.
2. The choice of offset is still undecided. Under the premise of an unreliable reference model in the SimPO concept, $M$ outperforms $\delta$, offering valuable insights.

We hope these additional clarifications address your concerns comprehensively. Thank you again for your thoughtful review and the opportunity to improve our work.

We sincerely thank you for your support and for raising the score of our work. We deeply appreciate your recognition of our contributions to the $\alpha$-DPO. Regarding your valuable suggestions on improving the presentation, we will incorporate them into the final version. Thank you again for your thoughtful feedback, which has been instrumental in enhancing the quality of our work.