AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization

Q1: Justification of Loss Function

While space constraints limited introductory intuition, we provide multi-faceted justification through:

1. Weak-to-Strong Generalization (Lines 197-200): Similar to weak-to-strong alignment, our adaptive reference models enable policy specialization while preserving exploration.
2. Theoretical Link to TDPO (Sec. 4): Lemma 4.1 shows AlphaDPO's margin term approximates TDPO's KL divergence control, connecting sequence optimization with token-level stability.
3. Appendix Analysis: Utility theory (C.1) and gradient analysis (C.2) demonstrate how our adaptive margin prevents reward hacking while maintaining diversity.

We will add a roadmap paragraph in the introduction to better signpost these analyses.

Q2: Theoretical Comparison to SimPO

We clarify our theoretical progression:

1. TDPO's success demonstrates the effectiveness of $r(x,y\_w)-r(x,y\_l)-\delta$ structures for KL control.
2. AlphaDPO adopts a similar offset structure $r(x,y\_w)-r(x,y\_l)-M$ where:
   • TDPO uses $\delta=\beta(D\_{KL}(y\_l) - D\_{KL}(y\_w))$ with $z\sim\pi\_{ref}$
   • AlphaDPO uses $M=\beta(\log\frac{\pi\_\theta(y\_w)}{\pi\_{ref}(y\_w)} - \log\frac{\pi\_\theta(y\_l)}{\pi\_{ref}(y\_l)})$ with uniform sampling
3. As shown in Appendix D.4, AlphaDPO's $M$ achieves superior performance to TDPO's $\delta$ (+3.2% AlpacaEval LC), demonstrating the advantage of sequence-level KL approximation over token-level computation.

Q3: Crude Approximation
Due to space limits, A comprehensive theoretical analysis is provided in the Response to Reviewer bC1g.

Q4: Base Model Validation
To address potential confounding from pre-aligned models: We conducted additional experiments on Llama3-8B-Base:

Improvements remain consistent, confirming AlphaDPO's effectiveness independent of initial alignment.

Q5: TDPO Comparison Current Appendix D.4 shows: AlphaDPO achieves 58.7% LC vs TDPO's 52.8% on Llama3-8B, demonstrating clear superiority. Thanks for your suggestion, and we will add direct comparisons to Table 1.

Q6: $\gamma$ Formulation The uniform distribution $U(y|x)$ and its role in $\gamma$ can be rigorously defined as follows:

1. Theoretical vs. Empirical $U(y|x)$

   • Theoretical: For vocabulary $\mathcal{V}$, $U(y|x) = \prod\_{t=1}^{|y|} \frac{1}{|\mathcal{V}|} \quad \text{(uniform over tokens)}$
   • Empirical: Responses $y\_1, \dots, y\_5 \sim \pi\_{\text{SFT}}(y|x)$ are scored, with $y\_w$ and $y\_l$ selected via: $y\_w = \arg\max \text{score}(y\_i), \quad y\_l = \arg\min \text{score}(y\_i)$ This induces implicit subspaces:$\mathcal{V}\_{\text{win}} = \\{y\mid \text{score}(y)\geq\tau\\},\quad\mathcal{V}\_{\text{lose}}=\\{y\mid\text{score}(y)\leq \tau'\\}$

2. Effective $U(y|x)$ and $\gamma$.
   The practical uniform distributions become: $U(y\_w|x) = \prod\_{t=1}^{|y\_w|} \frac{1}{|\mathcal{V}\_{\text{win}}|}, \quad U(y\_l|x) = \prod\_{t=1}^{|y\_l|} \frac{1}{|\mathcal{V}\_{\text{lose}}|}$ Thus, $\gamma = \beta (\log U(y\_w|x) - \log U(y\_l|x))$. Since $\mathcal{V}\_{\text{win}}$ and $\mathcal{V}\_{\text{lose}}$ vary per instance, SimPO's fixed $\gamma$ is suboptimal.

Q7: Stop Gradient Analysis

We highlight the primary distinction:

• TDPO: Implements asymmetric gradient flow—enabled for $y\_l$, stopped for $y\_w$.
• AlphaDPO: Employs symmetric stop-gradient on both terms, using $\pi\_{\text{ref}}$ as a fixed anchor.

Both TDPO and AlphaDPO utilize the stop-gradient operation. Examining the detailed impact of single-term versus multi-term stop-gradient is an intriguing avenue for future work.

Q8: $\alpha=1$ Misstatement
Due to space constraints, please refer to the response for Question 3 in Response to Reviewer ciXp.

Q9: Experimental Rigor
We report the standard deviations and confidence intervals of current evaluations in (https://anonymous.4open.science/r/AlphaDPO-431F/significant_exp.md).

Hyperparameter Fairness. We strictly followed community standards:

• All methods underwent equal hyperparameter tuning (Appendix Table 3)
• $\alpha$ was selected via grid search over [1e-2, 5e-2, 0.1, 0.2] with 5 random seeds
• DPO received equivalent tuning effort ($\beta \in [0.01,0.05,0.1]$)

KL Analysis We will enhance Figure 3(c) by:

• Adding more sampling points (every 50 steps)
• Including Pareto frontiers for compared methods

Q10: Marginal Improvement in Table 2
We clarify that Table 2 presents ablation studies within AlphaDPO's design space rather than cross-method comparisons. The results demonstrate that:
i) Dynamic configurations consistently achieve better performance.
ii) The AlphaDPO formulation retains optimality.

We thank the reviewer for raising this important concern. Below, we provide a rigorous theoretical justification for the approximation in Lemma 4.1 and clarify its empirical validity.

Theoretical Justification

1. Problem Formulation with Robustness Constraints
Our objective is to minimize the sequential KL divergence between the optimal reference policy $\pi\_{\text{ref}}^*$ and the policy $\pi\_\theta$, under the uncertainty of $\pi\_{\text{ref}}^*$:

Since $\pi\_{\text{ref}}^*$ is unobserved, we assume bounded deviation from an observable reference policy $\pi\_{\text{ref}}$:
$|\pi\_{\text{ref}}^*(\cdot) - \pi\_{\text{ref}}(\cdot)| \leq C, \quad C > 0.$ This leads to a constrained robust optimization problem:

2. Simplification via Worst-Case Analysis
For the inner maximization, the worst-case $\pi\_{\text{ref}}^*$ is determined by the sign of the log-ratio:

Substituting this into the objective yields:

3. Asymptotic Approximation
When the deviation bound $C$ is sufficiently large (i.e., $\pi\_{\text{ref}}^*$ is less constrained), the absolute value term dominates, and the objective simplifies to:

This corresponds to the sequence-level approximation in Lemma 4.1. The $\approx$ symbol reflects the asymptotic regime where higher-order terms vanish under large $C$, which aligns with practical scenarios where $\pi\_{\text{ref}}$ is imperfect but provides a reasonable prior.

4. Addressing the Uniform Policy Assumption
We clarify that Lemma 4.1 does not assume $\pi\_{\text{ref}}$ is uniform. Instead, it leverages the structure of the KL divergence to show that the margin term $M(x,y\_w,y\_l)$ approximates the sequential KL difference under bounded deviations. The uniform policy in SimPO is a special case of our framework (when $\alpha=0$), but our method generalizes to non-uniform $\pi\_{\text{ref}}$ by adaptively scaling with $\alpha$.

Empirical Validation

Performance
As demonstrated in Appendix Table 6, AlphaDPO achieves significant performance gains over TDPO, with LC win rates increasing from 52.8% (TDPO) to 56.9% (AlphaDPO w/ $\delta$) and further to 58.7% (AlphaDPO w/ $M$). This progressive improvement highlights the effectiveness of the sequence-level approximation in mitigating token-level noise through variance reduction. By replacing TDPO's token-level margin $\delta$ with the adaptive sequence-level margin $M$, AlphaDPO enhances robustness while maintaining alignment, underscoring the superiority of sequence-level optimization in handling imperfect reference models.

Mitigating Reference Model Bias
Figure 2 (main paper) empirically validates that $\pi\_{\text{ref}}$ struggles to distinguish $y\_w$ from $y\_l$ ($\log\pi\_{\text{ref}}(y\_w|x) - \log\pi\_{\text{ref}}(y\_l|x)$ exhibits random fluctuations). AlphaDPO's adaptive margin $M(x,y\_w,y\_l)$ explicitly compensates for this bias, ensuring stable optimization even when $\pi\_{\text{ref}}$ is suboptimal.

Conclusion

While Lemma 4.1 uses an approximation symbol ($\approx$), our analysis rigorously justifies its validity under bounded reference model mismatch. The empirical success of AlphaDPO further supports this design choice, demonstrating that sequence-level approximations enhance robustness without sacrificing performance. We acknowledge that deriving a formal error bound remains an open question and will explore this in future work.

Q1: Clarification on DPO's Reference Model Limitation
We appreciate this insightful question. The necessity for $\pi\_{\text{ref}}$ to distinguish between $y\_w$ and $y\_l$ stems from two fundamental aspects of KL-regularized policy optimization:

1. Theoretical Foundation of KL-Regularized Objectives: The RLHF objective (Equation 2) regularizes the policy $\pi\_\theta$ to stay close to $\pi\_{\text{ref}}$ via KL divergence. This regularization implicitly assumes that $\pi\_{\text{ref}}$ provides a meaningful prior for distinguishing high-quality ($y\_w$) and low-quality ($y\_l$) responses. If $\pi\_{\text{ref}}$ lacks discriminative power (e.g., assigns similar probabilities to $y\_w$ and $y\_l$), the KL term loses its grounding, leading to unstable optimization.

2. Empirical Evidence:Recent studies [1,2] demonstrate that $\pi\_{\text{ref}}$ quality significantly impacts DPO performance. DPO's reliance on a static $\pi\_{\text{ref}}$ introduces both theoretical and practical limitations.

[1] Gorbatovski et al. Learn your reference model for real good alignment. ICLR 2025.
[2] Liu et al. Understanding Reference Policies in Direct Preference Optimization. arXiv preprint arXiv:2407.13709.

Q2: Formalization of $U(y|x)$ and Its Impact
We thank the reviewer for prompting this clarification. The uniform distribution $U(y|x)$ and its role in $\gamma$ can be rigorously defined as follows:

1. Theoretical Framework: Let $\mathcal{V}$ denote the vocabulary. The theoretical $U(y|x)$ is:

   $$U(y|x) = \prod\_{t=1}^{|y|} \frac{1}{|\mathcal{V}|} \quad \text{(uniform over all tokens)}$$

   However, empirically, $y\_w$ and $y\_l$ are selected via:

   • Sampling: Generate $y\_1,...,y\_5 \sim \pi\_{\text{SFT}}(y|x)$
   • Selection: $y\_w = \arg\max\_{y\_i} \text{score}(y\_i)$, $y\_l = \arg\min\_{y\_i} \text{score}(y\_i)$ This induces implicit vocabulary subspaces: $\mathcal{V}\_{\text{win}} = \\{y \in \mathcal{V} \mid \text{score}(y) \geq \tau\\}, \quad \mathcal{V}\_{\text{lose}} = \\{y \in \mathcal{V} \mid \text{score}(y) \leq \tau'\\}$

2. Effective $U(y|x)$ in Practice: The effective $U(y\_w|x)$ and $U(y\_l|x)$ become: $U(y\_w|x) = \prod\_{t=1}^{|y\_w|} \frac{1}{|\mathcal{V}\_{\text{win}}|}, \quad U(y\_l|x) = \prod\_{t=1}^{|y\_l|} \frac{1}{|\mathcal{V}\_{\text{lose}}|}$
   This leads to:$\gamma = \beta \left( \log U(y\_w|x) - \log U(y\_l|x) \right)$. The performance impact arises because $\mathcal{V}\_{\text{win}}$ and $\mathcal{V}\_{\text{lose}}$ differ across instances, making SimPO's fixed $\gamma$ suboptimal.

Q3: Alignment of AlphaDPO with DPO at $\alpha=1$
We sincerely appreciate the reviewer's careful observation and apologize for the ambiguity in our initial formulation. The implicit reference model $\hat{\pi}\_{\text{ref}}(y|x)$ is defined as: $\hat{\pi}\_{\text{ref}}(y|x)\propto U(y|x) \left( \frac{\pi\_\theta(y|x)}{\pi\_{\text{ref}}(y|x)} \right)^\alpha,$
where $\alpha$ interpolates between two extremes:

1. When $\alpha = 0$: $\hat{\pi}\_{\text{ref}}(y|x)$ reduces to the uniform distribution $U(y|x)$, aligning with SimPO's implicit reference model.
2. When $\alpha > 0$: $\hat{\pi}\_{\text{ref}}(y|x)$ increasingly incorporates the dynamic term $\frac{\pi\_\theta}{\pi\_{\text{ref}}}$, creating an adaptive reference model. Critically, AlphaDPO does not strictly reduce to DPO for any finite $\alpha$. Instead, it introduces a novel framework that balances exploration (via $U(y|x)$) and exploitation (via $\frac{\pi\_\theta}{\pi\_{\text{ref}}}$).

We will revise the manuscript to eliminate the misleading claim about AlphaDPO reducing to DPO and instead emphasize its unique interpolation mechanism.

Q4: Statistical Significance of Performance Gains
We respectfully disagree. Table 1 shows statistically significant improvements across benchmarks:

These gains are consistent and meaningful, particularly given the saturated performance of modern LLMs. To rigorously validate the robustness of our improvements, we report the standard deviations and confidence intervals of current evaluations in (https://anonymous.4open.science/r/AlphaDPO-431F/significant_exp.md).

Q5: Comparison with Online DPO Methods
AlphaDPO is orthogonal to online preference optimization. While OAIF/IDPO focus on data collection dynamics, our work addresses reference model design in offline settings. Notably, AlphaDPO's adaptive reference can be integrated into online frameworks by replacing static $\pi\_\text{ref}$ with $\hat{\pi}\_{\text{ref}}$. We acknowledge this as valuable future work and will explore it in subsequent studies.

Q1: What is the practical utility of the theoretical connection between AlphaDPO and online methods given that online methods themselves lack strong theoretical guarantees?

We appreciate the reviewer raising this important point. Although we did not explicitly emphasize the theoretical connection to online methods in our submitted manuscript, we acknowledge that exploring this relationship represents a promising direction for future research. We believe this connection offers two meaningful benefits:

1. Algorithmic Insights:
   By establishing a theoretical link between AlphaDPO and online methods, we gain a unified view of how adaptive reward margins and the implicit reference model influence policy optimization. Specifically, AlphaDPO's framework demonstrates how sequential KL divergence control naturally arises, thereby providing a clearer understanding of its inherent ability to balance alignment and diversity—even when the reference model is suboptimal. Such theoretical insights were previously unaddressed explicitly in existing online methods.

2. Empirical Robustness and Practical Guidance:
   Despite the lack of rigorous theoretical guarantees in existing online methods, the theoretical analysis of AlphaDPO indicates that its adaptive mechanism implicitly mitigates typical online optimization pitfalls, such as over-optimization. This robustness is empirically demonstrated through AlphaDPO's stable performance across various KL divergence budgets, as illustrated in Figure 3(c) of our paper.

We agree with the reviewer's point and will pursue a thorough theoretical investigation of this connection as part of our future work, aiming to further clarify its theoretical foundations and practical implications.

Q2: Why is AlphaDPO effective when the reference model is not well-calibrated at the token level, given that it operates at the sequence level?

Full Details: A comprehensive theoretical analysis (including Lemma 4.1 and robust optimization derivations) is provided in the Response to Reviewer bC1g.

AlphaDPO’s robustness to token-level miscalibration stems from sequence-level KL divergence approximation and adaptive margin design, which mitigate noise propagation from unreliable token-level signals.

1. Theoretical Foundation

   • Problem Context: When $\pi\_{\text{ref}}$ is miscalibrated at the token level, token-wise KL terms (e.g., in TDPO) amplify noise.
   • Key Insight: By approximating the sequential KL divergence (Lemma 4.1), AlphaDPO aggregates token-level uncertainties into a sequence-level margin $M(x,y\_w,y\_l)$. This reduces sensitivity to token-level errors, as the sequence-level signal is statistically more stable.
   • Robust Optimization: Our framework explicitly models bounded deviations from $\pi\_{\text{ref}}$ (Section 3.2), ensuring stability even when token-level probabilities are imperfect.

2. Empirical Validation

   • Performance Gain: As shown in Appendix Table 6, AlphaDPO outperforms TDPO (58.7% vs. 52.8% LC win rate on Llama3-8B), demonstrating superior robustness to reference model noise.
   • Bias Compensation: Figure 2 (main paper) shows $\pi\_{\text{ref}}$ fails to distinguish $y\_w$ from $y\_l$ at the token level. AlphaDPO’s adaptive margin $M$ compensates for this by leveraging sequence-level discrepancies, ensuring stable alignment.

Key Takeaways

• Sequence-Level Robustness: AlphaDPO avoids token-level noise amplification via sequence-wise KL control, making it less reliant on perfect token calibration.
• Adaptive Margin: The margin $M(x,y\_w,y\_l)$ dynamically adjusts to instance-specific reference model errors, enhancing robustness.
• Empirical Edge: AlphaDPO’s design consistently outperforms token-level methods (e.g., TDPO) in scenarios with miscalibrated $\pi\_{\text{ref}}$.

Q3: When does AlphaDPO degrade to DPO?

Currently, the $\alpha$-DPO algorithm cannot be transformed into DPO merely through parameter adjustments, similar to how SimPO cannot be converted to DPO by altering $\gamma$. However, I believe this topic presents significant promise, allowing us to propose a more generalized formulation:

where $U(\cdot|x)$ is a uniform distribution. This formulation encompasses:

• DPO: Set $\alpha\_1 = 0$, $\alpha\_2 = 1$, recovering the explicit reference model $\pi\_{\text{ref}}$.
• SimPO: Set $\alpha\_1 = \alpha\_2 = 0$, yielding a uniform reference model.
• AlphaDPO: Set $\alpha\_1 = \alpha$, $\alpha\_2 = -\alpha$, enabling adaptive margin control via $\pi\_\theta/\pi\_{\text{ref}}$.

Q1: Similar idea in previous works.

We sincerely thank the reviewer for pointing out relevant prior works. While both our method and previous approaches involve model interpolation, we highlight three key distinctions:

• Adaptive reward margin: Unlike existing works that focus on regularization strength [1] or data generation [2], AlphaDPO introduces instance-adaptive reward margins by dynamically reparameterizing the reference distribution through the policy-to-reference ratio ($\pi_θ/π_{ref}$), enabling personalized preference learning that accounts for per-sample preference strength.
• Implicit KL control: Our theoretical analysis reveals that the $\alpha$-weighted ratio term implicitly controls sequential KL divergence between iterative policy updates, achieving stability without explicit constraints.
• Generalized Preference Optimization: AlphaDPO generalizes SimPO as special cases ($\alpha=0$) while enabling smooth transitions between policy specialization ($\alpha>0$) and uniform exploration ($\alpha \to 0$). The empirical superiority across three model families (Table 1), demonstrates the critical advantage of adaptive margins.

We will add detailed comparisons to these works in the revised manuscript.

Q2: Sensitivity to $\alpha$

We appreciate the reviewer's insightful questions regarding hyperparameter sensitivity. Here we clarify our methodology:

(1) Primary $\alpha$ Search Space: As shown in Appendix Table 3, we conducted systematic searches over $\alpha \in$ {0.01, 0.05, 0.1, 0.2} based on validation performance. Figure 5 demonstrates that $\alpha=0.05$ achieves optimal results across most models except Llama3-IT-8B-v0.2 where $\alpha=0.2$ works best, reflecting architecture-dependent calibration needs.

(2) Extended Analysis in Figures: The expanded $\alpha$ values in Figures 3-4 (including 0.3, 0.5, etc.) were intentionally explored to demonstrate our method's reward distribution characteristics across a broader spectrum, not to claim performance improvements. We acknowledge this caused unintended confusion and will explicitly label these as "analysis beyond primary search space" in revisions.

(3) Baseline Fairness: All methods including SimPO used identical hyperparameter search budgets. For SimPO's $\gamma$, we strictly followed the original paper's recommendation space {0.3, 0.5, 1.0, 1.2, 1.4, 1.6} as shown in Appendix Table 3, ensuring fair comparison through equivalent tuning efforts.

This controlled approach ensures our conclusions about AlphaDPO's advantages remain valid despite model-specific $\alpha$ variations, while the extended analyses provide valuable insights into the method's behavioral patterns.