Mitigating Reward Over-optimization in Direct Alignment Algorithms with Importance Sampling

Thank you for your review and positive feedback on our paper. We appreciate your acknowledgment of the clarity and organization of our work. Please find our response to your comments below.

Q1. Why should we consider the Taylor expansion? And the KL term is only valid at the start of training

Answer: Thanks for the great question. You are correct that directly computing the gradient does not yield the KL term as shown in Equation 8. In Section 2, our goal is to highlight that the KL regularization term is only meaningful at the beginning of training when using DAAs. Without importance sampling or on-policy sampling, this KL term in DAAs fails to accurately reflect the KL regularization used in RLHF as training progresses with more updates. Hence, by multiplying with the importance ratio, we can enforce more effective regularization and better reflect the KL regularization in RLHF without requiring on-policy samples.

Q2. Missing assumptions: DAAs do not necessarily require $\pi_{\text{ref}}$ to produce $\mathcal D$ and the necessity of SFT for IS-DPO.

Answer: Thank you for pointing this out. We first apologize for the implicit assumption in our formulation that the policy produced $\mathcal D$ is the same as $\pi_{\text{ref}}$. We will make it more explicit in the final version.

Thank you for a great comment on the necessity of (SFT) on the preference dataset for IS-DPO to be effective. We'd like to clarify that this requirement is not unique to IS-DPO but is also essential for DPO and other DAAs [11,12].

Nevertheless, we agree that the suggestion of the reviewer can be interesting, thus we also provide experiments exploring how both DPO and IS-DPO perform when trained on preference data not aligned with $\pi_{\text{ref}}$. Specifically, we construct a new dataset $\mathcal{D}\_{\text{shift}}$ by sampling responses from final Online-DPO policies, which achieve a high win rate of $\approx 97\%$ but exhibit a significant shift from $\pi_{\text{ref}}$ $\left(\sqrt{\text{KL}(\pi_\theta || \pi_{\text{ref}})} \approx 7.2\right)$. These responses are then labeled using the golden reward to generate 100k preference samples. We train both IS-DPO and DPO on $\mathcal{D}_{\text{shift}}$ with $\beta=0.05$ and $\log\epsilon=0.5$. We present our results below:

Both methods achieve lower win rates compared to training on the original dataset, reaffirming the importance of SFT as a crucial warm-up step. Interestingly, IS-DPO achieves a slightly higher win rate than DPO while maintaining significantly lower KL divergence from the $\pi_{\text{ref}}$. This suggests that IS-DPO may exhibit more robust to distribution shifts even when preference data is not sampled from $\pi_{\text{ref}}$.

Q3. Theorem 1 just says that importance sampling is an unbiased estimator, which is not a new result, and strange statement.

Answer: Thank you for your suggestions. While the unbiased nature of IS is not new, this theorem, and our paper in general, formalize the impact of distribution shift in DAAs and provide a theoretical analysis (see Section 3.1) to justify the IS correction under DPO objective. Specifically, we show that minimizing the implicit regularization in DPO when applied with importance sampling can also minimize the KL divergence. This connection not only grounds our method in addressing distribution shift but also provides a crucial link between our proposed regularization and KL regularization in RLHF via Theorem 1, as acknowledged by reviewer PaV2. Nevertheless, we will consider the reviewer's suggestion and revise the theorem accordingly to reflect what's known and what's our focus (as in this discussion).

Q4. The support constraint in Importance Sampling (Sec 4.3)

Answer: Thank you for your question. To further elaborate, we consider the support of a discrete distribution $P$ over sample space $\mathcal X$ as follows: $\text{supp}(P)=\{x\in\mathcal X|P(x) > 0\}$. The IS estimator is defined as $\mathbb E_{x\sim Q}\left[\frac{P(x)}{Q(x)}f(x)\right]=\sum\limits_{x\in\text{supp}(Q)}Q(x)\frac{P(x)}{Q(x)}f(x)=\sum\limits_{x\in\text{supp}(Q)}P(x) f(x)$. When we assume $\text{supp}(P) \subseteq \text{supp}(Q)$, we can see that the IS estimator is unbiased, $\sum\limits_{x\in\text{supp}(Q)}P(x) f(x) =\sum_{x\in\text{supp}(P)}P(x) f(x)=\mathbb{E}_{x\sim P}\left[f(x)\right]$.

While it's true that the optimal policy under the DPO objective satisfies $\pi^* \propto \pi_{\text{ref}} e^{r}$, and thus shares the same support as $\pi_{\text{ref}}$. However, prior works [1,2,3,4] have shown that DAAs assume that $\pi_{\text{ref}}$ has full support over the entire prompt-response space to achieve $\pi^*$. In practice, this assumption rarely holds, as preference datasets only cover a small fraction of the prompt-response space. Consequently, multiple distinct policies can achieve the same global optimum of the DPO objective [2,3,4], including policies that assign probability mass to responses outside the support of $\pi_{\text{ref}}$ [1, 3]. This contrasts to the original RLHF framework, which prevents generating responses that are of support $\pi_{\text{ref}}$ via explicit KL regularization. Thus, our proposed importance sampling method further enforces the support constraint, helping to avoid this failure mode.

Q5. Limited experiments with HH and TLDR are simple datasets.

Answer: Thank you for your feedback. We employ HH and TLDR, as they are widely adopted standard benchmarks in the alignment literature [4,5,6,7] for evaluation of DPO and its variants. Consequently, our work should be evaluated on these benchmarks for a fair evaluation with the existing baselines; evaluating on additional datasets should indeed incur significant computational constraints.

Nevertheless, please note that we have also conducted additional experiments on the UltraFeedback dataset in the paper, where we evaluate IS-DPO using popular benchmarks such as AlpacaEval 2.0 and MT-Bench, designed to measure alignment with human preferences in more practical settings. These results (Section 4.2) further support the effectiveness of IS-DPO beyond HH and TLDR. We hope this extended evaluation addresses your concerns regarding the scope of our experimental validation.

Q6. The invariance of IS-DPO to $\beta$.

Answer: Thank you for your great comment. We confirm that there is no bug in Fig. 4 of IS-DPO. We observed that $\text{KL}(\pi_\theta,|, \pi_{\text{ref}})$ during training typically remains within the range of 22.87 to 27.06. As discussed in Section 4.3, this behavior arises because the importance ratio in IS-DPO gradually decreases throughout training. As the importance weights diminish, resulting in an implicit early stopping effect.

This mechanism provides two key benefits. First, a low importance ratio implies that the current policy $\pi_\theta$ assigns very low likelihood to certain responses, and further updating on such samples can lead to over-optimization [5, 9]. IS-DPO naturally avoids this issue by down-weighting such updates. Second, a low importance weight may signal that on-policy sampling is needed, which is especially beneficial in iterative learning settings [10].

Q7. Why 2 epochs for training instead of 1 epoch?

Answer: Thank you for your question. Prior work [9] has shown that DPO can suffer from over-optimization even before completing a single epoch, with further training resulting in performance degradation. Motivated by this, our work focuses on settings where over-optimization is a particular concern. In the Controlled Summarization task, we deliberately train for 2 epochs to highlight the robustness of IS-DPO under prolonged training and weak regularization (i.e., low $\beta$); this setting is also consistent with prior studies [5, 6]. To ensure fair comparisons with standard setups, we still follow the standard 1-epoch configuration used in Instruction Following tasks [4, 7, 8] to demonstrate IS-DPO effectiveness in aligning with human preferences.

Q8. The values of $\beta$ in Fig. 3? Why IS-DPO exhibit worse performance at different KL budgets Answer: Thank you for your questions. We consider a range of $\beta\in\{0.01, 0.05, 0.1\}$ to consider different KL values in Fig 3. Viewing KL as a resource to be spent, we observe that while DPO exhibits better performance under a low KL budget, IS-DPO tends to consume more of the effective KL budget, ultimately achieving a higher peak win rate. Interestingly, we found that this phenomenon also exhibits in Online-DPO and previous work [8,13].

---

[1] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML24 Oral.
[2] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS24.
[3] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage. NeurIPS24.
[4] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS24.
[5] Disentangling Length from Quality in Direct Preference Optimization. ACL24.
[6] Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR24 Spotlight.
[7] REBEL: Reinforcement learning via regressing relative rewards. NeurIPS24.
[8] Direct Language Model Alignment from Online AI Feedback. CORR24
[9] Learning Dynamics of LLM Finetuning. ICLR25 Oral.
[10] Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences. CoRR24.
[11] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS23 Oral. 
[12] ORPO: Monolithic Preference Optimization without Reference Model. ACL24.
[13] Understanding the performance gap between online and offline alignment algorithms. CoRR24

Thank you for your thoughtful review and valuable feedback. Please find our response to your comments below.

Q1. The core proposed by the authors that offline algorithms suffer from distribution shift as the policy moves away from the data-generating policy, is a well-known problem in the field of offline reinforcement learning. The proposed solution, applying importance sampling, is a standard method to tackle this, which has already applied in PPO. As such, the paper's conceptual contribution feels incremental rather than novel, amounting to the application of a known technique to the specific context of DPO.

Answer: Thank you for your comment. We'd like to highlight that our contributions goes beyond the use of Importance Sampling (IS) to address the distribution shift, due to the following reasons:

• Foundational Theoretical and Empirical Analysis of IS in DPO: Our paper provides a theoretical connection of overoptimization and distributional shift between the learned and preference policies in DPO (Section 3.1). Then we propose IS the correction for overoptimization in DPO. However, the most significant analysis of our work is theoreatically demonstrating that IS correction implicitly (and approximately) regularize the DPO's policy in a similar manner as that of minimizing the KL divergence with online samples (see Theorem 1, and Section 5.2's discussion, and also our empirical evidence for online DPO as a response to fRzo's Q2). In summary, our paper provides rigorous theoretical analysis of the overoptimization problem and its solution (via IS) in the context of DAAs, which is the significant contribution beyond simply an application of IS.

• The roles and impact of Importance sampling and clipping are fundamentally different in PPO and IS-DPO: While IS has also been used to address distribution shift in PPO, its usage is fundamentally different from our paper. PPO uses clipped importance sampling (IS-clip) to improve sample efficiency. Since each batch is typically discarded after a single gradient update in standard policy gradient methods, IS allows PPO to reuse the same batch multiple times, extracting the most information from each batch. In contrast, our objective is to enforce an effective regularization, to prevent the from pushing up the probability of responses that are out of the offline data distribution, mitigating the over-optimization issue. In addition, in PPO, the clipping mechanism acts as a surrogate trust region to prevent the policy from deviating too far from the previous one (which can help mitigate over-optimization), but in a fully offline setting (as in our paper), this mechanism becomes overly conservative-where gradient is zero when the probability ratio falls outside the clipping range $[1-\epsilon, 1+\epsilon]$, limiting the potential improvement from $\pi_{\text{ref}}$ [4,5]. Our clipping strategy, conversely, mitigates excessively large updates with IS while still providing meaningful updates even when the ratio lies outside the clipping region.

In summary, these contributions are significant and original, and we hope the response has addressed the reviewer's concerns. We thank the reviewer again for the thoughtful feedback.

Q2. The method's stability and performance rely entirely on the clipping ratio. However, the paper offers no principled way to set this crucial hyper-parameter, aside from a single ablation study shown in Figure 5. This introduces a new, sensitive parameter that a practitioner must optimize via expensive trial-and-error, which counteracts the claimed simplicity of the method and could easily negate its benefits in practice.

Answers: Thank you for the comment. IS-DPO indeed has a hyperparameter, much similar to prior works, with multiple hyperparameters, tackling over-optimization or distributional shift [1,2,3]. However, our ablation studies in Figure 5 demonstrate the robustness of IS-DPO's hyperparameter -- specifically that IS-DPO maintains strong performance across a reasonable range of clipping values robustness and in Section 4.1, we empirically suggest the use of $\log \epsilon = 1.0$ as the preferred choice in practice, which we use for all our experiments.

In addition, we will now provide a theoretical insight that better understands the relationship between the regularization coefficient $\beta$ and the clipping parameter $\epsilon$, which helps provide a principled tuning of $\epsilon$.

Proposition: Assuming $\text{Supp}(\pi_\theta)=\text{Supp}(\pi_{\text{ref}})$, then the variance of the importance weight $\mathbb{Var}\left(\frac{\pi_\theta}{\pi_{\text{ref}}}\right)$ under $\pi_{\text{ref}}$ distribution is an upper bound of KL divergence: $\text{KL}\left(\pi_\theta || \pi_{\text{ref}}\right) \leq \mathbb {Var}\left(\frac{\pi_\theta}{\pi_{\text{ref}}}\right)$

The proof is straightforward; we will include the detailed proof in our camera-ready version. This proposition highlights that when regularization is weak (i.e., small $\beta$), the policy $\pi_\theta$ can deviate significantly from $\pi_{\text{ref}}$, leading to high variance in the importance weights. This motivates the use of a smaller clipping threshold $\epsilon$ to control variance and stabilize training under such regimes.

Additionally, we also conduct the relationship between the regularization parameter $\beta \in\\{0.01,0.05,0.1,0.5\\}$ and the clipping ratio $\log\epsilon \in\\{0.25, 0.5, 1.0\\}$. We present the win-rate and KL values below:

Win-rate table:

KL table:

$\epsilon$ as an additional regularization parameter: We observe that increasing the clipping threshold allows for larger policy updates, which in turn leads to higher KL divergence across all values of $\beta$. Interestingly, at low regularization levels (small $\beta$), using a smaller clipping ratio yields better performance, as it helps prevent over-optimization. In contrast, at higher regularization levels (e.g., $\beta = 0.5$), the stronger regularization larger clipping ratios that are more effective.

Q3. The paper claims the method has "minimal computational overhead," but this is not thoroughly justified. Calculating sequence-level importance ratios for every sample in a batch adds computational steps that, while not requiring new forward passes, are not free. Considering its added complexity, it is not as simple as implied by the paper.

Answer: Thank you for your comment. We'd like to emphasize that the computational overhead of calculating the importance ratios is negligible and results in almost the same time complexity as DPO. Despite adding the additional importance ratio calculation step to the original DPO loss, this ratio can be efficiently computed using intermediate quantities already available in the DPO loss calculation. Specifically, as presented in the paper, the importance ratio can be computed in the log-space as follows:

$w(\mathbf y_w, \mathbf y_l) = e^{\log\pi_\theta (\mathbf y_w)+\log\pi_\theta(\mathbf y_l)-\log\pi_{\text{ref}}(\mathbf y_w)-\log\pi_{\text{ref}}(\mathbf y_l)}$

Hence, IS-DPO only adds a small calculation, negligible compared to the much more heavier computations in the vanilla DPO. This implementation is also straightforward and simple, requiring only a single additional line of code.

To empirically validate this claim, we report the average training time over 100 steps for both DPO and IS-DPO. We use 4 A100-80GB GPUs, a batch size of 16, and LLaMA-3.2-3B as the base model:

As can be observed, the training time for IS-DPO is nearly identical to that of the vanilla DPO, confirming that our approach is computationally efficient and negligible in overhead.

---

1. Disentangling Length from Quality in Direct Preference Optimization. ACL24
2. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS24.
3. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR25 Spotlight.
4. The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure. AAAI23.
5. Off-Policy Proximal Policy Optimization. AAAI23.

We thank the reviewer for their helpful feedback and will respond to the raised questions below.

Q1. Discussion between the connection of clipped Importance sampling in IS-DPO and PPO.

Answer: Thank you for your comment. We'd like to emphasize that the contributions of our paper go beyond the mere application of Importance Sampling (IS) in DAAs. We aim to provide a theoretical understanding of overoptimization, an important problem, in DAAs; based on this understanding, we then propose Importance Sampling (IS) as an approximation of the KL regularization between the reference policy and the training policy under the offline setting. In addition, the roles of IS in PPO (an online algorithm) and in our IS-DPO (an offline algorithm) are also fundamentally different. Specifically:

• Foundational Theoretical and Empirical Analysis of IS in DPO: Our paper provides a theoretical connection of overoptimization and distributional shift between the learned and preference policies in DPO (Section 3.1). Then we propose IS the correction for overoptimization in DPO. However, the most significant analysis of our work is theoreatically demonstrating that IS correction implicitly (and approximately) regularize the DPO's policy in a similar manner as that of minimizing the KL divergence with online samples (see Theorem 1, and Section 5.2's discussion, and also our empirical evidence for online DPO as a response to fRzo's Q2). In summary, our paper provides a rigorous theoretical analysis of the overoptimization problem and its solution (via IS) in the context of DAAs, which is the significant contribution beyond simply an application of IS.

• The roles and impact of Importance sampling and clipping are fundamentally different in PPO and IS-DPO: While IS has also been used to address distribution shift in PPO, its usage is fundamentally different from our paper. PPO uses clipped importance sampling (IS-clip) to improve sample efficiency. Since each batch is typically discarded after a single gradient update in standard policy gradient methods, IS allows PPO to reuse the same batch multiple times, extracting the most information from each batch. In contrast, our objective is to enforce an effective regularization, to prevent them from pushing up the probability of responses that are out of the offline data distribution, mitigating the over-optimization issue. In addition, in PPO, the clipping mechanism acts as a surrogate trust region to prevent the policy from deviating too far from the previous one (which can help mitigate over-optimization), but in a fully offline setting (as in our paper), this mechanism becomes overly conservative-where gradient is zero when the probability ratio falls outside the clipping range $[1-\epsilon, 1+\epsilon]$, limiting the potential improvement from $\pi_{\text{ref}}$ [4,5]. Our clipping strategy, conversely, mitigates excessively large updates with IS while still providing meaningful updates even when the ratio lies outside the clipping region.

In summary, these contributions are significant and original, and we hope the response has addressed the reviewer's concerns. We thank the reviewer again for the thoughtful feedback.

Q2: The sensitivity of clipping ratio $\epsilon$ and practical guidance how to set it.

Answer: Thank you for your question. We'd like to clarify that the ablation studies in Figure 5 already provide practical guidance to set $\epsilon$. Specifically, the studies demonstrate the robustness of IS-DPO's hyperparameter -- specifically that IS-DPO maintains strong performance across a reasonable range of clipping values robustness and in Section 4.1, we empirically suggest the use of $\log \epsilon = 1.0$ as the preferred choice in practice, which we use for all our experiments.

Nevertheless, we follow the reviewer's suggestion and now also provide a more theoretical insight on the relationship between the regularization coefficient $\beta$ (DPO's hyperparameter) and the clipping parameter $\epsilon$. We hope this will provide additional guidance on selecting $\epsilon$.

Proposition: Consider the two distribution $\pi$ an $\pi_{\text{ref}}$. Assuming $\text{Supp}(\pi_\theta)=\text{Supp}(\pi_{\text{ref}})$, then the variance of the importance weight $\mathbb{Var}\left(\frac{\pi_\theta}{\pi_{\text{ref}}}\right)$ under $\pi_{\text{ref}}$ distribution is an upper bound of KL divergence: $\text{KL}\left(\pi_\theta || \pi_{\text{ref}}\right) \leq \mathbb {Var}\left(\frac{\pi_\theta}{\pi_{\text{ref}}}\right)$

The proof is straightforward by utilizing the inequality $\log x\leq x-1,\forall x>0$. We will include the detailed proof in our camera-ready version. This proposition highlights that when regularization is weak (i.e., small $\beta$), the policy $\pi_\theta$ can deviate significantly from $\pi_{\text{ref}}$, leading to high variance in the importance weights. This suggests a smaller clipping threshold $\epsilon$ to control variance and stabilize training under such regimes and vice versa when regularization is strong.

We also conduct the relationship between the regularization parameter $\beta \in\\{0.01,0.05,0.1,0.5\\}$ and the clipping ratio $\log\epsilon \in\\{0.25, 0.5, 1.0\\}$. We present the win-rate and KL values below:

Win-rates

KL values

$\epsilon$ as an additional regularization parameter: We observe that increasing the clipping threshold allows for larger policy updates, which in turn leads to higher KL divergence across all values of $\beta$. Interestingly, at low regularization levels ($\beta=0.01$), using a smaller clipping ratio yields better performance, as it helps prevents over-optimization. In contrast, at higher regularization levels (e.g., $\beta \in\\{0.05, 0.1, 0.5\\}$), larger clipping ratios are more effective. Overall, performance remains robust across a wide range of $\log \epsilon$ values. Consistent with our ablation studies, we recommend setting $\log \epsilon = 1$ as a default choice in practice.

Q3. Comparison with Online-DPO

Answer: Thank you for a great suggestion. As suggested, we conduct experiments to understand how well the IS-DPO can approximate the Online-DPO. We present our findings below:

DPO result:

IS-DPO result:

Online-DPO result:

IS-DPO can closely approximate Online-DPO up to a specific KL divergence threshold: We observe that IS-DPO also follows a similar performance trajectory to Online-DPO in the low to moderate KL divergence regime $\sqrt{\text{KL}}\left(\pi_\theta ||\pi_{\text{ref}}\right)\approx 6$. In this region, both methods show a similar and steady increase in win rate. This suggests that IS-DPO is capable of closely approximating the behavior and performance of Online-DPO within a certain KL "budget".

On-policy benefits from large KL "budget": While IS-DPO does not show signs of over-optimization, it also does not extrapolate to higher KL regimes, while Online-DPO can find better policies under higher KL regimes. This indicates that on-policy optimization remains beneficial when the optimal policies lie farther from the $\pi_{\text{ref}}$, as Online-DPO is able to leverage a larger KL budget.

Comparison with DPO: We also provide a comparison between Online-DPO and DPO. Interestingly, we observe that under low KL budget, DPO appears to outperform Online-DPO $\sqrt{\text{KL}}(\pi_\theta ||\pi_{\text{ref}}) \approx 4.5$. However, it sharply declines as KL increases, indicating over-optimization, while IS-DPO and DPO effectively utilize a larger KL budget to achieve higher performance.

Q4: Hypothesis on why $\chi$-PO more effect on AlpacaEval.

Answer: Thank you for a great observation. Despite the consistently strong performance of IS-DPO, $\chi$-PO indeed achieves a slightly (though marginally) better result in AlpacaEval. While we do not have a definitive explanation, we suspect IS-DPO's hyperparameter selection (i.e., $\epsilon$) may not be optimal for this dataset. A deeper investigation, e.g., why $\chi$-PO performs well in AlpacaEval, could be valuable, but this requires extensively more experiments and analysis on many benchmarks to confirm this observation. We believe this deserves an independent study beyond the scope of our paper.

---

1. Disentangling Length from Quality in Direct Preference Optimization. ACL24
2. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS24.
3. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR25 Spotlight.
4. The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measure. AAAI23.
5. Off-Policy Proximal Policy Optimization. AAAI23.

Thank you for your great comments and efforts in reviewing our paper. Please find our response to your comments below.

Q1. A key issue is regarding the KL regularization to the reference policy, which is a crucial component in the alignment objective. It is not extremely clear in the scenario where there is clipped importance weight being used, how the KL is still ensured mathematically w.r.t the original alignment objective? Theorem 1 provides the details, but an explicit description showing how the importance weight affects the KL balls is crucial? Empirical analysis is also crucial

Answer: Thank you for your question. We acknowledge that Theorem 1 only provides the details for importance weight. Below, we provide an explanation to clarify how the clipped importance weight affects alignment with the KL regularization. We consider the $\mu$-squared loss with importance sampling: $\nabla_\theta\mu_{\text{IS}}(y_1,y_2)=w\nabla_\theta\rho, \quad \nabla_\theta\mu_{\text{clip}}(y_1,y_2)=\min(w, c)\nabla_\theta \rho$ where $w=\frac{\pi_\theta(y_1,y_2)}{\pi_{\text{ref}}(y_1,y_2)},\rho=\log\frac{\pi_\theta(y_1)}{\pi_{\text{ref}(y_1)}}-\log\frac{\pi_\theta(y_2}{\pi_{\text{ref}(y_2)}}$. As shows in Theorem 1, we can see that $\mathbb E_{\pi_{\text{ref}}}[\nabla_\theta\mu_{\text{IS}}]=\nabla_\theta\text{KL}(\pi_\theta||\pi_{\text{ref}})$.

Clipped IS-DPO ensures alignment with KL regularization: Notice that the gradient of the clipped objective can be expressed as

$\\nabla\_{\\theta}\mu\_{\text{clip}}=\nabla\_\theta\mu\_{\text{IS}}-\mathrm{1}\_{\{w > c\}}(w-c)\nabla\_\theta\rho$

This decomposition shows that clipping introduces a bias by down-weighting the influence of samples with large importance weights. However, the clipped gradient still maintains alignment with the importance-weighted gradient, as indicated by the dot product. $\langle\nabla_\theta\mu_{\text{clip}}, \nabla_\theta\mu_{\text{IS}}\rangle \geq 0$

In effect, clipping acts as a form of regularization that mitigates gradient explosion from high-weight outliers while preserving alignment with the KL regularization.

Q2. Does the unbiasedness of the gradient estimate still hold true with the importance weight and smoothening? It will be helpful to highlight the same. In the importance weight estimation, there can be trajectories generated whose probability is low under the reference policy making the weights unbounded, how the smoothing helps in this regard?

Answer: We really appreciate this question. We sincerely apologize for the typo in Equation (13) — the operator should indeed be $\min$ instead of $\max$.

While the clipping operator introduces into the estimator, its purpose is to keep the gradient variance under control. This clipping mechanism is especially important in cases where certain trajectories have very low probability under $\pi_{\text{ref}}$. The clipping mechanism will be applied the keep the importance weight bounded to stabilize training.

Q3. Additionally, the Taylor approximation is local and mainly valid around 0, i.e, when the policies are close to reference. However, how this helps in scenarios when they are apart which is the main reason of the dist shift issue leading to overoptimization, then the approximation might not hold. Can you provide a detailed discussion regarding the same?

Answer: Thank you for your question. You are correct that the Taylor approximation used in Offline DAAs only enforces KL regularization when the policies $\pi_\theta$ are close to $\pi_{\text{ref}}$. This is the exact objective our our analysis in this section: we aim to show that existing DAAs will only have effective regularization locally (via this Taylor expansion) and become susceptible to overoptimization as the learned policy diverges further.

More specifically, this approximation shows that DAAs are not well regularized when the learned policy $\pi_\theta$ moves away from $\pi_{\text{ref}}$. Optimizing Offline DAAs can lead to over-optimization or extrapolation into out-of-support regions of $\pi_{\text{ref}}$ $\left(\text{KL}(\pi_\theta||\pi_{\text{ref}}=\infty\right)$ [3,4,5,6], and the local regularization used in Offline DAAs is insufficient to prevent these issues compared to standard RLHF.

Q4. Discussion with works that rely on the principles of pessimism needs to be discussed [1,2].

Answer Thank you for the insightful suggestion. We will include the following discussion, citing the works on principles of pessimism, in the camera-ready version:

At a high level, the principle of pessimism explicitly subtracts uncertainty-based penalties such as KL divergence or $\chi^2$ divergence from the estimated value with the aim of preventing overestimation of the value of candidate policies in regions with low data coverage or high uncertainty.

IS-DPO(IS) also incorporates a form of pessimism. IS-DPO assigns lower weights to trajectories that are less likely under the learned policy $\pi_\theta$, effectively penalizing policies that place high probability mass in areas poorly covered by the data. This mechanism aligns conceptually with pessimism-based regularization [1,2].

Moreover, there is a deeper theoretical connection worth exploring: the variance of the IS estimator is proportional to the chi-squared divergence between $\pi_\theta$ and $\pi_{\text{ref}}$. This observation suggests a promising direction to formalize the connection between importance sampling and pessimism. $\chi^2$ divergence has been used in LLM alignment to impose stronger regularization, thereby effectively mitigating over-optimization issues. We believe that bridging this connection more formally, possibly through uncertainty-aware confidence intervals, is a promising avenue for future work.

---

[1] Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF. ICML24.
[2] Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS24.
[3] Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. ICML24 Oral.
[4] A General Theoretical Paradigm to Understand Learning from Human Preferences. AISTATS24.
[5] The Importance of Online Data: Understanding Preference Fine-tuning via Coverage. NeurIPS24.
[6] Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. NeurIPS24.
[7] Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR25 Spotlight.

Thank you for your review and positive feedback on our paper. We appreciate your acknowledgment of the clarity and organization of our work.

Please find our response to your comments below.

Q1. The statistical significance of the results in Figures 3 and 4 are not clear as many of the baseline approaches are overlapping, with the exception of a low KL penalty (beta) setting where the importance sampling-based approach performs better.

Answer: Thank you for your comment. We will revise the figures in the camera-ready version to clearly indicate statistical improvements. We'd also like to clarify that our evaluation setup follows prior works [Rafailov et al. 2024, Tang et al. 2024a], where Figures 3/4 illustrate the trade-off between KL divergence and the policy performance for all the methods. We kindly ask the reviewer to disregard the visual overlap in the plots. The key takeaway from these figures is that "the other approaches suffer from an early convergence problem, while IS-DPO continues to improve or stay flat at later epochs with a significantly lower KL budget." (lines 241-243). Consequently, the final win-rate results (summarized below) demonstrate the effectiveness of IS-DPO:

Q2. The analysis of the online DPO baseline, which is cited as a motivation to approximate, is missing in the results. It would be good to understand the headroom in the approximation proposed between this computationally intensive alternative. Adding missing evaluations would improve the soundness of the results.

Answer: Thank you for a great suggestion. We have conducted experiments to understand how well the IS-DPO can better approximate the Online-DPO. As we're not allowed to include a Figure, we provide the (reduced) tabular version of the results instead and will add the scaling law curve version of Online-DPO in our camera-ready version. Our findings are below:

IS-DPO can closely approximate Online-DPO up to a specific KL divergence threshold: We observe that IS-DPO also follows a similar performance trajectory to Online-DPO in the low to moderate KL divergence regime $\sqrt{\text{KL}}\left(\pi_\theta ||\pi_{\text{ref}}\right)\approx 6$. In this region, both methods show a similar and steady increase in win rate. This suggests that IS-DPO is capable of closely approximating the behavior and performance of Online-DPO within a certain KL "budget".

On-policy benefits from large KL "budget": While IS-DPO does not show signs of over-optimization, it also does not extrapolate to higher KL regimes, while Online-DPO can find better policies under higher KL regimes. This indicates that on-policy optimization remains beneficial when the optimal policies lie farther from the $\pi_{\text{ref}}$, as Online-DPO is able to leverage a larger KL budget.

Comparison with DPO: We also provide a comparison between Online-DPO and DPO. Interestingly, we observe that under a low KL budget, DPO appears to outperform Online-DPO $\sqrt{\text{KL}}(\pi_\theta ||\pi_{\text{ref}}) \approx 4.5$. However, it sharply declines as KL increases, indicating over-optimization, while IS-DPO and DPO effectively utilize a larger KL budget to achieve higher performance.

DPO result:

IS-DPO result:

Online-DPO result: