We wholeheartedly appreciate your insightful comments, which we will carefully incorporate should the paper be accepted. During the rebuttal, we have conducted extensive theoretical and empirical studies to address the concerns raised by you and other reviewers. Below, we summarize your major comments, followed by our responses.

Technical novelty (Weakness 1).

We understand your concern regarding novelty, given that both doubly robust (DR) methods and preference-based RLHF have been well-studied individually. Before discussing our technical novelty, we would like to highlight that the connection our work establishes between these two areas itself is novel and, we believe, highly valuable. While there is extensive literature on LLM fine-tuning, no prior work has approached it from the perspective of DR or semiparametrically efficient estimation. In that sense, our proposal not only introduces a novel methodology but also opens the door for integrating other DR and more general OPE methodologies (e.g., marginalized IS, Liu et al., 2018; conditional IS, Rowland et al., 2020; more robust DR, Farajtabar et al., 2018; to mention a few) into LLM fine-tuning.

Second, regarding technical novelty, even if one aims to apply DR to RLHF, this is far from straightforward. The primary challenge stems from the difference in settings: in RLHF, we do not observe scalar reward signals but instead receive pairwise human preference. Consequently, extending standard DR -- which are designed for scalar rewards -- to reward-based RLHF (see Section 2) for learning doubly robust policy values poses significant challenges. Our approach to this challenge is to shift the focus from learning policy values to directly learning preferences. By adopting the preference-based framework, we manage to apply DR to LLM fine-tuning.

Finally, we highlight a unique aspect of our DR estimator that sets it apart from existing estimators. Due to our focus on pairwise preference feedback, we are able to leverage the inherent symmetry in the data: we can treat the second response as a reference and estimate the probability that the first response is preferred using a DR estimator, or alternatively, treat the first response as the reference and estimate the probability that the second response is preferred. Our analysis shows that neither estimator alone achieves semiparametric efficiency. However, averaging the two -- taking advantage of the symmetry in pairwise preferences -- does yield a semiparametrically efficient estimator. This insight explains why our proposed estimator (see Equation (8)) is constructed as the average of two estimators. This structure arises uniquely from the nature of pairwise comparison and does not appear in prior DR or efficient RL estimators.

Improvement over PPO/DPO (Questions 1, 4 & 5)

Thank you for raising this important point. We appreciate the opportunity to clarify how our algorithm improves upon both DPO and PPO. We structure our response as follows, discussing our improvements in three settings: (i) when the BT model holds; (ii) when BT does not hold; (iii) where the reference policy is correctly specified. Notice that (ii) and (iii) directly address your Questions #4 and #1, respectively.

• When BT model holds. Theoretically, our proposal enjoys two key properties: (i) double robustness; (ii) (semiparametric) efficiency. The first, as you noted, ensures robustness over DPO/PPO -- specifically, the error vanishes if either of two terms converges to zero, as is required by PPO or DPO individually. To quantitatively characterize our improvement over PPO and DPO, the second property -- efficiency -- comes into play.

  Specifically, semiparametric efficiency means that, in the context of preference evaluation, the proposed estimator achieves the smallest possible asymptotic MSE (see Theorem 2 and Corollary 4). As for preference optimization, this efficiency is precisely characterized in Theorem 7, where we show that the suboptimality gap of our algorithms scales as the product $\|\|\widehat{\pi}\_{\textrm{ref}}/\pi\_{\textrm{ref}} -1\|\| \|\|\widehat{r}-r^*\|\|$, whereas for PPO and DPO, their gaps scale linearly with one of these terms alone:

  • PPO: $\|\| \widehat{r}-r^*\|\|$
  • DPO: $\|\| \widehat{\pi}\_{\textrm{ref}}/\pi\_{\textrm{ref}} -1\|\|$

  The product structure means that the impact of each estimation error on our algorithm is second-order, while in PPO and DPO it is first-order. Consequently, when both errors converge to zero (thanks to the approximation capabilities of transformer-based architectures), our algorithm's suboptimality converges to zero at a much faster rate than that of PPO and DPO. This provides a clear quantitative theoretical improvement over both PPO and DPO.

  Additionally, our concrete algorithm is detailed on Page 12 of the Appendix. It is not merely a conceptual framework.

• Beyond BT model. Yes, our algorithm retains meaningful performance guarantees beyond the BT framework. In fact, Theorems 2 and 5, as well as Corollaries 3, 4, and 6, do not rely on the BT model assumption. When the BT model does not hold, a scalar reward function that measures the goodness of each response may not even exist, rendering its value function undefined. In this case, we use the total preference as the performance measure. For instance, Theorem 5 upper bounds the gap in total preference between our estimated policy and the one that maximizes the total preference. These guarantees hold under the coverage (Assumption 1) and model complexity assumptions (Assumption 4).

  Next, suppose the scalar reward function does exist, but human preference does not satisfy BT. For instance, $\mathbb{P}(y_1 \succ y_2|x) = g(r(y_1,x) - r(y_2,x))$ for some monotonic link function $g$ that is not a sigmoid. In this case, the BT model is violated, and DPO/PPO no longer retain their theoretical guarantees. However, under Assumptions 2 and 3, we can extend our Theorem 7 to this setting and derive a similar suboptimality bound for our algorithm. This highlights another advantage of our approach: PPO and DPO require to correctly specify the form of the link function $g$, whereas our algorithm does not.

• When $\widehat{\pi}\_{\textrm{ref}}=\pi\_{\textrm{ref}}$. We focus on the case where the BT model does hold, since without this assumption, DPO and PPO no longer retain their theoretical guarantees. As discussed in our earlier response, the reward estimation error does not affect the suboptimality gap of our algorithm -- this term becomes zero. In contrast, PPO’s suboptimality still depends linearly on the reward estimation error.

We hope this response clarifies our theoretical improvements over DPO/PPO. Please let us know if you have any follow-up questions.

Lower bound (Weakness 2 & Question 2)

We focus on Theorem 7 which provides the suboptimality gaps for PPO, DPO and DRPO. For each algorithm, its gap contains three components:

• A bias term induced by KL-regularization (proportional to $\beta$);
• A statistical complexity term of the form $\sqrt{v/n}$, which depends on the sample size $n$ and the complexity measure $v$ of the policy class;
• A reward/reference policy estimation error term.

Since the first term can be made arbitrarily small by choosing a sufficiently small $\beta$, we discuss the tightness of the second and third terms below. For the second term, our upper bounds for PPO/DPO match the lower bounds developed in Nika et al. (2024, arXiv:2403.01857), indicating their tightness. Finally, our theoretical investigation during the rebuttal reveals that under certain settings -- e.g., when the reward is linear and the prompt distribution is multivariate Gaussian -- the suboptimality gap of PPO depends linearly on the estimation error of the regression coefficient, which itself is proportional to the reward model estimation error. Meanwhile, for DPO, when there is a constant gap between the specified and oracle reference policy, the algorithm suffers from a constant suboptimality gap that will not converge to zero. This demonstrates the tightness of the third term.

Given the tightness of the established bounds, we are more confident that our DRPO's suboptimality gap -- being second-order in the estimation error -- can be much smaller than those for PPO/DPO.

Coverage assumption (Question 3)

Good point. First, we can clip the IS ratio (as implemented in our experiments) to improve the algorithm's stability. Second recent works such as Huang et al. (2025, arXiv:2407.13399) propose to replace the KL divergence with a chi-square divergence. This relaxes the uniform coverage assumption to a weaker partial coverage condition -- namely, coverage is only required to hold for the optimal policy rather than for all policies. We will discuss these approaches shall our paper be accepted.

Weaknesses 3&4

We will adopt the term performance gap and revise the bound to make its dependence on the magnitude of the reward function explicit.

---

We hope these address your concerns, and we would be happy to discuss further if you have any follow-up questions.

We wholeheartedly appreciate your insightful comments, which we will carefully incorporate should the paper be accepted. Below, we first respond to your concern on our experiments. We next respond to your questions.

The experiment result might be weak. The base model only uses 125m parameters. That's way below standard.

We apologize for any confusion and are grateful for the chance to clarify. First, the 125M model was used only for the preference evaluation in Section 6.1, where the goal is to evaluate the MSE of our proposed doubly robust preference estimator. For the preference optimization experiments on the HH and TL;DR datasets, we use much larger base models -- each with billions of parameters -- as discussed in Appendix C.2.

Second, in response to your concern, we have conducted an additional experiment during the rebuttal period using a 7B base model instead of 125M for the preference evaluation. The results summarized in Table A1 below again validate the double robustness property of our estimator. We are happy to replace the original 125M results with these updated results if needed.

Third, we have substantially expanded our numerical study during the rebuttal by:

• adding 7 new baseline algorithms for comparison;
• using new benchmarks for evaluation.

Refer to our responses to Reviewers 8Qi6 and ZHXd for details.

Table A1: MSE of the proposed preference estimator with a 7B base model. The preference model and reference policy can be misspecified or correctly specified.

In Line 110, it says that Y1 and Y2 are sampled from the reference policy. But there are cases where the two responses are not generated from the reference policy like Anthropic’s HH dataset? Do your results, in particular, double robustness, rely on that assumption? How to deal with the case when the reference policy is unknown?

We make some clarifications. First, $Y_1$, $Y_2$ are the compared responses in the data, rather than those simulated based on our algorithms. We define the reference policy as the distribution of these responses given the prompt.

Second, our proposal accommodates unknown reference policies, as is the case with the HH dataset. It does not require access to the oracle reference policy. In such cases, we can estimate the reference policy via SFT and plug in this estimator for preference optimization. Indeed, the double robustness property is particularly relevant in these cases: it guarantees consistency as long as either the reference policy or the reward (or preference) model is correctly estimated—not necessarily both.

When estimation errors exist, our algorithm's performance gaps in Theorems 5 & 7 depend explicitly on these estimation errors. However, due to the double robustness property, these errors enter our suboptimality bound in a product (second-order) form, whereas in PPO and DPO, they appear linearly (first-order). As a result, when both estimation errors converge to zero -- thanks to the strong approximation capabilities of transformer-based architectures -- our algorithm’s suboptimality gap converges to zero much faster than that of PPO and DPO.

Typo in line 195-196? (9) should be reduced to IS estimator when setting $\hat{g}$ to zero and the DM estimator when setting the IS ratio to zero?

Thanks for pointing it out. You are right. We will correct the typo.

Assumption 4 is unlikely to be relevant for LLM where the model capacity is really large and VC dim is thus big too. While it seems fine asymptotically, because one can have bounded and fixed VC dimension and simply increase sample size. But typically, as one increases the sample size, one would use a larger model and thus larger VC dimension as well?

Excellent comment -- thank you for raising it. Before clarifying the VC dimension, we note that Assumption 4 is only required in Theorems 5, 7 and Corollary 6. It is not needed in Theorem 2 and Corollaries 3 & 4.

Second, the VC dimension in Assumption 4 can be restricted to the complexity of the policy class $\Pi$ used during fine-tuning, rather than to the entire function class represented by the underlying LLM. While LLMs do have high capacity, fine-tuning typically restricts optimization to a much smaller and more structured subset of the parameter space -- for instance, For example, fine-tuning is often performed for only a few epochs, updates may employ low-rank adaptations such as LoRA (Hu et al., 2021), and KL regularization keeps the fine-tuned policy close to the reference policy. As a result, the effective policy class can have a much smaller VC dimension than the full model class, making this assumption more practical.

I am kind of surprised my the relative poor performance of PPO here even compared to PPO. Are you using standard hyper-parameter? Or do you do the hyperparmater search for PPO yourself?

Are you referring to the relatively poor performance of PPO compared to DPO, rather than PPO itself? We employed two datasets in our main paper: TL;DR and HH. Interestingly, PPO outperforms DPO on TL;DR, but performs worse than DPO on HH.

To explain this, as detailed in Appendix C.2 of our paper:

• For the HH dataset, we train the reference policy via SFT using the HH dataset itself. This ensures that the reference policy is well-aligned with the data and likely well-specified.
• For the TL;DR dataset, we use the public model cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr as the reference policy. However, this SFT model was trained on a filtered version of the dataset, which differs from the original dataset used in our experiments (see Huang et al., 2024). As a result, this reference policy may be misspecified for TL;DR.

This explains the difference in PPO's performance compared to DPO: DPO performs better on HH (where the reference policy is correctly specified), whereas PPO performs better on TL;DR (where it is not). It also aligns with our theoretical analysis, which shows that DPO's suboptimality gap depends on the estimation error of the reference policy.

Regarding hyperparameters, we conduct our own hyperparameter search for PPO on the HH dataset. Specifically, as we mentioned in Appendix C.2, training PPO on the HH dataset is highly sensitive to the reward model. We experimented with different numbers of training epochs for the reward model and found that when the reward model overfits, PPO tends to suffer from policy collapse, hacking the reward model by repeating high-reward tokens. To mitigate this, we limit the reward model training to just one epoch to avoid overfitting. We then conduct a hyperparameter search over KL coefficients $\beta \in (0.05, 0.1, 0.2)$ and learning rates $\alpha\in (1\text{e-}7, 1\text{e-}6, 3\text{e-}6)$. Finally, we set $\beta=0.05$ and $\alpha=1\text{e-}7$ as it yields the most stable training performance.

In line 265, when do you mean by “when the BT model assumption is violated”? In that case, how does one define preference probability?

Violation of the BT model assumption refers to settings where human preference does not follow the BT model -- the probability of preferring one response over another cannot be expressed as a sigmoid function of the difference between their scalar rewards. Moreover, in these settings, a scalar reward function that measures the goodness of a response may not even exist.

Nonetheless, the preference probability remains well-defined as a function of two responses and the prompt, even though it is no longer a function of reward differences. Additionally, these preference functions can be modeled using more general preference functions (see e.g., Zhang et al., 2024), without assuming an underlying scalar reward function.

Since the reward function may not exist in these settings, the value of a policy is not well-defined. As such, we use the gap in total preference as the performance measure for our algorithm (see Theorem 5 and Corollary 6).

---

We hope this clarifies your question, and we would be happy to elaborate further if you have any follow-up questions.

We wholeheartedly appreciate your insightful comments, which we will carefully incorporate should the paper be accepted. Below, we respond to your comments and questions.

No code provided. Would the authors be willing to open-source the code?

We would like to clarify that we did include our code in the Supplementary Materials.

Of course, we are happy to open-source our code. Indeed, we have already published our code in an anonymous GitHub repository (Due to NeurIPS policy, we are unable to include the link).

While the paper is generally well written, I got lost in the details of section 4. I was confused because the authors mention that eq 6 and 7 require correct preference and reference policies. Then, through eq 8, the authors mention that the correction term addresses misspecification of the preference model. It was unclear to me how misspecification of the reference policy is handled. I think it would be helpful at the end of section 4 to show how each part of eq 8 supports the main claims of the paper.

Can the authors please clarify in eq 8, which part of the equation addresses preference model misspecification, reward model misspecification and reference policy misspecification?

We greatly appreciate the above two thoughtful comments. Following your suggestion, we plan to include a discussion at the end of Section 4 to clarify how each type of model misspecification is handled by Eq (8).

To begin with, we clarify that Eq (6) and (7) correspond to the DM and IS estimators, respectively. As you pointed out, their consistency depends on the correct specification of the preference model (for DM) and the reference policy (for IS).

The DR estimator combines the virtue of DM and IS by involving both the preference model and reference policy in Eq (8). Each type of misspecification is addressed as follows:

• Reference policy misspecification. In this case, the preference model is correct (i.e., $\widehat{g}=g^\*$). The reference policy misspecification is addressed by the inclusion of the residual $Z-g^\*$ in the second term of Eq (8). Specifically, since $\mathbb{E}[Z-g^\*(X,Y^{(1)},Y^{(2)})|Y^{(1)},Y^{(2)},X]=0$, by the law of total expectation, the second term in Equation (8) -- although it involves an incorrect IS ratio -- has zero expectation by the law of total expectation. As a result, the expectation of Eq (8) reduces to that of the first term, which corresponds to the DM estimator and remains consistent under correct specification of the preference model.

• Preference model misspecification. In this case, we have $\widehat{\pi}\_{\textrm{ref}}=\pi\_{\textrm{ref}}$, and we can rewrite Eq (8) into $\textrm{Eq}~(7) +\Big[\frac{1}{2}\sum_{a=1}^2\mathbb{E}\_{y\sim \pi(\bullet|X)} [\widehat{g}(X,y,Y^{(a)})]-\frac{\pi(Y^{(1)}|X)}{2\pi_\text{ref}(Y^{(1)}|X)}\widehat{g}(X, Y^{(1)}, Y^{(2)})- \frac{\pi(Y^{(2)}|X)}{2\pi_\text{ref}(Y^{(2)}|X)}[1-\widehat{g}(X, Y^{(1)}, Y^{(2)})]\Big]$ Here, the misspecification of the preference model is addressed by the inclusion of the IS ratio in the second term within the square brackets. Specifically, with a correctly specified reference policy, the second term is of mean zero. As such, the expectation of Eq (8) is the same to that of Eq (7), which corresponds to the IS estimator and remains consistent under correct specification of the reference policy.

• Reward model misspecification. Under the BT model assumption, one can estimate the reward and plug in this estimator to estimate the preference function. If the reward model is misspecified, the resulting preference model $\widehat{g}$ is also misspecified. Nonetheless, as in the previous case, the inclusion of the IS ratio in Eq (8) corrects this error. Thus, the same logic used to address preference model misspecification applies here as well.

Can the authors expand on how this method can be generalised beyond Bradley-Terry?

Absolutely. Our proposed doubly robust algorithm relies on estimating two nuisance functions: (i) the reference policy, and (ii) the preference function -- which defines the probability that one response is preferred over another, given the prompt. When the BT model assumption is violated, this preference probability can no longer be expressed as a sigmoid function of the reward difference between two responses. However, the preference probability itself remains well-defined and can still be directly estimated from data.

For example, general preference models (e.g., Zhang et al., 2024) model preferences without assuming the existence of an underlying scalar reward function. Other models such as PairRM also provide valid estimates of pairwise preference probabilities without relying on the BT assumption. These models can be used to estimate the preference probability, which can be plugged into our total preference estimator for the subsequent preference optimization.

From a theoretical perspective, without the BT model assumption, a scalar reward function that measures the goodness of individual responses may not exist, rendering the policy value undefined. To address this, we use the gap in total preference as the measure to gauge the theoretical performance of our algorithm (see Theorem 5 & Corollary 6). If a reward function does exist but the preference probabilities follow a general (non-sigmoid) link function, our Theorem 7 can be extended to derive similar suboptimality bounds in terms of policy values. We elaborate on this in our Response #2 to Reviewer YVEF (see Beyond BT model paragraph).

---

We hope this clarifies your question, and we would be happy to elaborate further if you have any follow-up questions.

We wholeheartedly appreciate your insightful comments, which raise three concerns about (i) insufficient related work discussion (Weakness 1 & Question 1); (ii) weak baseline comparisons, including the lack of robust DPO algorithms and other advanced or closely related algorithms (Weakness 2 & Question 2) and (iii) limited experimental evaluations. In response, during the rebuttal, we have:

• Provided a detailed discussion of the connections and distinctions between our doubly robust algorithm and existing robust DPO methods. We plan to incorporate this discussion into the paper shall it be accepted.
• Added comparisons with seven additional DPO-style algorithms to further demonstrate the competitiveness of our proposed DRPO.
• Employed AlpacaEval 2 to strengthen our evaluation by benchmarking across different algorithms.

We next elaborate on these points below.

Insufficient related work discussion. We appreciate you pointing out these references on robust DPO, and we fully agree that they are relevant to our work and worthwhile to cite. We have also compared against these algorithms empirically (refer to our next response). Shall our paper be accepted, we plan to use the extra page to expand the discussion section to further elaborate on the connections and differences between our work and these approaches, as follows:

Recent studies have developed robust variants of DPO (Chowdhury et al., 2024; Liang et al., 2024; Wu et al., 2025), focusing on distributional robustness -- that is, robustness to discrepancies between training and test distributions. In particular, these methods studied pairwise noise, where preference labels may be flipped in the training data. In contrast, our algorithm does not address distributional shift or pairwise noise. Instead, it focuses on model misspecification and is designed to remain consistent under misspecification of the preference model or reference policy, assuming the training and test data share the same distribution.

Thus, the two lines of work are robust in different senses. Our newly conducted numerical study further highlights this distinction (see our response to your next comment). Specifically, robust DPO methods tend to perform better or comparably when the level of pairwise noise or distributional shift is high, whereas our method outperforms them when such issues are mild or absent.

That said, our DRPO framework is general and can be extended to handle pairwise noise: for instance, one could incorporate reward functions learned via robust DPO methods into our total preference estimator for preference optimization, or apply the weighting function $w$ from Dr. DPO to reweight each observation, in order to handle pairwise noise.

If the reviewer has additional related works to suggest, we would be glad to compare and discuss them shall our paper be accepted.

Experiments. Following your suggestion, we compared against seven baseline algorithms during the rebuttal. These include the robust DPO methods rDPO and Dr. DPO that you mentioned, as well as cDPO (Mitchell, 2023). We also included more advanced DPO-type algorithms such as RSO (Zhao et al., 2023) and CPO (Xu et al., 2024). Finally, we included IPO (Azar et al., 2024) and ORPO (Hong et al., 2024), which are two closely related algorithms to ours. Most algorithms were implemented using Hugging Face’s trl library. An exception is DR-DPO, which we trained using a modified version of the DPO trainer based on the authors’ released code, as its official trl implementation is not available. Unfortunately, due to the lack of publicly available code for ROPO, we were unable to include it in our experimental comparison at this time.

Table B1 below reports the win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Both win rates are evaluated using AlpacaEval 2.0 you suggested. Table B2 reports the win rate of the proposed DRPO method compared to the seven baseline algorithms on the TL;DR dataset.

On the HH dataset, DRPO achieves higher win rates than DPO-style algorithms (RSO, CPO) as well as IPO and ORPO, performs comparably to robust DPO methods rDPO and cDPO, and underperforms Dr. DPO. On the TL;DR dataset, however, DRPO consistently achieves win rates above 50% across all comparisons—often exceeding 60% and in some cases 90%.

This performance difference likely arises from the presence (or absence) of pairwise noise in the datasets. DRPO is specifically designed to handle model misspecification rather than noisy labels. Its strong performance on TL;DR suggests that this dataset contains minimal pairwise noise—making DRPO more effective than methods tailored to robustness under label noise. In both the rDPO and Dr. DPO papers, the authors found that setting the flip probability to $\epsilon = 0.1$ yields the best performance on the HH dataset, suggesting that HH contains certain levels of pairwise noise. We adopt the same $\epsilon$ in our experiments to ensure these algorithms attain their best performance. These pairwise-noise-robust methods thus outperform or match DRPO on HH, as expected.

Finally, we note that TL;DR is a domain-specific summarization task. General-purpose benchmarks such as AlpacaEval 2.0 may not be appropriate for the evaluation on this dataset. Therefore, we follow the evaluation protocol from DPO and P3O to assess the quality of the generated responses from each algorithm (Note that rDPO and Dr. DPO did not report results on this dataset).

Table B1. Win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Higher win rates indicate better performance.

Table B2. Win rate of the proposed DRPO method compared to various baseline algorithms on the TL;DR dataset. Higher win rates indicate better performance of DRPO over the baseline algorithm.

---

We hope these address your concerns, and we would be happy to discuss further if you have any follow-up questions.

We wholeheartedly appreciate your insightful comments. During the rebuttal, we have conducted extensive experiments to strengthen our empirical evaluation and address your concerns. In summary, we have: (i) scaled up the model size; (ii) included seven more recent baseline algorithms for comparison; and (iii) adopted more comprehensive benchmark protocols for evaluation. We next elaborate on these points. Finally, we discuss the appropriateness of our experimental setup for demonstrating preference model misspecification and reference model misspecification.

Weakness 3: Given that this work targets robust alignment, there should be relevant baselines for comparison, but they are absent. Comparisons with methods like IPO or Dr. DPO should be included.

We appreciate you pointing out these robust alignment algorithms such as Dr. DPO, and we fully agree that they are relevant to our work and have compared them empirically during the rebuttal. Before elaborating on our experiments, we first clarify our proposed DRPO and robust alignment methods like Dr. DPO address different forms of robustness. Specifically, algorithms such as Dr. DPO are robust to the discrepancies between training and test distributions, such as pairwise noise in the training data. In contrast, our DRPO is robust to the model misspecification (misspecification of the preference model or reference policy), assuming the training and test data share the same distribution. Our experiments, detailed below, also highlights this difference.

To address your comment, we compared against seven baseline algorithms during the rebuttal. These include the robust DPO methods such as Dr.DPO that you mentioned, as well as cDPO (Mitchell, 2023) and rDPO (Chowdhury et al., 2024). We also included more advanced DPO-type algorithms such as RSO (Zhao et al., 2023) and CPO (Xu et al., 2024). Finally, we included IPO that you mentioned, and ORPO (Hong et al., 2024), another recent preference optimization algorithm. Most algorithms were implemented using Hugging Face’s trl library. An exception is Dr. DPO, which we trained using a modified version of the DPO trainer, as its official trl implementation is not available.

Table A1 below reports the win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Both win rates are evaluated using AlpacaEval 2.0 (following the suggestion from Reviewer ZHXd). Table A2 reports the win rate of the proposed DRPO method compared to the seven baseline algorithms on the TL;DR dataset.

On the HH dataset, DRPO achieves higher win rates than DPO-style algorithms (RSO, CPO) as well as IPO and ORPO, performs comparably to robust DPO methods rDPO and cDPO, and underperforms Dr. DPO. On the TL;DR dataset, however, DRPO consistently achieves win rates above 50% across all comparisons—often exceeding 60% and in some cases 90%.

This performance difference likely arises from the presence (or absence) of pairwise noise in the datasets. DRPO is specifically designed to handle model misspecification rather than noisy labels. Its strong performance on TL;DR suggests that this dataset contains minimal pairwise noise—making DRPO more effective than methods tailored to robustness under label noise. In both the rDPO and Dr. DPO papers, the authors found that setting the flip probability to $\epsilon = 0.1$ yields the best performance on the HH dataset, suggesting that HH contains certain levels of pairwise noise. We adopt the same $\epsilon$ in our experiments to ensure these algorithms attain their best performance. These pairwise-noise-robust methods thus outperform or match DRPO on HH, as expected. Meanwhile, it is also possible to integrate these robust alignment algorithms with our DRPO to enhance its performance in handling pairwise noise: for instance, one could incorporate reward functions learned via robust DPO methods into our total preference estimator for preference optimization, or apply the weighting function $w$ from DR. DPO to reweight each observation, in order to handle pairwise noise.

Table A1. Win rate and length-controlled win rate of different algorithms relative to the SFT policy on the HH dataset. Higher win rates indicate better performance.

Table A2. Win rate of the proposed DRPO method compared to various baseline algorithms on the TL;DR dataset. Higher win rates indicate better performance of DRPO over the baseline algorithm.

Weakness 2 and Question 1: The experiments were conducted on models under 1B parameters. I question whether this approach scales effectively to larger models.

We apologize for any confusion. First, in the main paper, we clarify that only the preference evaluation experiments in Section 6.1 were conducted with models with less than 1B parameters. For the preference optimization experiments on the HH and TL;DR datasets, we use larger base models -- each with billions of parameters -- as discussed in Appendix C.2.

Second, to directly address your concern, we have conducted an additional experiment during the rebuttal period using a 7B base model for the preference evaluation. The results summarized in Table A3 below again validate the double robustness property of our estimator. We are happy to replace the original results with these updated results shall our paper be accepted.

Table A3: MSE of the proposed preference estimator with a 7B base model. The preference model and reference policy can be misspecified or correctly specified.

Weakness 1: Whether the experimental setup is appropriate for demonstrating preference model misspecification and reference model misspecification.

We demonstrate the appropriateness of our employed experimental datasets for investigating preference model misspecification and reference model misspecification below.

First, the HH dataset is well-suited for illustrating preference model misspecification. As mentioned earlier, both the rDPO and Dr. DPO papers noted that this dataset likely contains unknown pairwise noise that is not captured by standard preference (e.g., BT) -- thus violating the assumed preference model. Indeed, the rDPO paper deliberately avoids flipping preferences in this dataset due to the suspected presence of inherent noise, and finds that setting the flip probability to $\epsilon = 0.1$ yields optimal performance. The Dr. DPO paper similarly reports improved results when applying a flip rate of 0.1, even when the dataset itself is not explicitly flipped. These observations indicate that the HH dataset contains mixed preferences -- a setting you recommended to conducted experiments -- making it appropriate for testing the impact of preference model misspecification.

Second, the TL;DR dataset is well-suited for illustrating reference model misspecification. As detailed in Appendix C.2, we use the public model cleanrl/EleutherAI_pythia-1b-deduped__sft__tldr as the reference policy. However, this SFT model was trained on a filtered version of the dataset, which differs from the full TL;DR dataset used in our experiments (see Huang et al., 2024). As a result, the reference policy may be misspecified for our setting. Our experimental findings support this: we observe that DPO, which requires a correctly specified reference policy for consistency, underperforms PPO, suggesting potential reference model misspecification.

Finally, the IMDb dataset serves as a useful illustrative example for investigating both types of misspecification. As it is a synthetic dataset, we have access to the ground-truth preference and reference models, allowing us to precisely control model misspecification. Notably, both the rDPO and Dr. DPO papers also use this dataset. For instance, rDPO introduces pairwise noise to simulate preference model misspecification, while in our setup, we intensify this by randomly assigning preferences (each response being preferred with 50% probability). Consequently, our simulated data has even larger preference complexity than in rDPO.

---

We hope these address your concerns, and we would be happy to discuss further if you have any follow-up questions.