PILAF: Optimal Human Preference Sampling for Reward Modeling

We thank the reviewer for their thoughtful review, particularly for recognizing the strength of both the theoretical and experimental parts, and for checking the proofs of all theoretical results.

1. "Value-Incentivized Preference Optimization". The authors should also add this work into discussion.

We thank the reviewer for bringing this up. Please note that we did cite Cen et al. around line 621. As we discussed there, their method does not modify the sampling scheme; rather, it adds a regularization term to the DPO objective to encourage departure from the calibration samples. Therefore, we keep this reference in the appendix and focus on those papers - more directly comparable to our work - that modify the sampling scheme in the main body of our paper. We can relocate this distinction in related works to the main body of the paper if you would consider that it adds clarity to our exposition.

2. Add VPO to the baselines.

At the reviewer’s suggestion, we have added VPO as a baseline in the online setting. Due to the need to generate responses in all experiments, completing the full VPO comparison is infeasible within the rebuttal period. Therefore, we are adding this baseline for the online setting and plan to complete the comparison for the iterative setting in the camera-ready version. We report the figure at anonymous URL. The results show that PILAF outperforms VPO, even after a small hyperparameter search for VPO (details can be found in the provided link).

3. However, this contribution is limited because it is specifically designed for DPO, whose performance is not SOTA on most benchmarks. The authors can try whether this sampling strategy is also useful in other algorithms like IPO, KPO and PPO.

We would like to clarify that our motivation—and the scope of our theoretical and experimental contributions—is not limited to DPO, but rather to reveal and address a fundamental misalignment that pervades the two‑phase RLHF framework. A prevailing assumption in recent RLHF work is that on‑policy data generated during training constitutes “good” alignment data. However, we demonstrate that—even when using on‑policy samples—the alignment process remains suboptimal.

Specifically, RLHF consists of two sequential phases: first, preference data are collected and used to extract human values and train a reward model— via maximum‑likelihood estimation either explicitly (as in PPO) or implicitly (as in DPO, IPO, and KPO)—and second, the learned reward model guides policy optimization. Our main theorem shows that this two‑phase design and the MLE optimization of the reward creates the misalignment between the update gradient and policy gradient maximizing the true human values. Although this issue affects both PPO and DPO equally, we present our theorem in the context of DPO for clarity. We articulate this motivation in the Introduction and provide a detailed discussion of its implications for PPO in Appendix G. By the same reasoning, our theoretical principle extends to IPO; however, because IPO modifies the optimization objective (and thus the gradient), its optimal sampling scheme differs from that derived for DPO. For clarity and focus, we leave a detailed analysis of IPO and its corresponding optimal sampling scheme to future work.

The contribution of our work is multifaceted. First, we rigorously identify and characterize the misalignment problem inherent to the two‑phase RLHF framework—an issue largely overlooked by prior work. Second, we develop a comprehensive, assumption‑light theoretical analysis that directly yields an optimal sampling strategy; unlike Cen et al. (VPO), our approach makes no restrictive assumptions (e.g., reward‑model linearity) and therefore holds under very general conditions. This requires substantial theoretical innovation. Third, we empirically validate PILAF's effectiveness on modern large language models, demonstrating significant and consistent improvements over existing baselines. Consequently, our contribution extends far beyond proposing a new algorithm: it offers a universal, theory‑driven perspective on addressing misalignment for RLHF.

Finally, we respectfully disagree that DPO is not state‑of‑the‑art. Recent studies confirm that the principal driver of performance in RLHF is whether an algorithm incorporates online data generation. When implemented online, DPO matches PPO’s performance as reported by [Noukhovitch et al. 2025, Tang et al. 2024]. Accordingly, we evaluate our method in both iterative and online settings.

Noukhovitch, Michael, et al. "Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models." ICLR 2025.

Tang, Yunhao, et al. "Understanding the performance gap between online and offline alignment algorithms." arXiv preprint arXiv:2405.08448 (2024).

Please let us know if our responses address your concerns.

We thank the reviewer for their time in providing the review.

1. HH-RLHF not enough. More benchmarks.

Please allow us to put our work into more context. Our contributions extend beyond simply empirically validating a new sampling algorithm. Rather, first, we identify and rigorously characterize a previously overlooked misalignment problem in RLHF. Second, we develop a general theoretical framework—free of restrictive assumptions—that directly informs a principled solution for addressing the problem with guarantees. Third, we translate these insights into T‑PILAF, a practical algorithm which we validate at scale using 8B LLMs and the HH‑RLHF dataset. Together, we believe the novel theoretical results and large‑scale experiments convincingly demonstrate (1) the existence of misalignment in standard RLHF, (2) how it can be addressed in a first‑principles manner, and (3) the effectiveness of our method in practice. For further discussion of our contributions, please refer to response #3 to Reviewer kgC8.

We believe the strength of our proposed algorithm lies in its theoretical grounding, provably solving the alignment problem that we expose. As such, we feel that validation on a prominent benchmark, HH-RLHF, with a large LLM summoning a significant amount of compute at our disposal, is convincing evidence.

2. The approach is online iterative DPO, but they simply compare with the vanilla DPO, which is unfair, because online DPO outperforms the vanilla one. The authors are suggested to compare their algorithm with more online methods, such as the general online DPO [1,2,3]

Following the definitions in [1,3], we distinguish between iterative and online data collection as follows: iterative sampling generates all preference data from the policy network at the beginning of each iteration, whereas online sampling produces preference data continuously, with each new batch drawn from the current policy network.

Importantly, we are not comparing vanilla DPO against these modes; instead, we hold the underlying RLHF iterative/online framework fixed and vary only the sampling strategy. Concretely, in both the iterative and online setups, every method collects preference data from the policy network—iteratively in the iterative setup (Sections 6.1) and batch‑by‑batch in the online setup (Section 6.2)—with the sole difference being how those samples are generated. This distinction is detailed in the implementation paragraphs and via different values of $n_t$ in Algorithm 1. The term Vanilla only denotes the sampling method as vanilla, as shown in Table 1. Thus, we are ensuring a fair comparison exactly as the reviewer suggested.

3. Besides, the works [1,2] are actually online DPO instead of simply iterative DPO since they collect samples generated from the trained policy and get them labeled by a preference oracle.

We adopt the definitions of “online” and “iterative” sampling exactly as presented in the referenced papers. In particular, collecting samples from a fully trained policy at each iteration is referred to as iterative DPO in [1, 2].

4. The theoretical analysis is a little confusing and it is hard to get the intuitions why they use such sampling strategy \pi- and \pi+. The authors are suggested to clarify the insights more clearly instead of just listing theorems and equations that seems distinct from the algorithms. Now, it's hard to find connections between the theorem and experiments.

To provide an intuition for the misalignment problem, note that DPO implicitly defines the reward as $r_\theta(x,\vec y) = \beta \cdot \log\left(\frac{\pi_\theta(\vec y \mid x)}{\pi_{\mathrm{ref}}(\vec y \mid x)}\right)$ which is trained via maximum‑likelihood estimation. When training with preference data generated by $\pi_\theta$, the optimization is biased. Reviewer PDrJ also summarizes this aptly as "it may not generalize well to reflect true preferences because the sampled comparisons do not represent the broader preference landscape." Nonetheless, this intuition does not prescribe a concrete sampling strategy for correcting the misalignment.

Instead, we use theoretical analysis to reveal the root cause and derive a principled solution. By comparing Equations 14 and 15a, we demonstrate how the gradient produced by standard sampling diverges from the true alignment gradient. Following Reviewer qMsM’s suggestion, we have added a lemma in the main text that explicitly shows this discrepancy. These insights directly inform T‑PILAF, a sampling algorithm that leverages $\pi^-$ and $\pi^+$, to realign the empirical gradient with its theoretical counterpart (as explained in line 245 left). PILAF is then the practical instantiation of T‑PILAF, and our experiments validate its effectiveness. This seamless integration of theory and practice is precisely what Reviewer PDrJ praised: “these theoretical results motivate the practical design of PILAF.”

Please let us know if our responses address your concerns.

Thanks for appreciating our work, especially its development from theory to algorithm design. We are sincerely pleased that the reviewer aacknowledged the misalignment problem we identified, found the combination of mathematical proofs and experiments to provide a reasonable justification, and considered our experimental results to be extensive.

1. The authors may consider adding an ablation study to analyze whether the gains from PILAF are truly due to its interpolated sampling mechanism and gradient alignment rather than other confounding factors. For example, comparing PILAF with versions that remove or vary the interpolation component would help validate this core contribution.

Thank you for the suggestion. Following it, we added two ablation studies to isolate the contributions of PILAF’s interpolation and extrapolation components. Each component was replaced individually with vanilla sampling, yielding two baselines: one with $({\vec y}^a, {\vec y}^b)=(\pi_\theta^+, \pi_\theta)$ (ablation of the interpolation component) and one with $({\vec y}^a, {\vec y}^b)=(\pi_\theta, \pi_\theta^-)$ (ablation of the extrapolation component). We denote these ablation variants as PILAF-extrapolate and PILAF-interpolate, where one response is obtained via vanilla sampling and the other via extrapolation or interpolation, respectively. Due to time constraints, we completed these ablations only for the online setup; we plan to extend this to the iterative setting in the camera-ready version.

We include the figures at anonymous URL. Our theory suggests that the two sampling responses should come from different distributions in order to yield a controlled difference that the model can effectively learn from. Both ablation variants introduce such differences and outperform vanilla sampling. However, the variant with only interpolation (combined with vanilla sampling for the other response) performs much worse than full PILAF, highlighting the importance of the extrapolation response. The PILAF-extrapolate variant achieves slightly worse final results, and its convergence is much slower (each dot in our figure represents one evaluation after 50 steps). Overall, these ablation results confirm our theoretical prediction that the full PILAF algorithm is the best performing approach.

We thank the reviewer again for their time and constructive feedback.

We appreciate the reviewer’s insightful comments, which have helped improve the presentation of the misalignment problem. We are also glad that the reviewer enjoyed our theoretical analysis and empirical validation.

1. uniform sampling - misaligned gradients

We thank the reviewer for raising this point — a clear, formal statement helps improve our presentation. The discrepancy arises from the difference in gradient formulations between Eqns 14 and 15a. To make notations concise, we introduce the following shorthands: $\Delta r^* := r^*(x, y^a) - r^*(x, y^b)$, $\Delta r_{\theta} := r_{\theta}(x, y^a) - r_{\theta}(x, y^b)$ and $g := \nabla_{\theta} r_{\theta}(x, y^a) - \nabla_{\theta} r_{\theta}(x, y^b)$.

Lemma C.2. $\nabla_{\theta} J(\pi_{\theta}) = \frac{1}{2 \beta} E_{y^a, y^b \sim \pi_{\theta}(\cdot | x)} [ ( \Delta r^* - \Delta r_{\theta} ) g]$.

(Corollary of) Lemma C.3. For vanilla response sampling scheme, $\nabla_{\theta} L(\theta) = -E_{y^a, y^b \sim \pi_{\theta}(\cdot | x)} [(\sigma(\Delta r^*)-\sigma(\Delta r_{\theta}))g]$.

These two gradients share a similar structure. The key difference is $\Delta r^* - \Delta r_{\theta}$ for $\nabla_{\theta} J(\pi_\theta)$ and $\sigma(\Delta r^*) - \sigma(\Delta r_{\theta})$ for $\nabla_{\theta} L(\theta)$. To correct for this mismatch, T-PILAF adjusts the response sampling distribution: It reweights the pairwise response sampling so that the density ratio between the vanilla scheme and T-PILAF approximates the derivative $\sigma'(\Delta r_\theta)$. This bridges the gap between the non-linear sigmoid differences and the linear reward differences, leading to better gradient alignment during training.

2. Q1: Section 2.3

Let us clarify the root cause of the misalignment. The true objective for both PPO and DPO is to optimize the expected return under the true reward function $r^*$ in (6). In PPO, a reward model $r_\theta$ is trained from human preferences and then used to guide policy optimization. In DPO, $r_\theta$ is learned via $r_\theta(x,y) = \beta \log\frac{\pi_\theta(y | x)}{\pi_{\mathrm{ref}}(y | x)}$. In both frameworks, the final policy’s performance depends on how well $r_\theta$ approximates $r^*$, as you noted.

Prior work assumes that on‑policy samples from $\pi_\theta$ suffice for improving both the policy and its reward model. We show this is incorrect: gradients from standard on‑policy sampling do not align with the true policy gradient under $r^*$, limiting $r_\theta$’s ability to close the approximation gap. As Reviewer PDrJ perfectly summarizes: "it may not generalize well to reflect true preferences because the sampled comparisons do not represent the broader preference landscape." This applies to both PPO and DPO; we present the result in the DPO setting for clarity.

In contrast, our T‑PILAF sampling scheme is designed to align the empirical gradient with the true policy gradient under $r^*$. This ensures each update of $r_\theta$ approximates $r^*$ in the optimal first-order direction. Our statistical results also shows T-PILAF minimizes variance. All of our results focus on comparing between the empirical objective and the true objective in (6), showing precisely how vanilla sampling produces misaligned gradients when updating the reward model.

3. How to turn $y^a$ and $y^b$ into $y^w$ and $y^l$.

Before Eqn (1), we noted that $y^w$ and $y^l$ were human-annotated as the preferred and unpreferred responses, respectively. Eqn. (1) introduced the commonly used Bradley-Terry (BT) model, and we stated explicitly that the BT assumption was adopted throughout this paper.

4. Definition of $\pi^*$ in eqn 9

The ground-truth reward function in the BT model (Eqn 1) is denoted by $r^*$. Then, the notation $\pi^*$ refers to the optimal policy that maximizes the value function $J(\pi)$, as in Eqn (6):

$

\pi^(y | x) = \frac{1}{Z(x)} \pi_{ref}(y | x) \exp(\frac{1}{\beta} r^(x,y)).

$where$Z(x)$is a partition function that ensures$\pi^*(\cdot | x)$ sums to 1.

5. line 294 and 295, proofs.

Following the suggestion, we add the derivation:

Proof:

Starting from $\pi_\theta^+(y_t | x, y_{1:t-1}) = \frac{1}{Z(x, y_{1:t-1})} \pi_\theta(y_t | x, y_{1:t-1}) \big(\frac{\pi_\theta(y_t | x, y_{1:t-1})}{\pi_{ref}(y_t | x, y_{1:t-1})}\big)^\beta$,

we rewrite it as: $\pi_\theta^+(y_t | x, y_{1:t-1}) \propto \exp\big((1+\beta)\log \pi_\theta(y_t | x, y_{1:t-1}) - \beta \log \pi_{ref}(y_t | x, y_{1:t-1})\big)$.

Define the logits: $h_\theta(y_t | x, y_{1:t-1}) = \log \pi_\theta(y_t | x, y_{1:t-1})$, $h_{ref}(y_t | x, y_{1:t-1}) = \log \pi_{ref}(y_t | x, y_{1:t-1})$.

Then $\pi_\theta^+(y_t | x, y_{1:t-1}) \propto \exp((1+\beta) h_\theta(y_t | x, y_{1:t-1}) - \beta h_{ref}(y_t | x, y_{1:t-1})).$

Normalizing over all $y_t$ leads to the softmax form: $\pi_{\theta,\beta}(\cdot | x, y_{1:t-1}) = \mathrm{softmax}((1+\beta) h_\theta - \beta h_{ref})$.

Please let us know if our responses address your concerns.