Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

Your valuable comments have greatly helped us improve our manuscript. Below are our specific responses to the raised questions:

Weakness 1: Analysis of possible optimization outcomes.

• In Section 3, we analyzed categorical LLM policies, i.e., tabular stochastic policies without function approximations. In this setting, for any prompt $x$ with the chosen and rejected responses $y_w$ and $y_l$, the optimal policy after RLHF must satisfy $\pi^*(y_w|x)=1$ and $\pi^*(y_l|x)=0$ in order to maximize the expected reward, i.e., $\max_\pi E_{y\sim\pi(\cdot|x)}[r(x, y)]$, since $r(x, y_w) > r(x, y_l)$. When only slightly more annotators prefer $y_w$ over $y_l$, i.e., $r(x, y_w)$ is slightly larger than $r(x, y_l)$, $\pi^*(y_l|x)=0$ will cause the LLMs unnecessarily unlearn the high-quality $y_l$. Similar limitations are discussed in more detail in Section 3.1.
• For function-approximated LLM policies, the analysis becomes significantly more complex. To demonstrate that these issues persist in practice, we provided empirical evidence in Figure 3 and will incorporate the newly conducted ablation on the O.O.D. dataset HelpSteer 2:

Weakness 2: Missing related works including [1] and [2].

• The comparison with R-DPO [2] was provided in Table 12. We thank the reviewer for pointing out the missing related work [1], which we empirically compared in the following table. We report the MT-Bench scores of performing our method on the Llama3-SFT checkpoint:

• We will add the above results to experiments and incorporate the following paragraph to related work:
  "Similar to our work, [1] also investigates how preference optimization can overlook qualitative aspects of responses. However, their focus is on overfitting to preference data, and they propose incorporating quality margins into the optimization objective. In contrast, our approach does not involve algorithmic modifications, but rather directly targets the limitations identified in Section 3. Our work also differs from [2], which introduces a constraint-based regularization term specifically aimed at mitigating verbosity bias."

Weakness 3: Lack of empirical comparisons with DPO variants such as IPO and alternative approaches that use quality scores.

In addition to DPO, we compared our method against 15 SOTA baselines. These include approaches that enhance DPO from various perspectives, such as IPO, as well as methods that incorporate quality scores during fine-tuning, including SteerLM, DPA, and MMPO. The results are presented in Figure 4 and Tables 10 and 12. For your convenience, we summarize the results below and will move Tables 10 and 12 to the main body of the manuscript.

If the reviewer has other baseline methods in mind, please let us know and we will be happy to include them as comparisons.

Question 1: How do the models perform on the flip task, i.e., evaluating the score of a given response similar to a type of generative verifier?

Our method, which is designed for direct preference optimization, is not directly applicable to training generative verifiers, which typically involve predicting reward tokens during training. That said, our approach is not incompatible with generative verifiers. For instance, at inference time—when the model is prompted to generate high-reward responses—the probability assigned to a given response can serve as an approximate measure of its quality. More generally, for LLM-as-a-Judge settings, reward-conditioned training can be applied to preference data that reflects the quality of judgments. By conditioning on the highest quality scores, the model exhibits its best judgment capabilities, rather than simply assigning high scores indiscriminately.

---

We hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments.

[1] Kim et al., ''Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback.''
[2] Park et al., ''Disentangling Length from Quality in Direct Preference Optimization.''

The only baselines are DPO and SPPO.

In addition to DPO and SPPO, we compared with 15 baselines in Figure 4 and Table 12.

Gains across LLMs are inconsistent. Marginal improvements on NLP benchmarks.

• The effectiveness of our method is demonstrated on 5 LLMs. It consistently offers improvements, with most gains substantial. Since our method's hyperparameters were not extensively tuned for each model, variability in performance gains is expected. Notably, even SOTA alignment methods [1, 2] reported inconsistent improvements across different models.
• Alignment tax [1, 3] can reduce common-sense QA performance. So we primarily evaluated on instruction-following benchmarks, where our method yields strong improvements while avoiding the alignment tax.

Claim of comparable performance between Half RA and RA.

• While we initially considered RA and Half RA comparable, each outperforming the other on at least one benchmark, we agree that this claim is not essential. We will remove it as the comparison with DPO already supports this ablation: fine-tuning on reward-augmented data yields better performance with half of the prompts and the same compute.
• We conducted main experiments across five models and selected one or two for ablation studies. Due to resource constraints, we were unable to run all 10 ablations on all 5 models.
• We extended this ablation to Llama, in addition to Qwen and Gemma. The results are consistent with our original findings.

A version of Figure 2 across all LLMs is provided in Table 9 but not referenced.

Figure 2 and Table 9 present the same results, both across all LLMs.

The gains are small for MTBench.

• Since we did not perform extensive hyperparameter tuning for each benchmark, it is expected that performance gains are modest on some benchmarks.
• On MTBench, the average gains obtained by our method are ~1.55 more than DPO. In comparison, all four models fine-tuned with SimPO [1] fail to outperform DPO on MTBench.

The analysis in Figure 3 should also be reported on O.O.D data.

We conducted additional experiments on HelpSteer2 and had similar observations as in Fig. 3:

No evidence or citation about how DPO behaves is provided.

• In introduction, we cited [4] as the first to identify the unlearning issue of DPO, and compared with it in related work and experiments.
• We offered empirical evidence in Fig. 3. We will also include the new ablation on HelpSteer2 at L384 and add pointers to these results in Sec. 1 and 3.

Limitations are partially addressed by PPO.

Our primary motivation is to address the limitations of direct alignment, instead of PPO that exhibit different limitations not covered here (see Sec. 5). We will include a comparison with Llama-3-PPO, which scores $21.27$ on AlpacaEval.

SFT on the augmented prompt + response is an important baseline.

Please refer to Fig. 4, where we compared with SOTA conditional SFT baselines including DPA and SteerLM.

The strategy for selecting the hyperparameters must be detailed.

We will add the following paragraph:
"We tune $\beta$ within $[0.001, 0.01, 0.1]$ and batch size within $[64, 128, 256]$. We find $\beta=0.01$ and batch size $128$ yield the overall best performance for DPO across models. Our method uses the same hyperparameters as DPO."

Additional experiments in Appendix are not referenced.

We will add references to these ablation headers in the experiments and move most of them to the main body as space permits.

The error bars for AlpacaEval should be reported.

Reporting error bars requires training at least three times more models, which was not feasible given our resource constraints. For similar reasons, most prior alignment works [1, 2] also report results from single runs.

Missing related works.

We will add the following paragraph to L262:
"Pang et al. (2024) addressed DPO’s tendency to reduce the probability of the chosen response by incorporating an NLL loss. In contrast, our work focuses on a different limitation of DPO—its tendency to overlook qualitative aspects of responses—and proposes a data relabeling approach that requires no algorithm changes. It also differs from conditional sequence modeling based on SFT (Chen et al. 2021, Lloret et al. 2024). Due to the lack of textual feedback in UF, we empirically compare with the reward feedback variants of Lloret et al. (2024), including SteerLM and DPA."

[1] Meng et al. ''SimPO.''
[2] Wu et al. ''SPPO.''
[3] Askell et al. ''Language Assistant as Alignment Laboratory.''
[4] Adler et al. ''Nemotron-4 340B.''

We thank the reviewer for identifying our work's soundness and technical contributions. Your valuable comments have greatly helped us improve our manuscript. Below are our specific responses to the raised questions:

Weakness 1 and Question 1: The authors directly apply DPO on the re-labeled dataset, rather than using the training objective introduced in the method.

We will incorporate the following clarifications—extending lines 150–219—in the revised manuscript. The training objective introduced in our method is $\max_{\pi\in\Pi}\mathbb{E}_ {x,g\sim\mathcal{D},y\sim \pi(\cdot\mid x, g)}[R(x,y, g)-\beta_0\text{KL}(\pi(\cdot\mid x,g)\| \pi_{\text{ref}}(\cdot\mid x,g))],$ where $\Pi$ denotes the class of all goal-conditioned policies $\pi(y\mid x,g)$ and the relabelling distribution for $g$ is $\mathbb{P}( g= r(x,y_w) \mid x,y_w,y_l) = P(g=(r(x,y_l) \mid x,y_w,y_l)=1/2.$ It has the closed-form solution as follows: $\pi_ R(y\mid x, g)\propto \pi_{\text{ref}}(y\mid x,g)\exp(\beta_0^{-1} R(x,y,g)).$ Using the reward reparameterization trick from DPO, the reward can be written as $R(x,y,g) = \beta_ 0 \log\pi_R(y\mid x,g) -\beta_0 \log\pi_{\text{ref}}(y\mid x,g) -Z_ R(x,g).$ Based on the relabeling distribution, we define the augmented preference dataset $\overline{\mathcal{D}}$ as $\{(x^i,\tilde{y} _ w^i=y_ w^i,\tilde{y}_ l^i = y_ l^i,g^i = r(x^i,y^i_ w))\}_ {i\in[M]}\cup\{(x^i,\tilde{y} _ w^i=y_ l^i,\tilde{y}_ l^i = y_ w^i,g^i = r(x^i,y^i_ l))\}_ {i\in[M]},$ which doubles the size of the original dataset $\mathcal{D}$. The resulting goal-conditioned DPO objective becomes:

$$\max_{R} \mathbb{E}_ {x,\tilde{y}_ w, \tilde{y}_ l, g\sim \overline{\mathcal{D}}}[\sigma(R(x,\tilde{y}_ w,g) - R(x,\tilde{y}_ l,g))] =\max_{\pi}\mathbb{E}_ {x,\tilde{y}_ w,\tilde{y}_ l,g\sim \overline{\mathcal{D}}}\Bigl[\sigma\Bigl(\beta_0 \log\frac{\pi(\tilde{y}_ w\mid x,g)}{\pi_{\text{ref}}(\tilde{y}_ w\mid x,g)}- \beta_0 \log\frac{\pi(\tilde{y}_ l\mid x,g)}{\pi_ {\text{ref}}(\tilde{y}_ l\mid x,g)}\Bigr)\Bigr].$$

This objective directly corresponds to our implementation, which applies DPO on the relabeled dataset.

Weakness 2: The experimental setups could be more detailed.

We have included our hyperparameters and prompts in Appendix B.1, which we list as follows. On 8 A100 GPUs, the training takes about 5-7 hours.

Question 2: Could you provide the prompt settings for the IRA experiments?

The prompts used in the IRA experiments are identical to those in the main experiments (as detailed in Appendix B.1). The only difference lies in the rescaling of reward values to the range $[1, 10]$ using the following linear transformation: $\max(\min(10*(\text{reward} - \text{low}) / (\text{high} - \text{low}), 10), 1)$, where high and low denote the maximum and minimum implicit rewards computed from a small subset of the training data.

Question 3: If possible, could you provide the code for your experiments?

We provided the code in the anonymous link: https://anonymous.4open.science/r/anonymous-9208-id.

Question 4: Figure 3 is quite striking but does not show whether the log probability for low-quality rejected responses also increases.

We report the log probabilities for low-quality rejected responses (with scores less than 5) in the following table:

It can be observed that the log probability of low-quality rejected responses for our method DPO (RA) has a similar scale as the vanilla DPO (UF) and is smaller than that of Qwen2-7B-Instruct.

Question 5: Relationship to traditional data-driven robust training.

We will incorporate the following discussions to our manuscript:
"Both our method and robust training techniques are motivated by enhancing the model generalization by leveraging augmented datasets that capture alternative outcomes. However, robust training methods typically focus on generic uncertainty or perturbations, which are not involved in our method. Instead, it concerns the limitations in direct alignment algorithms, such as the unlearning issue, by explicitly conditioning on quality scores to steer the model toward learning patterns associated with varying levels of response quality. Moreover, robust methods often emphasize worst-case scenarios or boundary conditions, which can lead to conservative generalization. In contrast, our method promotes generalization toward sparse, high-quality responses."

---

We hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments.

We thank the reviewer for identifying our work's soundness and technical contributions. The valuable comments have greatly helped us improve our manuscript. Below are our specific responses to the raised questions:

Weakness 1 and Question 2: SOTA works cited in the related work but not compared, such as recent works that improve on the DPO, e.g., RPO (Adler et al., 2024), and RLAIF line of works.

We have compared with 15 additional baselines in Figure 4 and Table 10, 12 in the Appendix, including RPO and other SOTA RLAIF methods. We listed the results below for your reference and will incorporate Tables 10 and 12 into the main body of the manuscript:

If there are other baselines you would like us to consider, please let us know and we would be happy to include them in our comparisons.

Weakness 2 and Question 1: LLM evaluators are typically not that stable when generating scalar rewards.

• The LLM or reward model (RM) judges are typically trained on human preference data, and the resulting reward scores reflect the expected preferences across annotators—akin to Elo scores in Bradley-Terry models.
• Compared to direct preference optimization methods such as DPO, our approach is more robust to the instability of scalar rewards. For instance, when only a sample-based estimate of the true preference is available or when using function approximators, it is common for high-quality rejected responses to be favored by more annotators. DPO, which attempts to maximize the reparameterized reward gap, may degrade the model in such cases. In contrast, our method is sensitive to varying response quality and effectively learns from the full spectrum of rewards.
• We further performed an ablation study to assess the robustness of our method to different scalar reward scales generated by LLM evaluators. Using the UltraFeedback dataset, which provides reward scores in the range of 1–10, we linearly rescaled the rewards to 1–5 and 1–100. We then applied our method to these modified datasets. We observed that our method is robust to the scale of the reward scores. The results are summarized in the following table:

Suggestion 1: Expand the method section with more proofs and guarantees on how the proposed direction can change the preference selection research direction.

We thank the reviewer for the helpful suggestion. In the revised version, we will include a proof sketch in Section 4, incorporating the key analysis and lemmas from Appendices A.5 and A.6. Additionally, we will highlight the observed limitations of DPO—such as those illustrated in Figure 3 and similar trends reported on the O.O.D. data HelpSteer2 below—to provide a more comprehensive discussion.

Question 3: How to identify the optimal goal reward value?

The optimal goal reward depends on the value range of the judge model and its scale as presented in the training prompt. For instance, the optimal goal is $1$ when using sigmoid-based reward models, or $5$ when using LLM judges that follow evaluation criteria with a maximum score of $5$. Both types of reward values can be rescaled, e.g., via linear transformations in the training prompt, as demonstrated in Appendix B.3.

---

We hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments.