SeRA: Self-Reviewing and Alignment of LLMs using Implicit Reward Margins

We are very grateful for your constructive comments. We have expressed your concerns as we understood them and provided answers as follows. If there is any concern that we may have misunderstood or if you have additional questions, we look forward to addressing them through further comments from you.

Q1. Additional literature review

A1. Thank you for pointing out the missing references [1-6]. We have expanded the literature review to address the over-optimization issue in LLM alignment. Regarding self-generated data, we have already included relevant discussions in the appendix; however, we have expanded on this further based on your suggestions. All updates can be found in the latest version of our paper.

Q2. Further extension to domains including coding, reasoning, and math

A2. Our evaluation benchmarks, including Evol-Instruction, Vicuna, and UltraFeedback, already contain coding, reasoning, and math tasks. We have reported detailed grading results for the sub-categories of code (both code generation and code debugging), reasoning, and math problems using Evol-Instruct, which explicitly categorizes each evaluation sample. The results below are for TinyLlama-1.1B, Pythia-2.8B, and Mistral-7B:

As shown in the table, SeRA achieves the highest performance in most categories, except for reasoning with Mistral, where it ranks second. These results demonstrate that SeRA is also effective across coding, reasoning, and math problems.

Q3. Further explanation for Remark 1 (or Theorem 1)

A3. To address the reviewer's concerns, we provide clarifications in two directions:

1. Explanation of Remark 1 (and Theorem 1): The purpose of Remark 1 (or Theorem 1) is to provide mathematical support for the empirical phenomena observed in Fig. 2, which motivated our use of the implicit reward margin (IRM). Theorem 1 indicates that the upper bound of the risk $f(x, y_w, y_l)$ is reduced when training on samples with clear preferences, as opposed to using a dataset that includes ambiguous samples. This helps prevent memorization of spurious features and reduces the gap between predicted and true probabilities, as illustrated by the second term on the RHS of the formula in Remark 1. These theoretical results align with our observations in Fig. 2.

2. Clarification on the BT Preference Model: While BT models handle preferences stochastically by assigning multiple labels to a single sample, practical implementations for DPO (and other DAAs not based on BT models) are typically deterministic. Due to labeling costs and efficiency considerations, we usually train on only a few, or even a single, preference pair per prompt. In practice, all annotations are treated as: $p(y_w > y_l \mid x) = 1$ for each sample $(x, y_w, y_l)$, meaning DPO treats all preference samples equally. Our theoretical results are valid under this practical setup, which aligns with previous empirical findings.

To further support our theoretical results, we evaluated the $R^2$ scores between GPT-4 scores vs. implicit rewards, as well as response length vs. implicit rewards, using varying selection ratios (0.6 to 1.0 in intervals of 0.1) to illustrate the relationship between IRM and spurious correlations.

From these results, we draw three key conclusions:

1. Even with fewer training iterations at a selection ratio of 0.7, the $R^2$ score (vs. GPT-4) remains comparable to higher ratios (0.8 to 1.0).
2. The $R^2$ score (vs. Length) consistently increases as the selection ratio and training iterations grow.
3. At a selection ratio of 0.6, both $R^2$ scores are lower than at 0.7, indicating that the LLM has not yet fully fit the dataset.

These findings support our claim that LLMs first learn explicit features (i.e., true preferences) before gradually memorizing spurious correlations (e.g., response length). Our use of IRM helps the model focus on true preferences while minimizing spurious correlations, supporting both mathematically and empirically, thereby justifying its use as a core component of our proposed method.

Q4. Minor suggestions regarding terminology clarification and benchmark details

A4. We modified "reward" to "implicit reward" in Algorithm 1 and added more descriptions for the benchmark in Appendix C.1, highlighted in magenta. Thank you for your suggestions.

Q5. Computation budget for SeRA

A5. The cost overhead introduced by SeRA is mainly divided into three parts: (1) sample selection for the off-policy dataset, (2) generation for preference bootstrapping, and (3) sample selection for the bootstrapped dataset from the trained LLM. The fine-tuning process remains computationally equivalent to DAAs. Below, we report the computational time for each component of SeRA:

Results were obtained using a machine with 4xA100 (40G) GPUs on TinyLlama-1.1B with 𝑘 = 0.7N and $\tilde{k}$ = 0.3N. For generation, we utilized vLLM [7], an advanced LLM inference framework. Overall, SeRA requires only 36% additional computational budget while achieving a 40%-65% performance gain (see Table 1 in the main manuscript) compared to off-policy training, with no extra labeling costs like on-policy training.

Q6. Discussion on dataset construction proposed in SPPO

A6. We would like to clarify that our experiments have already shown that increasing the number of self-generated samples can be beneficial (e.g., best-SeRA with $k=0.7N$ and $\tilde{k}=2N$, as presented in Table 1 and Fig. 7). This aligns with the reviewer's suggestion of using all possible preference pairs in the training dataset. However, as observed in Fig. 7, performance consistently improves up to 2N pairs, but starts to degrade beyond that, particularly at 3N where the number of responses is 3 (i.e., $R=3$ where $R$ is number of generations per single prompt) for SPPO dataset construction. This suggests that using all possible preference pairs is not always advantageous.

Additionally, while SPPO benefits from using more preference pairs, it relies on external reward models and introduces a different objective function, which diverges from our setup. The use of additional pairs may be effective in their framework, but may not be feasible in scenarios with limited budget constraints, which our method specifically addresses.

Q7. Clarification for UltraFeedback benchmark

A7. Yes, that’s correct. As mentioned in the main text (line 358) and the appendix (line 1074), we used the binarized UltraFeedback dataset, which includes a test split for evaluation.

References

[1] Coste et al., “Reward Model Ensembles Help Mitigate Overoptimization.” ICLR. 2024
[2] Liu et al., “Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer”. arXiv preprint. 2024
[3] Eisenstein et al., “Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking.” CoLM. 2024
[4] Liu et al., Direct Large Language Model Alignment Through Self-Rewarding Contrastive Prompt Distillation.” arXiv preprint. 2024
[5] Gao et al., “Scaling Laws for Reward Model Overoptimization.” ICML. 2023
[6] Rafailov et al., “Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms.” NeurIPS. 2024
[7] Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”. SOSP. 2023

Dear Reviewer AqA6,

In our response, we have addressed your questions by providing:

• An additional literature review on over-optimization for LLM alignment,
• Additional results for specific domains, including coding, reasoning, and math,
• An expanded description of Remark 1 and further experimental results to support our claims,
• An additional investigation into computational costs,
• A discussion of the dataset construction introduced in SPPO, and
• A clarification of the evaluation benchmark.

We hope our response provides additional clarity regarding your questions. We would appreciate it if you could confirm that you have read our response and share any further questions you might have before the discussion period ends.

Thank you for your time.

Sincerely,
The Authors

Dear Reviewer AqA6,

We hope our response and revised manuscript provided additional information regarding your questions. We would appreciate it if you could acknowledge that you have read our response and share any further questions or concerns before the discussion period ends.

If there are specific points you’d like us to clarify, please let us know, and we would be happy to elaborate to address your concerns.

If there are no further questions, we hope we have satisfactorily addressed your concerns and would be delighted if you might consider reflecting this in your re-evaluation.

Sincerely,
The Authors

Thank the authors for the replies and I will keep my positive scores.

We are very grateful for your constructive comments. We have expressed your concerns as we understood them and answered them as follows. If there is any concern that we may have misunderstood or if you have additional questions, we look forward to addressing them through further comments from you.

Q1. Takeaways from the theoretical perspective

A1. The purpose of Remark 1 (and Theorem 1) is to provide theoretical intuition that selecting samples with a large IRM reduces over-optimization to spurious correlations by clearly distinguishing chosen and rejected responses. As noted in our manuscript, the analysis aims to show how large IRM values improve sample selection, rather than covering every aspect of the algorithm. To address the reviewer’s concerns, our response is divided into two parts:

1. Connection with SeRA: Our theoretical result shows that selecting samples with a large IRM can lower the upper bound for misclassification of unseen samples, potentially leading to better generalization in distinguishing response pairs. This theoretical insight into the benefits of using IRM-based sample selection and preference data bootstrapping, manifests in the improved empirical performance of SeRA.

2. Consideration for multiple iterations: While our theorem primarily addresses the theoretical advantage of IRM-based sample selection (bounded risk), it does not focus on iterative refinement. Using IRM over multiple iterations, as introduced in Eqn. (5), might introduce unexpected model bias and performance degradation (see Fig. 5), making theoretical guarantees difficult, which we empirically resolve by introducing an ensemble of IRMs across iterations (Eqn. (7)).

Q2. Experimental results for further iterations

A2. To answer the question, we conducted additional experiments with further iterations, specifically from 2 to 5, evaluated on the UltraFeedback benchmark for both single and SFT performance. The experiments were performed on TinyLlama-1.1B with $k = 0.7N$ and $\tilde{k} = 0.3N$. The results are shown below:

The results suggest additional iterations may consistently improve performance (as performances for single grading are almost same across iterations 3 to 5), though the benefits diminish as iterations increase. We believe that future work could explore more scalable methods that continue to enhance performance as the number of iterations increases.

Q3. Discussion on uniform covering number for hypothesis classes

A3. It is challenging to compute the exact values for uniform covering numbers directly. However, we can derive an upper bound under certain conditions [1, 2]. For example, let $H$ be the set of $L$-Lipschitz functions with respect to the infinity norm, mapping from $[0, 1]^d$ to $[0, 1]$. In this case, the (uniform) covering number, $N_\infty(\epsilon, H, N)$, will be a function of $(L/\epsilon)^d$ as defined in Definitions 1 and 2. For both linear functions and neural networks, we can define a Lipschitz constant using activation functions such as ReLU or GeLU, which ensures that this covering number is finite.

To further clarify this concept, consider that if two responses are easily distinguishable through sample selection, the complexity of the hypothesis class is reduced compared to the case without selection. This is because the margin is larger, implying a smaller Lipschitz constant for the function class. Consequently, the (uniform) covering number is smaller with selection than without selection.

Q4. Extension of theoretical results to other DAAs

A4. Our theoretical results are specifically based on the fact that the implicit reward margin (IRM) can be directly represented in the logit values for probabilities in Bradley-Terry (BT) models within DPO. However, as noted in lines 348-350 in our manuscript, methods like IPO and SLiC-HF do not assume a preference model similar to BT, so the theoretical results in Remark 1 (or Theorem 1) cannot be directly extended to other DAAs. However, as demonstrated in Table 2 and Table 9, SeRA consistently performs well across IPO, SLiC-HF, and SimPO. Therefore, we believe it would be valuable future work to generalize our theoretical framework to apply to a broader range of DAA methods.

[1] Asymptotic Statistics, Aad van der Vaart. Cambridge. 1998
[2] Convergence of Stochastic Processes. David Pollard. Springer. 1984.

Dear Reviewer pPvp,

In our response, we have addressed your questions by providing:

• Descriptions regarding the takeaways of Remark 1 (i.e., the connection with SeRA and considerations for multiple iterations),
• Additional experimental results for further training iterations,
• An expanded description of the uniform covering number, and
• An additional explanation regarding the potential extension of Remark 1.

We hope our response provides additional clarity regarding your questions. We would appreciate it if you could confirm that you have read our response and share any further questions you might have before the discussion period ends.

Thank you for your time.

Sincerely,
The Authors

We are very grateful for your constructive comments. We have rephrased your comments as we understand them and provided corresponding answers. Please let us know if there are any misunderstandings or if you have any additional questions.

Q1. Comparison performance between SeRA and on-policy training with external reward models

A1. We previously reported a comparison between SeRA and OAIF [1], which uses on-policy training with external reward models (based on Mistral-7B), in Table 3. In these experiments, we evaluated TinyLlama-1.1B on the HH-RLHF [2] and TL;DR [3] datasets using a Mistral-7B-based reward model for on-policy training. As shown, SeRA achieved 66.1% on HH-RLHF and 62.7% on TL;DR, outperforming the on-policy counterpart which scored 64.6% and 59.5% respectively. These results demonstrate that SeRA can surpass on-policy training in this experimental setup.

Q2. Further explanation on how SeRA mitigates over-optimization

A2. As noted in the main manuscript (lines 238-245), DNNs initially learn explicit features (e.g., preference) but later memorize spurious features (e.g., sequence length). In Fig. 2, while both the first (no selection) and fourth (IRM selection) columns show similar correlations between implicit rewards and true preferences (GPT-4 scores), the first column had a much higher $R^2$ correlation (0.13) compared to the fourth (0.08). This suggests that the larger sample size in the first column allows more memorization of spurious features due to extended training and encountering ambiguous preference samples. These findings align with existing findings that longer training increases correlation with sequence length [4]. To mitigate this, we focused on learning from more distinct pairs using larger IRM values to reduce the risk of learning spurious correlations from ambiguous pairs.

To further support our theoretical results, we evaluated the $R^2$ scores between GPT-4 scores vs. implicit rewards, as well as response length vs. implicit rewards, using varying selection ratios (0.6 to 1.0 in intervals of 0.1) to illustrate the relationship between IRM and spurious correlations.

From these results, we draw three key conclusions:

1. Even with fewer training iterations at a selection ratio of 0.7, the $R^2$ score (vs. GPT-4) remains comparable to higher ratios (0.8 to 1.0).
2. The $R^2$ score (vs. Length) consistently increases as the selection ratio and training iterations grow.
3. At a selection ratio of 0.6, both $R^2$ scores are lower than at 0.7, indicating that the LLM has not yet fully fit the dataset.

These findings support our claim that LLMs first learn explicit features (i.e., true preferences) before gradually memorizing spurious correlations (e.g., response length). Our use of IRM helps the model focus on true preferences while minimizing spurious correlations, supporting both mathematically and empirically, thereby justifying its use as a core component of our proposed method.

Q3. Any other potential design for $r_t$ in Eqn. (7)

A3. An alternative for r_t ​ in Eqn. (7) is length-regularized (LR) reward shaping, which penalizes response length to reduce bias from implicit rewards [5]:

$r_{\text{LR}}(\mathbf{x}, \mathbf{y}; \alpha) = \beta \log \frac{\pi_\theta(\mathbf{y} \mid \mathbf{x})}{\pi_{\text{ref}}(\mathbf{y} \mid \mathbf{x})} - \alpha |\mathbf{y}|$

In our comparison for the third iteration DPO with TinyLlama-1.1B, while LR reward achieved 79.4% and 67.6% for Vicuna and Evol-Instruct, respectively, our proposed method achieved 85.8% and 72.3%, demonstrating superior performance over the LR alternative.

We note that the distincts margins in our case can capture other spurious correlations beyond length where ambiguity over the preference can make reward models/humans consider other aspects (such as length, style of writing, etc.) to give preference annotations.

[1] Guo et al., “Direct Language Model Alignment from Online AI Feedback.” arXiv preprint. 2024.
[2] Ganguli et al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.” arXiv preprint. 2022.
[3] Stiennon et al., “Learning to summarize with human feedback.” NeurIPS. 2020
[4] Rafailov et al., “Scaling laws for reward model overoptimization in direct alignment algorithms”. arXiv preprint. 2024
[5] Chen et al., “Bootstrapping Language Models with DPO Implicit Rewards.” arXiv preprint. 2024

Dear Reviewer FEME,

In our response, we have addressed your question by providing:

• A comparison with on-policy training using an external reward model
• An additional description of Remark 1, and
• An additional comparison with alternatives for implicit reward design.

We hope our response provides additional clarity regarding your questions. We would appreciate it if you could confirm that you have read our response and share any further questions you might have before the discussion period ends.

Thank you for your time.

Sincerely,
The Authors

Dear Reviewer FEME,

We hope our response and revised manuscript provided additional information regarding your questions. We would appreciate it if you could acknowledge that you have read our response and share any further questions or concerns before the discussion period ends.

If there are specific points you’d like us to clarify, please let us know, and we would be happy to elaborate to address your concerns.

If there are no further questions, we hope we have satisfactorily addressed your concerns and would be delighted if you might consider reflecting this in your re-evaluation.

Sincerely,
The Authors

Dear Reviewer FEME,

We sincerely hope our previous response has sufficiently addressed all your concerns. If you feel that your concerns have been resolved, we would be most grateful if you could kindly consider updating your evaluation accordingly.

With best regards,
Authors

Q4. Highlight for effectiveness of IRM on spurious correlation

A4. To further illustrate the relationship between IRM and spurious correlations, we evaluated the $R^2$ scores for GPT-4 scores vs. implicit rewards, as well as response length vs. implicit rewards, using varying selection ratios (0.6 to 1.0, in intervals of 0.1).

From these results, we draw three conclusions:

Even with fewer training iterations at a selection ratio of 0.7, the $R^2$ score (vs. GPT-4) remains comparable to higher ratios (0.8 to 1.0). The $R^2$ score (vs. Length) consistently increases as the selection ratio and training iterations grow. For a selection ratio of 0.6, both $R^2$ scores are lower than for 0.7, indicating that the LLM has not yet adequately fit the dataset.

These findings support our claim that LLMs initially learn explicit features (i.e., true preferences) before gradually memorizing spurious correlations (e.g., response length). Our use of the IRM helps the model focus on true preferences while minimizing spurious correlations, highlighting IRM as a core component of SeRA in addressing this issue.

Q5. Clarification for Theorem 1

A5. Theorem 1 provides mathematical support for the empirical phenomena in Fig. 2, motivating our use of IRM. It shows that selecting samples with a large IRM reduces over-optimization to spurious correlations by clearly distinguishing a chosen response from a rejected response. Precisely, Theorem 1 supports that training on clear preferences lowers the upper bound of the risk $f(x, y_w, y_l)$ compared to using datasets with ambiguous samples. This theoretical support for using IRM in sample selection for both off-policy and on-policy sampling manifests in the effectiveness of SeRA across DAAs, across datasets, etc.

References

[1] Zhang et al., “Reward-Augmented Data Enhances Direct Preference Alignment of LLMs”. arXiv preprint. 2024
[2] Rosset et al., “Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences”. arXiv preprint. 2024
[3] Guo et al., “Direct Language Model Alignment from Online AI Feedback”. arXiv preprint. 2024
[4] Zhong et al., “DPO Meets PPO: Reinforced Token Optimization for RLHF.” ICML 2024 Workshop MFHAIA. 2024
[5] Cui et al., "UltraFeedback: Boosting Language Models with High-quality Feedback." ICML. 2024
[6] HuggingFace., "https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized". 2023

We are very grateful for your constructive comments. We have expressed your concerns as we understood them and provided answers as follows. If there is any concern that we may have misunderstood or if you have additional questions, we would be happy to address them in further comments.

Q1. Difference from existing works

A1. We would like to clarify that our work differs from existing approaches in three key ways:

1. Our focus is on improving the efficiency of any direct alignment algorithms (DAAs) within a constrained cost budget, without relying on external reward models. We achieve this through sample selection and preference bootstrapping based on IRM of policy LLMs themselves rather than through simple relabeling. We believe that our goal significantly differs from the works mentioned by the reviewer, that either use external reward models [1] which require a lot re-labeling cost [2, 3] or provide token-wise reward to PPO training [4]. ($\color{magenta}{\text{Moreover, [1] was published on 10th October, 2024 on arxiv, which is after the paper submission deadline.}}$)
2. We provide a rationale for the effectiveness of implicit rewards (IRM) through both empirical (Fig. 1 & Fig. 2) and theoretical (Remark 1) evidence, showing that samples with a large IRM can mitigate memorization of certain spurious correlations unrelated to the preference function such as longer response lengths. Moreover, our IRM-based sample selection has been shown to be more effective than other types of reward models, including GPT-4 or the log-probability of a reference model, as shown in Fig. 1.
3. Unlike previous approaches that focus solely on techniques like DPO or PPO, we propose a complementary approach that can be applied to any DAA and showcase its effectiveness against and alongside IPO, SLiC, and even SimPO which utilize different definitions of implicit reward terms.

[1] Zhang et al., “Reward-Augmented Data Enhances Direct Preference Alignment of LLMs”. arXiv preprint. 2024
[2] Rosset et al., “Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences”. arXiv preprint. 2024
[3] Guo et al., “Direct Language Model Alignment from Online AI Feedback”. arXiv preprint. 2024
[4] Zhong et al., “DPO Meets PPO: Reinforced Token Optimization for RLHF.” ICML 2024 Workshop MFHAIA. 2024 \

Q2. Clarification for role of implicit reward margin

A2. To address the reviewer's concerns, our response is two-fold:

1. While our approach is in line with existing works mentioned by the reviewer, we have several important distinctions. First, we reiterate that we utilize the implicit reward margin both as a data selection metric for the offline dataset (Sec 3.2; lines 191-196) and for obtaining labeled pairs during data bootstrapping (Sec 3.3). Second, our reward computation considers an ensemble of models based on iterative improvement of the policy model. This approach is consistent with existing work, including those mentioned by the reviewers, as well as other selection baseline comparisons depicted in Fig. 1 and Fig. 2.
2. As shown in Table 3 of our manuscript, we compared SeRA with a popular baseline using an external (Mistral-7B-based) reward model for TinyLlama-1.1B training (i.e., OAIF [3]). SeRA achieved 66.1% on HH-RLHF and 62.7% on TL;DR, outperforming OAIF, which scored 64.6% and 59.5%, respectively.

Overall, we believe our use of the implicit reward for both selection metrics and labeling is well-founded, supported by both theoretical intuition and empirical evidence, leading to strong performance in our experiments.

Q3. Addressing concerns regarding superiority of SeRA is from strong SFT models

A3. To address the reviewer's concerns, our response is twofold:

1. SeRA’s performance is not driven by strong base models. First, we want to clarify that $\color{magenta}{\text{both original and binarized versions of UltraFeedback labeling was done using GPT-4, not GPT-3.5 [5, 6].}}$ In contrast, our base models, as shown in the Alpaca Eval results in Table 1 in the main manuscript for TinyLlama and Pythia SFT, are significantly weaker than GPT-4 and even GPT-3.5. Therefore, SeRA’s superiority cannot be attributed to the strength of its base models.
2. We have already reported results using iterative DPO on UltraFeedback, which serves as a variant of SeRA without sample selection or bootstrapping (see Fig. 4, leftmost points of solid lines). For clarity, we summarize those results below:

Based on these results, we believe that SeRA’s superiority does not stem from using stronger SFT models.

[5] Cui et al., "UltraFeedback: Boosting Language Models with High-quality Feedback." ICML. 2024
[6] HuggingFace., "https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized". 2023

Dear Reviewer SFx5,

In our response, we have addressed your questions by providing:

• A clear differentiation from existing works,
• A clarification of the role of the IRM,
• An additional explanation that strong SFT models are not the primary reason for SeRA's superiority,
• A highlight of the effectiveness of the IRM in addressing spurious correlations, and
• An additional clarification regarding Remark 1.

We hope our response provides additional clarity regarding your questions. We would appreciate it if you could confirm that you have read our response and share any further questions you might have before the discussion period ends.

Thank you for your time.

Sincerely,
The Authors

Dear Reviewer SFx5,

We hope our response and revised manuscript provided additional information regarding your questions. We would appreciate it if you could acknowledge that you have read our response and share any further questions or concerns before the discussion period ends.

If there are specific points you’d like us to clarify, please let us know, and we would be happy to elaborate to address your concerns.

If there are no further questions, we hope we have satisfactorily addressed your concerns and would be delighted if you might consider raising in your evaluation.

Sincerely,
The Authors

The authors' comments have addressed most of my concerns, and I have increased my score to 6. I hope the reviewers can incorporate the above discussions into the new iteration of the manuscript.