Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization

W5: The authors should explore how the two threshold hyperparameters, $V_\text{min}$, $\epsilon_\beta$, affect the method's performance.

Reply to W5: We sincerely thank you for your valuable question. Completing the experiments related to these two hyperparameters indeed enhances the contribution of our paper. Below, we provide a detailed explanation of the hyerparameter selection, experimental resulsts, and analysis.

• $V\_\text{min}$:

This hyperparameter is designed to suppress the $V$-values corresponding to OOD actions, necessitating that $V_\text{min}$ be lower than all possible $V$-values.

In general, $V_\text{min}$ can be determined by calculating the minimum $V$-value, such as $V_\text{min} = \frac{1}{1-\gamma} r_\text{min}$, where $r_\text{min}$ denotes the minimum value of the reward function. Specifically, for RLHF of LLMs, where only the final token receives a nonzero reward, we have $V(s) = \mathbb{E}\_{\tau \sim \pi}[\gamma^{\|\tau\|} r(\tau) \mid s\_0 = s] \geq \min(0, r\_\text{min}) = V_\text{min}$.

On the other hand, empirical evidence suggests that as long as $V_\text{min}$ remains smaller than all possible $V$-values, the performance of BSPO is not sensitive to the $V_\text{min}$. To evaluate this, we conducted experiments under the condition $r_\text{min} = -10$, testing a range of $V_\text{min}$ values: $V_\text{min} = -10, -15, -20, -25$. The resulting experimental outcomes are summarized in the figure below:

Robustness of Vmin

We observe that, across different values of $V_\text{min}$, the training curves of BSPO exhibit similar behavior, consistently improving both the proxy reward and the gold reward while effectively addressing the issue of over-optimization. These results demonstrate the robustness of the $V_\text{min}$ selection.

• $\epsilon_\beta$:

Since we use a parameterized model to predict the behavior policy. Thus, to prevent modeling errors, an infinitesimal value $\epsilon_\beta$ is used as a threshold, instead of 0, to determine whether a response is supported.

To analyze the sensitivity of the BSPO algorithm to $\epsilon_\beta$, we conducted experiments with a range of values: $\epsilon_\beta = 1e-3, 1e-4, 1e-5, 1e-6, 1e-7$. The experimental results are presented in the figure below:

robustness of eps

Our observations indicate that for $\epsilon_\beta = 1e-4, 1e-5, 1e-6$, BSPO mitigates reward over-optimization and identifies the optimal policy. However, the performance for $\epsilon_\beta = 1e-3$ and $\epsilon_\beta = 1e-7$ shows slower improvement. This behavior may result from excessively large $\epsilon$ values, which can incorrectly classify some actions as behavior-unsupported, or excessively small $\epsilon$ values, which may fail to account for certain OOD actions. In conclusion, BSPO is robust to $\epsilon_\beta$ within a reasonable range, but extreme values should be avoided.

W6: Prior work [2][3] considers a noisy label setting, where the over-optimization problem is more severe. Can BSPO be effective in this challenging setting?

Reply to W6: Thank you for your valuable suggestion. In response, we have conducted a new set of experiments with larger noise ratio. Specifically, we modified the training dataset of proxy model by replacing 15% of the samples with noisy data and conducted experiments under the same settings described in the paper.

The results of these experiments can be found at the following link: Noise experiment.

We observed the following phenomena:

• The inclusion of additional noisy data results in a decrease in the performance of all algorithms when evaluated on the gold reward, compared to the Figure 3 of the paper.
• Despite this, BSPO consistently mitigates reward over-optimization and achieves better performance than the baseline methods, demonstrating its robustness to noise.

We hope the experimental results addresses your concerns and further validate the effectiveness of BSPO. Should you have any additional questions, we welcome further discussion.

W7: Typos: In Eq 36, the use of Q function is not correct.

Thank you very much for pointing out this typo. We will correct it in the latest manuscript.

Finally, we express our sincere gratitude for the insightful questions and suggestions, which have been helpful to our work and will be integral to our revised manuscript. If our efforts can address your concerns, we sincerely hope you will recognize our work.

W2: There seems to be a problem in the proof of Corollary 1. The authors suggest that optimizing the objective in Eq 23 forces the learned policy to choose behavior-support actions. However, there are cases where all actions are OOD for a given prompt $s$, which could inevitably introduce OOD actions. This situation can arise when the SFT model and reward training use different datasets. Thus, the argument may not hold

Reply to W2: Thank you for your valuable questions. We are glad to discuss this issue with you.

Theoretically, Corollary 1 is valid because the ID of the reward model is defined as the next-token distribution over the reward training dataset by a behavior policy $\beta$. For a LLMs with a finite vocabulary, the condition $\sum_{a \in \mathcal{A}} \beta(a|s) = 1$ always holds. Consequently, for any $s \in \mathcal{S}$, there exists at least one $a$ such that $\beta(a|s) > 0$. Therefore, given a specified behavior policy, a behavior-support action that satisfies the conditions of Corollary 1 can always be identified.

However, the situation you mentioned is indeed worth discussing. In the limitations section of our paper, we emphasized that our algorithm primarily focuses on mitigating the issue of reward over-optimization during the RL phase, which arises due to OOD responses in reward evaluation. Thus, if the prompt set used for RL training includes prompts outside the distribution of the reward model's training prompts, it may lead to unforeseen biases in the behavior model. Nevertheless, we remain optimistic about addressing this issue for the following reasons:

• In typical scenarios, such as those described in [1][2], the prompt sets used for RL training and reward model training are carefully controlled and are drawn from the same distribution. The presence of OOD prompts during RL training negatively impacts most algorithms, making it essential to avoid such situations.
• The behavior model is built upon a pretrained LLM that has undergone extensive pretraining on large-scale datasets and fine-tuning on the same next-token prediction task. Consequently, it exhibits a degree of generalization capability.

Based on this discussion, we think this is a good future work to explore the data distribution shift between the reward model phase and the RL phase. In the revised manuscript, we will incorporate additional discussion of OOD prompts within the Limitations section. We hope our explanation addresses your concerns. Should you have any further questions, we are open to continued discussion.

References:

[1] Ouyang et al. Training language models to follow instructions with human feedback.

[2] Bai et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.

W1: The paper claims that prior constraint optimization methods "only suppress overestimation in the ID region.", which I disagree. The worst-case optimization method with ensemble has been shown to mitigate over-optimization for OOD actions. During PPO training, a worst-case reward can also penalize OOD responses generated by the trained LLM. The authors should provide a more accurate comparison of BSPO with previous methods.

Reply to W1: Thank you very much for highlighting this question. In fact, due to space limitations in the Introduction, we combined the descriptions of uncertainty quantifier methods and ensemble methods, which may result in a lack of clarity and potential misunderstanding. Thus, we provide a more detailed explanation here and will refine the wording in the revised manuscript to improve clarity.

• Uncertainty quantifier methods[1]: These method leverage generative models to directly predict uncertainty. However, as highlighted in prior work [2], these methods face challenges in accurately predicting uncertainty in OOD regions without additional data incorporated.
• Ensemble methods [3]: As you mentioned, ensemble methods can indeed mitigate reward over-optimization to some extent, especially in the ID region. However, recent findings [4] indicate that in the RLHF of LLMs, ensemble methods may still face challenges, such as consistent reward overestimation across multiple models, leading to over-optimization issues. In summary, reward model ensembles mitigate but do not eliminate reward hacking [4]. Additionally, ensemble approaches necessitate training multiple branches, resulting in higher computational and memory costs.

We hope this clarification addresses your concerns and provides a more accurate comparison. We will refine the wording in the revised manuscript to improve clarity. If you feel that we have not explained this point clearly, we welcome further discussion.

References:

[1] Zhang et al. Mitigating reward overoptimization via lightweight uncertainty estimation.

[2] Nalisnick et al. Do deep generative models know what they don’t know?

[3] Coste et al. Reward model ensembles help mitigate overoptimization.

[4] Eisenstein et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking.

Thank you very much Reviewer for your response. We appreciate the opportunity to address your two additional questions.

Regarding W2, I am curious whether there is any underlying assumption about state coverage in the proof; if there is, it would be better to explicitly state it. Additionally, the theoretical analysis appears to have a gap when compared to realistic situations, as we would prefer our policy to generalize to unseen prompts during PPO training to more effectively leverage the generalization capabilities of reward models.

Thanks for the reminder about state coverage. As outlined in our Preliminaries section, in token-level MDP, the state is defined by the prompt $x \in \mathcal{X}$ and the generated action sequences $a_i \in \mathcal A, i=0,1,2,\cdots$. In our study, the focus is on the shift in action distribution, which leads to reward over-optimization. Consequently, the assumption that requires further clarification in the Analysis section is that prompts are identically distributed during both the reward learning phase and the RL phase. We will explicitly state it in latest maniscript.

The reasoning behind this assumption is as follows:

1. Similar to the need for caution when addressing OOD actions, we argue that conversation data containing OOD prompts is more challenging for the reward model to evaluate accurately. Since the prompt set in RLHF is predefined and controllable, excluding OOD prompts directly is a more practical approach. Indeed, in many RLHF implementations [1][2], the prompts used for reward learning and RL are identically distributed—or the former explicitly includes the latter.
2. Throughout the paper (e.g., in the Abstract, Introduction, and Limitations sections), we clearly stated that our research focuses on the issue of reward over-optimization caused by training-time shifts in the token (action) distribution of a LLM, which results in OOD responses relative to the reward model. As noted in the Related Work section, research on the generalizability of reward models is parallel to our work and can be used in a composite way.

Secondly, regarding the theoretical and practical gap, we believe it can be addressed for the following reasons:

1. As we mentioned above, in prior practical work [1][2], the prompts for reward learning and RL are typically i.i.d. In this case, our theoretical setting aligns with reality.
2. In cases where a reward model must generalize to OOD prompts, it becomes necessary to establish a mechanism that determines whether the reward model can correctly evaluate subsequent responses. Several prior studies have highlighted that the generalizability of reward models largely depends on the language capabilities of their pre-trained foundations [3][4]. Our proposed ScoreLM approach retains language capabilities while modeling behavior distribution using the same hidden state. Therefore, our method is well-positioned to complement ongoing research on reward model generalization. This could represent a promising and valuable direction for future work.

Reference

[1] Ouyang et al. Training language models to follow instructions with human feedback. OpenAI. (Section 3.2)

[2] Bai et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Anthropic. (Appendix B.1)

[3] Lambert et al. Evaluating Reward Models for Language Modeling

[4] Yang et al. Regularizing hidden states enables learning generalizable reward model for llms.

For W5, as the authors note, "for $\epsilon_\beta=1e-4, 1e-5, 1e-6$, BSPO mitigates reward over-optimization and identifies the optimal policy." However, I have concerns that the small values of $\epsilon_\beta$ may not be robust to numerical noise, particularly during model training, which can lead to instability.

In fact, as stated in the paper (lines 1186–1187), it is precisely because of numerical noise considerations that we introduce $\epsilon_\beta$ rather than strictly following the theory and comparing the probability of an action to 0. Specifically, $\epsilon_\beta$ serves as a tolerance threshold for numerical noise in probabilistic predictions. While this approach differs from the strict theoretical formulation, it enhances the stability and reliability of the algorithm in practical applications.

Furthermore, we validate its robustness through various experiments:

1. As shown in our ablation study (Figure 4(b)), the prediction of behavior-unsupported actions aligns with the occurrence of reward over-optimization.
2. From our experiments addressing prior W5 responses, we observe that $\epsilon_\beta$ remains robust across a sufficiently large range.

We sincerely hope that above responses will address your concerns and help to recognize our work. If you have any further questions, we would be glad to discuss them with you

W3: As noted in the limitations, significant differences may exist between human preferences and model-predicted preferences, underscoring the importance of human evaluation for generated responses. In cases where human judgment is not feasible, using an LLM-as-a-judge (please refer to Reference) could serve as a practical alternative for conducting pseudo-human evaluation.

Reply to W3: Thank you for your constructive feedback on our paper. We sincerely appreciate the opportunity to refine our work based on your suggestions.

Following your suggestion, we also explored using GPT-4o as evaluators. However, a key issue remains: LLMs cannot provide global scalar scores in a timely manner to reflect over-optimization during training. To address this, we considered storing policy checkpoints throughout training. By comparing these checkpoints on the test set to compute the win rate and fitting an ELO score [1] as a global measure, we can effectively observe the over-optimization phenomenon.

Below, we provide a summary of our implementation and results:

Implementation Details: Since it is challenging to assign a scalar score to a response by human evaluation, an effective approach uses comparing the generated responses from different checkpoints on a evaluation dataset. Specifically, we train a 7B reward model (based on Alpaca-7B) on a preference dataset (UltraFeedback), achieving a test set accuracy of 82.94%. This reward model was used to conduct BSPO and baselines (PPO, WCO) training.

To quantify evaluation results, we compared the checkpoints on the test set to calculate the win rate between checkpoints and fit an ELO score [4] as the scalar evaluation metric. The ELO update formula is as follows:

Here, $S_{AB}$ represents the win rate of Model A over Model B on the test set, $K$ is the update coefficient, and $R_A$ and $R_B$ are the ELO scores of Models A and B, respectively.

Experimental results:

Table 1. Win rate between different checkpoints:

Table 2: The ELO scores for different checkpoints:

The curves of proxy reward and elo score in training see link: Non-Synthetic Setting

Our experiments reveal interesting insights: We observed that while the proxy reward exhibits consistently increases across the three approaches, their performance under human evaluation reveals differences:

• Naïve PPO shows performance improvement at 79 steps but suffers a severe decline by 158 steps, highlighting the issue of reward over-optimization.
• WCO maintains stable human evaluation performance across both steps, indicating its effectiveness in mitigating over-optimization. However, it does not identify the optimal solution as effectively as BSPO.
• BSPO demonstrates continuous improvement in both proxy reward and human evaluation, indicating its capability to address over-optimization while identifying optimal policies.

We hope these experimental results address your concerns and strengthen our paper's contributions to the field of RLHF.

References:

[1] Askell et al. A General Language Assistant as a Laboratory for Alignment

W1: The experimental results presented in this paper are demonstrated exclusively on the UltraFeedback dataset, making it essential to evaluate the generalizability of the proposed method by conducting experiments on additional benchmark datasets.

Reply to W1: We sincerely appreciate your valuable comments, which have greatly helped us improve the paper. In response, we have conducted additional experiments using the AlpacaFarm dataset [1], as detailed in the revised manuscript. In this experiment, our gold reward model, based on Llama3-8B, is trained on a combined preference dataset consisting of UltraFeedback and a 20k preference dataset annotated by GPT-4 from AlpacaFarm. The gold model with the best performance over three training epochs is selected. For the proxy reward model, we utilized TinyLlama, training it on the 20k re-annotated AlpacaFarm dataset. This proxy model is subsequently used to train PPO, WCO, and BSPO. The results are presented in the figure below.

Alpaca Farm

Due to the relatively smaller size of the AlpacaFarm dataset, the accuracy of the ScoreLM proxy model is lower compared to when using the UltraFeedback dataset. This limitation results in an earlier occurrence of the roo phenomenon in both PPO and WCO. However, only BSPO shows continuous improvement in both proxy rewards and gold rewards, highlighting its ability to mitigate over-optimization while identifying optimal policies effectively.

[1] Bubois et al. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

W2: The paper emphasizes the concept of the "In-Distribution (ID) region of the reward training dataset" as a key to avoiding reward over-optimization, with BSPO’s foundational idea built on this concept. Given this emphasis, it would have been helpful to see a performance comparison with DPO-based methods, which directly utilize the ID region of reward training datasets (e.g., win response $y_w$ and lose response $y_l$), to further contextualize BSPO’s effectiveness.

Q3: As noted in the Weakness section, is there a specific reason for not comparing BSPO with direct preference optimization methods like DPO, KTO, and CTO? In what ways do you consider BSPO to be superior to DPO-related methods?

Reply to W2,Q3: We sincerely thank the reviewer for reminding us of the relationship between our work and DPO-based methods.

In the initial version of the paper, we did not include DPO-based methods in the comparison because we focused primarily on RL-based algorithms. While DPO-based methods also utilize preference data, they have a completely different training framework than RL-based methods. As you mentioned, DPO-based methods do not use a reward model during training, relying instead on the preference dataset. Therefore, they are not subject to the reward over-optimization problem.

DPO-based vs. BSPO (RL-based): We believe the following is a plausible discussion for the comparison between BSPO and DPO-based methods:

• As demonstrated in Theorem 3, although we apply regularization to the value of OOD actions to avoid overestimation, BSPO can still converge to the same optimal solution for the ID region as the standard value. BSPO retains the exploration capability inherent in RL-based methods. Therefore, the comparison between BSPO and DPO-based algorithms primarily reflects the broader distinction between RL-based and DPO-based approaches.
• As reported in [1], compared to RL-based methods, DPO-based methods have several limitations: over-fitting to the training dataset (a different manner of over-optimization), generating a biased policy that favors OOD responses, and sensitivity to the quality of preference data.

Based on the above differences, we did not previously discuss the DPO-based methods. However, following the reviewer’s suggestion, we recognize that it is valuable to compare BSPO with DPO-based methods. The experimental results are shown below.

Experimental Results: The win rate of DPO results compared to other algorithms is shown in the table below:

Our experiments reveal interesting insights:

• Both DPO exhibit improvements compared to the original Alpaca-7B model.
• The performance of PPO training is relatively poor due to severe reward over-optimization issues. In contrast, DPO do not rely on the reward model during training, thereby avoiding reward over-optimization problems and outperforming PPO.
• Nevertheless, because DPO lacks the ability to explore new responses and depend heavily on the quality of the training dataset (similar to supervised learning) [1], its performance falls short of RL-based BSPO.

[1] Xu et al. Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study

Thank you very much for your response and for recognizing the value of our work. However, we would like to clarify a minor misunderstanding. We apologize for lack of clarity in our previous reply. As outlined in our revised manuscript’s Appendix D.5, we utilized GPT-4o, not Alpaca-7b, as the LLM for evaluation purposes. Specifically, Alpaca-7b was used as a proxy model for training RL experiments, not for evaluation.

In response to your new suggestion, we will further enhance the credibility of our experimental results by incorporating evaluations using additional LLMs. For instance, we have included an evaluation conducted with claude-3-5-sonnet-20240620, and the results are presented in the following table:

Table 2: The ELO scores for different checkpoints:

Finally, we sincerely appreciate the time and effort you have devoted to reviewing our paper.

W1: The paper primarily focuses on the synthetic set for evaluating reward over-optimisation, the alignment of the reward with real human annotators’ evaluation remains insufficiently validated.

Reply to W1: Thank you for your constructive feedback on our paper. We sincerely appreciate the opportunity to refine our work based on your suggestions. Studying reward over-optimization requires continuous evaluation during the training and effective comparison with the proxy reward. However, human evaluation is not only expensive but also unable to provide timely scalar feedback. As a result, to the best of our knowledge, nearly all prior works [1,2,3] rely on synthetic settings.

The differences between synthetic settings and real-world scenarios are an important concern. In response, we have conducted additional experiments in non-synthetic environments. Below, we provide a summary of our implementation and results:

Implementation Details: Since it is challenging to assign a scalar score to a response by human evaluation, an effective approach uses comparing the generated responses from different checkpoints on a evaluation dataset. Specifically, we train a 7B reward model (based on Alpaca-7B) on a preference dataset (UltraFeedback), achieving a test set accuracy of 82.94%. This reward model was used to conduct BSPO and baselines (PPO, WCO) training.

To quantify evaluation results, we compared the checkpoints on the test set to calculate the win rate between checkpoints and fit an ELO score [4] as the scalar evaluation metric. The ELO update formula is as follows:

Here, $S_{AB}$ represents the win rate of Model A over Model B on the test set, $K$ is the update coefficient, and $R_A$ and $R_B$ are the ELO scores of Models A and B, respectively.

Experimental results:

Table 1. Win rate between different checkpoints:

Table 2: The ELO scores for different checkpoints:

The curves of proxy reward and elo score in training see link: Non-Synthetic Setting

Our experiments reveal interesting insights: We observed that while the proxy reward exhibits consistently increases across the three approaches, their performance under human evaluation reveals differences:

• Naïve PPO shows performance improvement at 79 steps but suffers a severe decline by 158 steps, highlighting the issue of reward over-optimization.
• WCO maintains stable human evaluation performance across both steps, indicating its effectiveness in mitigating over-optimization. However, it does not identify the optimal solution as effectively as BSPO.
• BSPO demonstrates continuous improvement in both proxy reward and human evaluation, indicating its capability to address over-optimization while identifying optimal policies.

We hope these experimental results address your concerns and strengthen our paper's contributions to the field of RLHF.

References:

[1] Gao et al. Scaling Laws for Reward Model Overoptimization

[2] Coste et al. Reward Model Ensembles Help Mitigate Overoptimization

[3] Eisenstein et al. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

[4] Askell et al. A General Language Assistant as a Laboratory for Alignment

Thank you for your response. However, it seems there is still a misunderstanding that we would like to address. Since you did not previously express concerns regarding "additional stability," we were unaware that this aspect required further clarification.

The term "additional stability" refers to the fact that the V-value estimation is more stable than the Q-value estimation in the actual implementation. In fact, it is precisely because LLM has a large action space that the use of V value is more favorable.

The reasons for this are as follows:

1. Lower Complexity: V-value focuses solely on the expected return of a state, whereas Q-value includes the additional complexity of evaluating state-action pairs. LLMs typically operate in massive action spaces，and most of the actions are difficult to access on the same state, which makes the data for training Q-values insufficient.

2. Lower Variance: Estimating Q-values can involve higher variance because it depends on both the immediate reward and the optimality of subsequent actions [1]. V-values, which summarize the expected return from a state without needing action-specific granularity, tend to have lower variance, making them more stable during training. However, Q-values estimate rely on specific actions that may be sampled infrequently, leading to higher variance due to fewer samples.

3. Easy to Integrate: In RLHF, human feedback is often used to train a reward model that evaluates states (e.g., whether a generated response aligns with human preferences). A state-based value function (V-value) naturally aligns with this reward model, simplifying integration and improving performance.

As a result, the RLHF [2][3][4][5] for almost all LLMs uses V network as the value function estimate. We do not cite this as a reason why we outperform the baselines algorithm; in fact, we simply give matching implementations of BSPO that use the V-value version.

Finally, we wish to express our heartfelt regret that a misunderstanding overshadowed our hard work. We sincerely hope that this response will encourage a fresh re-evaluation to our work, and we extend our deepest gratitude for your consideration.

Reference:

[1] Sutton, et al. Reinforcement Learning: An Introduction (2nd ed.).

[2] Ouyang et al. Training language models to follow instructions with human feedback. OpenAI.

[3] Bai et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Anthropic.

[4] Kaufmann et al. A Survey of Reinforcement Learning from Human Feedback

[5] Casper et al. Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

We sincerely thank you for recognizing our work and for raising our score. It is an honor for us to address your concerns, and your valuable feedback will be incorporated into the final revised version.

Regarding your concerns about the Q and V critic networks, we will revise the manuscript to provide a more precise description, such as clarifying their equivalence in BSPO during LLM training. It is important to note that this part pertains to the implementation of our algorithm. We do not cite it as a reason for outperforming baseline algorithms; rather, we simply present matching implementations of BSPO using the V-value version.

Once again, we greatly appreciate the time and effort you have dedicated to reviewing our paper. Your insightful comments are highly valuable and will significantly contribute to improving our work.

W2: The experimental results lack an ablation study on the sensitivity of the parameter $\epsilon_\beta$.

Q1: How do you select the value of $V_\text{min}$ and $\epsilon_\beta$?

Reply to W2,Q1: We sincerely thank you for your valuable question. Complating the experiments related to these two hyperparameters indeed enhances the contribution of our paper. Below, we provide a detailed explanation of the hyerparameter selection, experimental resulsts, and analysis.

• $V\_\text{min}$:

This hyperparameter is designed to suppress the $V$-values corresponding to OOD actions, necessitating that $V_\text{min}$ be lower than all possible $V$-values.

In general, $V_\text{min}$ can be determined by calculating the minimum $V$-value, such as $V_\text{min} = \frac{1}{1-\gamma} r_\text{min}$, where $r_\text{min}$ denotes the minimum value of the reward function. Specifically, for RLHF of LLMs, where only the final token receives a nonzero reward, we have $V(s) = \mathbb{E}\_{\tau \sim \pi}[\gamma^{\|\tau\|} r(\tau) \mid s\_0 = s] \geq \min(0, r\_\text{min}) = V\_\text{min}$.

On the other hand, empirical evidence suggests that as long as $V_\text{min}$ remains smaller than all possible $V$-values, the performance of BSPO is not sensitive to the $V_\text{min}$. To evaluate this, we conducted experiments under the condition $r_\text{min} = -10$, testing a range of $V_\text{min}$ values: $V_\text{min} = -10, -15, -20, -25$. The resulting experimental outcomes are summarized in the figure below:

Robustness of Vmin

We observe that, across different values of $V_\text{min}$, the training curves of BSPO exhibit similar behavior, consistently improving both the proxy reward and the gold reward while effectively addressing the issue of over-optimization. These results demonstrate the robustness of the $V_\text{min}$ selection.

• $\epsilon_\beta$:

Since we use a parameterized model to predict the behavior policy. Thus, to prevent modeling errors, an infinitesimal value $\epsilon_\beta$ is used as a threshold, instead of 0, to determine whether a response is supported.

To analyze the sensitivity of the BSPO algorithm to $\epsilon_\beta$, we conducted experiments with a range of values: $\epsilon_\beta = 1e-3, 1e-4, 1e-5, 1e-6, 1e-7$. The experimental results are presented in the figure below:

Robustness of eps

Our observations indicate that for $\epsilon_\beta = 1e-4, 1e-5, 1e-6$, BSPO mitigates reward over-optimization and identifies the optimal policy. However, the performance for $\epsilon_\beta = 1e-3$ and $\epsilon_\beta = 1e-7$ shows slower improvement. This behavior may result from excessively large $\epsilon$ values, which can incorrectly classify some actions as behavior-unsupported, or excessively small $\epsilon$ values, which may fail to account for certain OOD actions. In conclusion, BSPO is robust to $\epsilon_\beta$ within a reasonable range, but extreme values should be avoided.

W1: A significant concern is that the experiments are conducted solely on synthetic data. This limitation raises questions about the applicability and effectiveness of the proposed method in real-world scenarios.

Reply to W1: Thank you for your constructive feedback on our paper. We sincerely appreciate the opportunity to refine our work based on your suggestions. Studying reward over-optimization requires continuous evaluation during the training and effective comparison with the proxy reward. However, human evaluation is not only expensive but also unable to provide timely scalar feedback. As a result, to the best of our knowledge, nearly all prior works [1,2,3] rely on synthetic settings.

The differences between synthetic settings and real-world scenarios are an important concern. In response, we have conducted additional experiments in non-synthetic environments. Below, we provide a summary of our implementation and results:

Implementation Details: Since it is challenging to assign a scalar score to a response by human evaluation, an effective approach uses comparing the generated responses from different checkpoints on a evaluation dataset. Specifically, we train a 7B reward model (based on Alpaca-7B) on a preference dataset (UltraFeedback), achieving a test set accuracy of 82.94%. This reward model was used to conduct BSPO and baselines (PPO, WCO) training.

To quantify evaluation results, we compared the checkpoints on the test set to calculate the win rate between checkpoints and fit an ELO score [4] as the scalar evaluation metric. The ELO update formula is as follows:

Here, $S_{AB}$ represents the win rate of Model A over Model B on the test set, $K$ is the update coefficient, and $R_A$ and $R_B$ are the ELO scores of Models A and B, respectively.

Experimental results:

Table 1. Win rate between different checkpoints:

Table 2: The ELO scores for different checkpoints:

The curves of proxy reward and elo score in training see link: Non-Synthetic Setting

Our experiments reveal interesting insights: We observed that while the proxy reward exhibits consistently increases across the three approaches, their performance under human evaluation reveals differences:

• Naïve PPO shows performance improvement at 79 steps but suffers a severe decline by 158 steps, highlighting the issue of reward over-optimization.
• WCO maintains stable human evaluation performance across both steps, indicating its effectiveness in mitigating over-optimization. However, it does not identify the optimal solution as effectively as BSPO.
• BSPO demonstrates continuous improvement in both proxy reward and human evaluation, indicating its capability to address over-optimization while identifying optimal policies.

We hope these experimental results address your concerns and strengthen our paper's contributions to the field of RLHF.

References:

[1] Gao et al. Scaling Laws for Reward Model Overoptimization

[2] Coste et al. Reward Model Ensembles Help Mitigate Overoptimization

[3] Eisenstein et al. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

[4] Askell et al. A General Language Assistant as a Laboratory for Alignment

We sincerely appreciate your acknowledgment and are deeply encouraged by your decision to raise our rating. It is our honor to address your concerns, which have been helpful to our work and will be integral to the improvements in our final version.