RRM: Robust Reward Model Training Mitigates Reward Hacking

We thank the reviewers for their constructive feedback and valuable insights. Below, we address each of the main concerns raised.

Re: First, the experiments in this paper only cover two possible artifacts, namely length and a deliberately designed phrase, which is not compatible with author's claim that this method should be capable of handling "all potential exploitation patterns". More patterns such as format and emojis should be investigated in the experiments.

We acknowledge that the artifacts covered in this paper are limited. To further investigate the effectiveness of our approach, we conduct a follow-up analysis as follows: With p=0.1, add bold face (wrap the whole response as bold face) to the chosen responses. After step 1, with p=0.1, further add emoji to the end of the chosen responses. Train RM and RRM and do the same analysis as Figure 5. We add a section “Additional Analysis with Mixed Artifacts” in Appendix including the above analysis.

Re: Second, the reward model is trained only using Gemma-2-9b-it. More experiments should be conducted on different model types such as llama, and larger model sizes such as 57B and 72B. It is in doubt that when the model size increases, the reward hacking problem can be naturally mitigated and the effectiveness brought by this method may diminish.

We didn’t run Llama 3 8B due to policy reasons (https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE). To verify the effectiveness of our approach on a different model size, we conduct experiments on Gemma-2-2b-it (Added to “Additional Results with Gemma-2-2b-it” in Appendix). For larger models such as 57B and 72B, we lack computation resources to conduct the experiment at that scale. Regarding the large models, the GPT-4 judge (a large sized model) is found to have length bias when rating sxs responses [1]. It indeed suggests that the bias pattern does not diminish as model size increases.

[1] Zheng, Lianmin, et al. "Judging llm-as-a-judge with mt-bench and chatbot arena." Advances in Neural Information Processing Systems 36 (2023): 46595-46623.

Re: Third, in terms of policy evaluation, this method is only verified in DPO training whereas various other methods should also be considered such as PPO, whose performance relies heavily on reward model quality. Last, the author is suggested to add more baselines in experiment, such as reward model ensemble which can also alleviate reward hacking.

Besides DPO, we also evaluate the policy of BoN. [1] shows that BoN policy performs similarly and is more KL efficient than PPO. BoN relies heavily on reward model quality and thus we believe that our evaluation on aligned policy is convincing. Regarding reward model ensemble, we argue that it is orthogonal to our work. We focus on debiasing the artifacts learned in the reward model, it is more on mitigating the bias. But the reward model ensemble is more on reducing the variance. It can not eliminate the artifact bias (such as length, emoji, format) existing in the reward model. Besides, our approach can be naturally combined with the reward model ensemble approach.

[1] Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." International Conference on Machine Learning. PMLR, 2023.

We thank the reviewers for their constructive feedback and valuable insights. Below, we address each of the main concerns raised.

Re: It seems suboptimal to me to train the reward model that is only dependent on the S, because the A part could contribute nontrivially to the reward function, in the case where both answers are equal in terms of the contextual quality, the labeler could prefer the answer which is organized as bullet points for readability.

We agree that responses with better organization, such as those formatted with bullet points, should be preferred in general. But we would like to address that conditioning on prompt, there can be richer information on determining the preference. There can be cases that the prompt specifies “please do not include bullet points in your response”. Even for factuality and safety attributes, the prompt can ask the model to create a virtual horror story which may violate the factuality/safety constraints. Regarding “in the case where both answers are equal in terms of the contextual quality, the labeler could prefer the answer which is organized as bullet points for readability.”, we have a few thoughts to share:

1. We would like to answer the question that “what the preference will be if both responses share the same artifacts” following length-controlled Alpaca [1]? Thus if we modify the one answer to remove the “bullet points”, what the preference will be? Maybe after removing the bullet points, the contextual quality can change, becaues the prompt can indicate whether bullet points are needed. If the prompt is for creative writing, removing bullet points can even improve the preference.

2. Even if there is such a pattern that should be uniformly preferred, our RRM can still learn this pattern because we only augment data, not delete any existing data. The purpose of adding augmented data is to force the model to learn better and avoid native shortcut prediction. The second point is, theoretically our proposed method can completely remove the effect of A, when we have unlimited samples. But in practice, we don't have infinite samples, and due to computation budget, we only add a small portion of fully augmented data distribution. The added augmented data helps improve the robustness.

3. We think the reviewer raises the valid point on prompt-agnostic artifacts that can uniformly contribute to the preference such as safety. To address this, in the conclusion section, we mention that “Future work will explore filtering augmented pairs and matching artifacts when constructing response pairs, further refining the training process.”. Thus, in the future, we should apply filters on the prompt and responses to guarantee that the losing responses are not too bad. But we believe that our work opens a door on debiasing RM with a data augmentation approach with a causal learning framework. Follow-up works can focus on matching the effects of chosen/rejected responses, and filtering data by prompt/responses.

[1] Dubois, Yann, et al. "Length-controlled alpacaeval: A simple way to debias automatic evaluators." arXiv preprint arXiv:2404.04475 (2024).

We thank the reviewers for their constructive feedback and valuable insights. Below, we address each of the main concerns raised.

Re: The main weakness of this paper is that the experiments are not that comprehensive. Although the results are strong, the experiments are only conducted on one dataset, with one model, and with one random seed. Therefore we cannot tell if the results are statistically significant or generalizable.

We acknowledge that the experiments are not comprehensive enough, although we have evaluated thoroughly on RewardBench, MTBench, and AlpacaEval2 with various alignment algorithms (DPO/BoN) and baselines (ODIN/RM). To further enhance the confidence of our results, we add experiments with a Gemma-2-2b-it reward model in Appendix (Additional Results with Gemma-2-2b-it) of the updated draft. To understand better on whether the artifacts can be effectively mitigated, we also add experiments with two additional artifacts: bold faces and emojis in Appendix (Additional Analysis with Mixed Artifacts). With additional experiments, it enhances the significance of our framework.

Re: The baseline is pretty weak with regards to reward bench. RLHFFlow with Llama 3 8B gets 87.1 on reward bench, while the author’s baseline gets 80.

We didn’t run Llama 3 8B due to policy reasons (https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE). We use the same dataset as those in RLHFFlow and for baseline and our approach, we use the same model (Gemma-2-9b-it) as a fair comparison. To further verify the results, we add a Gemma-2-2b-it in Appendix (Additional Results with Gemma-2-2b-it).

Re: The writing describing the methodology not that clear. I think more plainly describing the data augmentation would be better.

We thank the reviewer for the feedback. We illustrate our augmentation strategy in Figure 1 for illustration purpose. We describe our approach in plain language in line 086 as “Our pipeline is illustrated in Figure 1, where we augment the reward model training data by using responses from other examples to effectively balance the artifacts in chosen and rejected responses”. In line 224, we also mention that “In practice, we can shuffle the dataset twice to achieve $σ_1$ and $σ_2$.”. We can further improve the clarity of our approach in the camera-ready version.

Re: The data filtering step used in their method hurts their experiments interpretability, as it may be the case that the sole reason RRM gets good performance is due to the data filtering step. They should include a baseline where the original dataset is filtered, and then a RM is trained based on that.

Sorry for the confusion. In the paper we mentioned that “To reduce the augmented data size, we first conduct inference on random 50% of the augmented data using the trained RM, …”. We only filter the data on the augmented triplets and do not filter the original data. So in both baseline and our approach, we use the full copy of the original data.

Re: How did the authors tune the hyperparameters?

In line 342-344 (and footnote 14) and line 350-352, we cover how to tune the hyperparameters. We use grid search to pick the best hyperparameter based on evaluation metrics (evaluation accuracy for RM and AlpacaEval2 for RLHF).

Re: What are the “three possible prompts” described in line 226?

The three possible prompts correspond to $(x^{(i)}, x^{(\sigma_1(i))}, x^{(\sigma_2(i))})$, which corresponds to the original prompt, a prompt from shuffled dataset, and a prompt from a twice-shuffled dataset.

We thank the reviewers for their constructive feedback and valuable insights. Below, we address each of the main concerns raised.

Re: While the problem studied in this paper is significant, the proposed solution is relatively straightforward, focusing only on expanding the dataset. To remove prompt-independent artifacts, the dataset size needs to be several times larger, increasing the training cost. Additionally, as shown in Figure 5, the effect of this costly augmentation is not very significant—after inserting 30% artifacts, the RRM only decreases from the original RM’s 25% to 20%.

Re: While the problem studied in this paper is significant, the proposed solution is relatively straightforward, focusing only on expanding the dataset. To remove prompt-independent artifacts, the dataset size needs to be several times larger, increasing the training cost.

We acknowledge that our proposed solution may seem straightforward, as it involves dataset expansion. However, this simplicity belies the sophistication of the underlying causal learning framework, a novel theoretical approach not previously applied to this challenge. By disentangling prompt-independent artifacts from input-driven preferences, our method achieves both simplicity and effectiveness. This enables a broader range of applications across diverse use cases. Regarding training cost, we recognize the concern but argue that the cost increase remains manageable within the broader context of reinforcement learning with human feedback (RLHF). The reward model (RM), while crucial to alignment, incurs comparatively less computational burden than supervised fine-tuning or policy optimization. Thus, improving the RM, the key component for model alignment, justifies the additional investment in training. In our study, we trained both the RM and the robust RM (RRM) to convergence, fully utilizing the value and information embedded in our expanded dataset to ensure optimal performance.

Re: Additionally, as shown in Figure 5, the effect of this costly augmentation is not very significant—after inserting 30% artifacts, the RRM only decreases from the original RM’s 25% to 20%.

We understand that the observed improvement may initially appear modest. However, it’s essential to consider the context of the evaluation. With 30% artifacts introduced into responses, a random baseline would yield around 30% artifact presence in optimal responses. Thus, achieving 25% with RM and 20% with RRM represents a meaningful improvement compared to the baseline. The reduction of 5 percentage points below this baseline signifies effective artifact mitigation, illustrating the method’s impact in refining alignment robustness.