GRAM: A Generative Foundation Reward Model for Reward Generalization

Dear Reviewer Jdyw,

We appreciate that you find "the superior generalization of our GRAM compared to both discriminative and prior generative reward models" and our pre-training approach is "make sense".

We will provide explanations for the main points that you are concerned about.

---

W1: The results in Figure 2 and Table 1, where GRAM underperforms compared to discriminative models on ID data, raise the question of whether this reflects overfitting by discriminative models rather than better generalization.

We would like to clarify this question in two aspects.

• The ID test set evaluates the ability of reward modeling to learn human preferences from labeled data. Our goal is to excel in reward modeling and generalization, i.e., obtaining strong performance on ID and OOD tasks simultaneously. Thus, the LLaMA-3.1-8B-Instruct results in Table 1 show our method's effectiveness, achieving the best OOD results and second-best ID results.
• Although GRAM underperforms compared to the Discriminative RM+Regularization when using LLaMA-3.1-8B-Instruct, it significantly outperforms both the Discriminative RM (Baseline) and Discriminative RM+Freeze, supporting the validity of our method. We also see that Regularization may not be a universally effective method across all models, e.g., on the LLaMA-3.2-3B-Instruct model, Regularization performs notably worse than GRAM, suggesting that its effectiveness could be model-dependent.

The partial results of Table 1 are shown below:

|Method|UniFeed(ID)|RewardBench(OOD)|   |:-|:-:|:-:| |LLaMA-3.1-Instruct| |Discriminative RM (Baseline)|69.3|74.1| |Discriminative RM+Freeze|66.6|74.9| |Discriminative RM+Regularization|72.7|77.4| |GRAM (Ours)|70.4|85.1| |LLaMA-3.2-Instruct| |Discriminative RM (Baseline)|68.3|72.8| |Discriminative RM+Freeze|63.0|70.5| |Discriminative RM+Regularization|65.6|71.2| |GRAM (Ours)|70.6|83.6|

W2: The paper lacks reinforcement learning experiments where an LLM optimizes the reward model using RL.

We apologize for any misunderstanding regarding the RL experiments. In fact, we have already included reinforcement learning experiments in Appendix C.1. Specifically, we comprehensively compare PPO performance with different reward models, as shown in Figure 8 and Table 2. The results align with the response ranking, showing that GRAM exhibits strong generalization and provides more accurate rewards. To respond to your concern, and in the revised version, we promise to include partial results in the body rather than only in the appendix.

Q1: The paper does not detail how to prepare the two responses, particularly how to prompt an LLM for multiple responses (e.g., simply different responses or responses of varying quality).

Thanks for this insightful suggestion! We have explored this issue with two GRAM variants.

• GRAM-Sim-Diff: We explore the impact of response diversity on pre-training performance. Specifically, we select the 200k response pairs with the highest semantic differences from a dataset of 600k responses and compare them to 200k randomly selected pairs.
• GRAM-Qua-Diff: We test quality differences by scoring responses with GPT-4o on a 0-5 scale and compare the 200k pairs with the largest quality differences to 200k random pairs.

The results are as follows: |Method|RewardBench|   |:-|:-:| |GRAM-Sim-Diff|74.8| |GRAM-Qua-Diff|75.4| |GRAM|76.2|

We find that randomly selected pairs outperformed the carefully selected ones, possibly due to increased diversity from randomness. We also see that larger quality differences have little impact on reward model training, with random selection performing better. This is also shown in previous work (Filtered Direct Preference Optimization), where larger quality differences benefited DPO training but not reward model training. We promise to add more experiments and analysis in the revised version.

Q2: A discussion on extending the training scheme to multi-objective settings, where responses are evaluated by multiple criteria, would be insightful. Have you considered using reasoning models for generative reward models?

Thanks for your valuable suggestion! GRAM can easily be extended to multi-objective or reasoning settings, and this extension can be implemented in stage two. Specifically, we would describe the objectives, e.g., fluency and accuracy, in the prompt $c$. When generating preferences, we can use the format #Fluency Preferred: [labeled token] #Accuracy Preferred: [labeled token] instead of #Preferred: [labeled token]. Similarly, we can also add the CoT in front of the preferences. We can also use a reasoning model to develop GRAM further. These improvements don’t require changes to our pre-training, highlighting its robustness and scalability for reward modeling. We promise to include more analysis in the revised version.

---

We truly appreciate your positive feedback on our paper!

Best,

Authors

Dear Reviewer pUcN,

We sincerely thank the reviewer for your positive and insightful feedback.   We greatly appreciate your recognition of our paper's proposed training paradigm for the reward model as "interesting and sound", and are pleased that the method is considered to be "extensively evaluated across different tasks".

We will provide explanations for the main points that you are concerned about.

---

W1: I am wondering whether using such enhanced RM could improve the reasoning ability of existing LLMs.

Thanks for your insightful suggestions! Indeed, using an enhanced RM can improve the reasoning ability of existing LLMs. We have already conducted an experiment to validate this with a math comparison pair dataset (huggingface tag: reciprocate/math_dpo_pairs). Specifically, we used this dataset during the second stage of GRAM training, with all baseline RMs also trained on the same dataset. We performed reward accuracy experiments on the corresponding test set and with best-of-n sampling on GSM8K, respectively. For the best-of-n sampling, we sample 16 outputs on LLaMA-3.1-8B-Instruct with 8-shot for each input.

|Method|RM Accuracy|GSM8K (Best-of-n Sampling)|   |:-|:-:|:-:| |LLaMA-3.1-8B-Instruct|-|83.9| |Discriminative RM|63.2|84.6| |Generative RM|61.3|85.5| |GRAM|66.8|87.2|

The experimental results show that our method effectively improves RM accuracy on reasoning-related downstream tasks. Also, it further enhances the reasoning ability of LLMs with best-of-n sampling. We promise to include more experiments, including additional baselines and RL experiments, in the revised version to further verify the effectiveness of GRAM in improving reasoning abilities.

W2: The influence of pre-training data on target domain should also be discussed.

In Figure 6 of the paper, we already show the impact of pre-training data on general domain adaptation. In response to your concern, we further conduct experiments in specific domains using LLaMA-3.2-3B-Instruct, along with 5k labeled summarization and harmlessness data.

The results from these experiments align with those in Figure 6. We see that as the amount of unlabeled data increases, the accuracy of GRAM generally improves for both models. This also highlights the crucial role of unlabeled data and the scaling effect on performance, suggesting that using larger unlabeled datasets can lead to better reward models. We promise to include these experiments in the revised version.

Q1: Would the domain difference between pre-training and fine-tuning has a big impact on RM performance?

To respond to your concern, we conduct the following experiments on GRAM:

• GRAM-v1: Pre-training on 100k unlabeled summarization response pairs (derived from TL;DR comparison data;huggingface tag: openai/summarize_from_feedback), followed by fine-tuning on 5k labeled summarization data.
• GRAM-v2: Pre-training on 100k general unlabeled data (including summarization responses), followed by fine-tuning on 5k labeled summarization data.
• GRAM-v3: Pre-training on 100k general unlabeled data (without summarization-related data; specifically, we are using ChatGPT to filter out preference data related to summarization), followed by fine-tuning on 5k labelled summarization data.

In these experiments, summarization is the downstream domain. GRAM-v1 uses pre-training data closest to summarization, fully utilizing summarization responses. GRAM-v2 uses general pre-training data, which might include some samples aligned with the downstream summarization task. GRAM-v3 uses pre-training data that is the most different from the summarization domain. Specifically, we use ChatGPT to exclude all samples related to summarization tasks. The results are as follows:

The experimental results demonstrate that pre-training data more closely aligned with the target domain results in better performance in that domain. In fact, this experimental observation is consistent with the common practice in LLMs, where incorporating as much domain-specific data as possible during pre-training typically leads to better performance on downstream tasks. Additionally, our results show that the pre-training approach exhibits strong robustness—despite significant domain differences, it still yields positive contributions to performance.

We appreciate your valuable suggestion and promise to include more discussion and experiments in the revised version.

---

We sincerely thank you for your positive feedback on our paper!   Thank you once again for your time.

Best,

Authors

Dear Reviewer WL65,

We appreciate the reviewer’s constructive and thoughtful feedback.   We appreciate your recognition that the main claim of our paper is "empirically well supported", and that our experiments are "comprehensive, meticulously studying each of the design choices".

We provide explanations for the main points that you are concerned about.

---

W1: The technical contribution seems simple—pre-training and label smoothing—both well-established in ML. The contribution is empirical and may be more suited for NLP conferences than ML.

Thanks for your insightful comment! As you said "pre-training and label smoothing are well-established concepts in ML", we would like to clarify that our contribution is not about innovating them, but rather introducing these concepts to achieve superior reward modeling. The rationale behind this is as follows:

• While applying reward models to align LLMs is a compelling direction, training these models still heavily relies on labeled data. We expect to enhance reward modeling by pre-training on unlabeled data to reduce dependence on labeled data. However, this pre-training approach has never been explored and presents significant challenges. The difficulty arises from that, unlike self-supervised approaches, systems cannot directly generate their own supervision signals from text for training reward models. In this work, we thoroughly explore the use of unlabeled data in reward modeling and propose a specific pre-training reward model procedure. We also discuss its effectiveness from both theoretical and practical perspectives.

• Although label smoothing has been shown to be effective in many tasks, it has not been well-established in reward modeling. In this work, we theoretically demonstrate that the training objective of generative reward models can be reformulated into a more elegant form: we are essentially optimizing the Bradley-Terry model with modified label smoothing. This result is significant as it establishes a connection between discriminative and generative reward modeling methods—both of which fundamentally train LLMs to perform pairwise ranking, thereby pushing the boundaries of reward modeling and directly contributing to improved generalization.

Q1: On which dataset do authors run their stage 1 (task 1 pre-training)? The similar idea of pre-training reward models is mentioned in Bai et al. (2022).

The pre-training responses also come from Unified Feedback. Please note that while this data includes preference labels, we do not use these labels in our pre-training process. Instead, we only use the response pairs to simulate unlabeled data and validate the effectiveness of our method.

Since our pre-training approach only uses responses for unsupervised training (which can be directly sampled from LLMs), it allows for easier scalability. We also present two key insights from our experiments for pre-training. First, as shown in Figure 5, during downstream adaptation, our pre-training effectively achieves generalization and reduces the need for specific labeled data. For example, in the summarization task, we achieve performance comparable to training from scratch with approximately 100k labeled data using only 5k data points. Second, as demonstrated in Section 5.1, we find that the more unlabeled data used in training, the greater the benefit to downstream performance, regardless of whether the downstream labeled data is small or large. Building on these insights, in real-world reward model training, we could first collect many responses using cost-effective methods, such as sampling from LLMs, and then fine-tune with a small amount of labeled preference data.

In contrast, Bai et al. (2022) propose a pre-training approach that uses large, labeled preference data sets for supervised training, followed by domain-specific fine-tuning. This method is difficult to scale as it still relies heavily on labeled preference data, which is often scarce in real-world scenarios.

Thank you for your helpful suggestion! In the revised version, we promise to provide a more detailed description and experiment-based analysis for your concern.

Q2: In Figure 5, do RM fine-tuned with G/D-baselines mean RMs trained on UnifiedFeedback?

Yes, the data used to fine-tune the RMs for forming the G/D-baselines comes from UnifiedFeedback. Note that this data is the same as the one used by GRAM in the fine-tuning stage, which includes preference annotations.

S1: Insufficient description of related work and errors in reference presentation.

Thank you for your helpful feedback! We promise to correct the description and presentation of the related work in the revised version.

S2: Essential references are not discussed.

Thanks for your valuable suggestion! We promise to include the discussion of the essential references in the revised version.

---

We sincerely appreciate your positive feedback and thank you again for your time.

Best,

Authors

Dear Reviewer 3tRc,

We would like to thank the reviewer for the positive feedback regarding the novelty of pre-training for reward models and the strong results.   Below, we explain the main points you are concerned about.

---

One component that does not make sense for this problem is the pre-training stage.

We would like to clarify the motivation behind our pre-training design. As discussed in Section Appendix A, we can understand our method from a feature learning perspective. In generative reward modeling, the loss function can be given by

$$L\_{\mathrm{g}}(\theta) = L\_{\mathrm{gp}}(\theta\_{\mathrm{gp}})+L\_{\mathrm{gf}}(\theta)$$

The feature optimization term $L\_{\mathrm{gf}}(\theta)$ is implicitly defined and optimized by adjusting preference generation. Traditional generative reward modeling optimizes these objectives using costly preference labels. This work explores a pre-training approach to optimize $L\_{\mathrm{gf}}(\theta)$ first.

To this end, we propose using an auxiliary task to optimize these features without relying on labeled preferences. Specifically, we utilize conditional probabilities to characterize interrelationships, i.e., $-\log \pi\_{\theta}(y\_{b}|x,y_a)$. Besides, considering a distributional error arising from both $\pi\_{\theta}$ and the $y_a$ or $y_b$ to a given input $x$, we introduce an SFT loss as a regularization term, i.e., $-\log \pi\_{\theta}(y|x)$. After careful derivation, we obtain

$$L\_{\mathrm{gf}}(\theta) = -\mathbb{E}\_{(x, y_a, y_b) \sim D_u} [ \log \pi_\theta([y_a, y_b] | x) ]$$

This predicts two responses. The model can better generalize and improve its reward capability by focusing on learning meaningful features rather than surface-level ones (e.g., length). Also, this pre-training strategy has been highlighted to make sense by Reviewer Jdyw. Thanks for the feedback!

What is the purpose of comparing with a proxy model trained on less data? Which model is used as the policy for generating ranked responses?

Training a proxy model tests the method's modeling capability. A good proxy score indicates strong performance on ID, reflecting good modeling. Our goal is to design a method with both strong modeling and generalization. Thus, testing with the proxy score is crucial. This setup is also supported by Yang et al. (2024).

As for the sampling policy, we use the LLaMA-3.1-8B model, as described in Section 4.4 (line 319).

W1: In Table 1, what are the methods denoted (baseline)?

Our main baselines include several methods (such as Freeze and Regularization) aimed at enhancing generalization and discriminative models to demonstrate the promising generalization of our GRAM.

W2&Q4: Lack of detailed description on the baseline version of label smoothing (LS).

The LS we use differs from the standard LS. Our method applies LS to all candidate label tokens, as described in Section 3.3, whereas the traditional method applies it to all tokens in the vocabulary. We theoretically demonstrate that our method is more effective, potentially optimizing with a constrained Bradley-Terry model. Apart from the explanation in Appendix D.2, we have conducted further experiments on the LLaMA-3.2-3B-Instruct to explore this in more detail below.

As the experimental results show, our LS leads to better reward modeling compared to the standard LS. These results also show the role of LS in training GRAM. We promise to add more discussions on this in the revised version.

Q1&Q2: Are there two 400,000-example splits, one for pre-training and one for fine-tuning? Can we collect pre-training data by sampling from LLaMA models?

Yes, both splits of 400,000 examples are randomly selected from the Unified Feedback dataset. The pre-training responses also come from Unified Feedback, but we only use the responses without their preference labels to simulate unlabeled responses. Sampling responses from LLaMA to obtain unlabeled data is a good idea, and we commit to exploring this method further. This work focused on the time-consuming nature of sampling large-scale labeled response pairs. Also, we expect such available data to exist for direct training in real-world scenarios. Thus, we randomly select response pairs from Unified Feedback to simulate available labeled response pairs.

Q3: This scoring by Eq. 3 may yield vastly different scores depending on the reference response.

This setup is reasonable. In BoN sampling, we keep the same reference for all candidate outputs, ensuring the scores are comparable. In RL, the recent work Remax uses the difference from a reference as the reward and interprets its reasonableness from a reward baseline perspective.

---

We sincerely thank you for your positive feedback on our paper!

Best,

Authors