Generative Reward Models

Mixing RLHF and RLAIF gets GenRM both their advantages, but also inherits both their disadvantages, and the paper lacks a discussion of that. And the idea doesn't seem very novel.

Can the reviewer provide some examples of the mentioned disadvantages? In our experiments the proposed approach achieves comparable accuracy on in-distribution data and meaningfully better generalization on OOD data as compared to both fully RLAIF approaches and RLHF reward models. Moreover, we have added additional experiments on ArenaHard and MATH500 (consult the new Figure 5 and Section 5.5 for full results) showing that the improved accuracy translates to better performance when using LLaMa 3 Instruct 8B model (which has already been heavily RLHF-trained) with Best-Of-N sampling.

There have been a few concurrent works that propose similar approaches to our work (differences are more extensively discussed in the Related Works section), but as far as we are aware specifically training RLAIF approaches to align with human (RLHF) data is novel.

The foundation model in the experiment only uses Llama3.1 8B, which is not quite demonstrating the generality and robustness of the method. Experimenting on a more variety (e.g., mistral series) and size (e.g., 13B, 70B) of models would make this paper more solid.

We evaluate 8 approaches across two large preference datasets. The. LLaMa 3.1. 8B model is considered frontier for its parameter class and we have no reason to believe these results are not transferable to other models or larger parameter classes. Additionally, the requested experiments would require nearly 20,000 GPU-hours on H100 cards.

Why does this method perform better on out-of-distribution (OOD) data? It's not very intuitive, not very easy to understand, and it would be better if there was more discussion and analysis of it.

We believe our proposed approach generalizes better than explicit reward modelling (Bradley-Terry and PairRM baselines) because it leverages the underlying LLM’s generative capabilities rather than the discriminative nature of the standard reward models as they are trained on orders of magnitude less data. The question of why reasoning approaches generalize better or are more robust, as well as the dynamics of that process are quite deep and we cannot answer them within the confines of this paper.

The evaluation of RM is inadequate, lacking experiments that assess agreement with human preferences. Additionally, only one out-of-distribution dataset, RewardBench, is utilized for evaluation.

In our experiments we utilize the UltraFeedback and UltraInteract preference datasets, which are currently standard for preference modelling in the literature. Moreover, existing human preference datasets, such as TLDR and the Anthropic Helpful and Harmless are more easily solved by existing models and are significantly easier than UF and UI, which still remain a challenge for smaller models and even advanced non-specialized models.

We would also like to point out that RewardBench has become the standard benchmark for reward modelling with over 100 frontier reward models evaluated on the official leaderboard and covers multiple datasets with different use cases - general chat/instruction-following, reasoning and safety.

The qualitative results presented in Figure 3 are unconvincing. Both STaR-DPO and LLM-as-a-judge use the same system prompt, as noted in Appendix A.1, which states that evaluations should consider factors such as helpfulness, relevance, accuracy, depth, creativity, and detail. However, your explanation for why STaR-DPO performs better than LLM-as-a-judge suggests that you prioritize instruction following over depth, helpfulness, and detail. While STaR-DPO includes an additional animal species, which is more detailed and helpful, it does not strictly follow the instructions. Furthermore, the system prompt is not always appropriate, as it can sometimes be contradictory.

We should note that these models are acting as judges only and do not generate responses to the query. That is they are only evaluating the answers from the generator model. Their chains-of-thought serve to justify their rankings and are thus evaluated only for accuracy over the preference datasets.

We use standard LLM-as-a-judge prompts from MTBench that have been established in the RLAIF literature to ensure that our experimental results reflect a fair evaluation of different methods.

The paper's title is somewhat unclear, making it difficult to discern your contribution. Additionally, some model settings are not sufficiently clear, such as the differences between GenRM and the STaR-SFT or STaR-DPO models.

The contribution of the current work is to propose and evaluate algorithms to specifically train models to act as evaluators in RLHF training and align their judgements with a preference distribution. In essence we train LLMs to act as generative critics (reward models) for RLHF. STaR-SFT and STaR-DPO are two different objectives for training the Generative Reward Model (GenRM) as outlined in Section 4.

Are STaR-SFT and STaR-DPO both trained on the same base model, rather than STaR-DPO being trained on STaR-SFT? These are both trained on the same reference model. These are different objectives of training the Generative Reward Model as outlined in Section 4 with experiential details in Section 5 and 5.1.

Where does the RLHF component fit into your framework? I see that you mention "incorporates human feedback in initial stages" in Section 7, but what exactly do you define as the initial stage, and how is human feedback utilized within this stage?

The initial stage of standard RLHF pipelines is obtaining a reward model. We explicitly suggest how to use an LLM (RLAIF) and align its preferences with the human preference distribution. The LLM is then used as a judge to provide preference feedback for the policy evaluation or training.

Could you provide a detailed summary of all the model settings, including the eight models depicted in Figure 2? Specifically, what is the base model, if it has been trained, what the training objective is, and what the dataset construction process entails? Thank you.

All judge models use the same baseline LLaMa 3.1 Instruct 8B model. For the Best-Of-N experiments we use the slightly weaker LLaMa 3 Instruct 8B model, since LLaMa 3.1 seems to mostly saturate the considered benchmarks in its parameter class due to its more extensive post-training. The training approaches we evaluate are described in Section 4 (GenRM, STaR-SFT, STaR-DPO, STaR-Rationalizer). The Bradley-Terry RM (of which PairRM is a variant) and LLM-as-a-judge baselines are described in the Preliminaries section.

We thank the reviewer for the detailed comments on the clarity and presentation of our work, we have incorporated their comment into our manuscript with several typo/style fixes, new experiments and full numerical tables in the Appendix.

We propose and evaluate a large number of design choices and ablations and it would be difficult and significantly computationally expensive to also train policies for each proposed experiment. Instead we have used existing instruction-tuned and RLHF models with further Best-Of-N inference-time sampling experiments using the proposed reward models on ArenaHard and Math500. In summary, we see that the better generalization performance of GenRM-CoT translates to better downstream policy performance as well. Please consult the updated Figure 5 and Section 5.5 for full results.

We would like to thank the reviewer for the detailed feedback!

The evaluation is limited in demonstrating what the preference accuracy improvements mean for actual policy performance. While the paper shows improvements in preference modeling accuracy, this metric heavily depends on the policy sample distribution. Notably absent is an analysis of Best-of-N (BoN) performance with the proposed reward model, which would be crucial for understanding practical impact.

We have added a set of evaluations in the Appendix for Best-Of-N using different reward model approaches on MTBench, ArenaHard and Math500 showing that the generative reward model with chain-of-thought’s higher classification accuracy translates to better downstream performance as well. Please consult Section 5.5 for the full updated results.

The practical implementation raises significant concerns. The requirement for 32 majority votes and the limitation to pairwise comparison settings could make it challenging to implement in real-world scenarios, particularly in PPO. The paper doesn't adequately address how to handle these computational requirements in practice.

We describe the connection between the classical RLHF objective and preference modelling in Section 3. In particular classical PPO pipelines still optimize the probability that the generated answer out-performs some reference source distribution. Under this framework, we can directly substitute with the pairwise preference likelihood obtained from a GenRM model.

It is true that our approach requires a higher amount of inference time compute, similar to all RLAIF approaches. While this is indeed a limitation, we believe that inference-time techniques will lead to significant capabilities improvements going forward.

The core concepts of GenRM (no CoT) and CoT-RM have been previously explored in literature, such as PairRM [1] and Self-Rewarding LMs [2]. The paper would benefit from more thorough comparisons with existing methods to better establish its novel contributions.

Our key suggestion is to leverage the existing LLM capabilities to directly model preferences (rewards) in natural language. While PairRM is also a pairwise preference model it still uses classical reward parameterization and training formulations (instead of natural language).

Our approach is a form of RLAIF, which the mentioned work, Self-Rewarding LMs belongs to. However prior RLAIF approaches have leveraged the model’s already existing capabilities to act as a judge (LLM-as-a-judge baseline in our paper). Instead, we suggest explicitly training a model to act as a judge. We propose an algorithm to align the judge model with datasets of human preferences using self-generated reasoning chains and evaluate a slew of design choices empirically, which we believe is a novel contribution.

Can you provide experimental results showing the Best-of-N performance with your reward model? This would help understand how the improvements in preference modeling translate to actual policy performance.

We have updated our manuscript with examples on ArenaHard and Math 500, please consult the updated Section 5.5.

Given the computational requirements of majority voting, how would you recommend implementing this approach practically in PPO? What modifications or optimizations might be needed?

The core idea of our approach is to specifically train an evaluation model to align with human preferences and act as a judge. In this framework, the trained model is fully compatible with existing RLAIF pipelines, including approaches outlined in Section 3.

Could you elaborate on the comparisons with similar existing methods in the literature? Specifically, how does your approach differ from and improve upon previous work like PairRM [1] and Self-Rewarding LMs [2]?

Please see above, we will expand our related works section to include that discussion.