Critique-out-Loud Reward Models

We would like to thank reviewer pirR for their time spent reviewing our paper and for their thoughtful questions. We appreciate that you find the technique “neat and novel”. Our responses to the revier's questions are listed below.

1. "The authors use Llama 405B to generate oracle critique data for SFT, and replace the oracle data with fine-tuned model's self-genereated critique. Usually when you distill from larger models, there will be direct gain in model performance. How do we know that if the performance gain is mainly from distilling responses from large models, or from the methodology itself?"

   In both the CLoud and classic examples we are distilling the larger model preferences into the reward models as the preference labels are generated by the 405B model. While there is more information distilled to the CLoud models as they additionally see critiques, we view this as a strength of the method as, while the same information from the large model is available to both the reward models, only the CLoud model can leverage it. In this work we only use LLMs to generate oracle critiques as a proxy for human labelers. When human labelers are available we would argue methods such seek to distill as much as possible.

   To investigate what degree of performance gains come from distilling the larger model, we test CLoud models where we output a reward without generating a critique. We find that while the CLoud trained model that assigns a reward without generating a critique still outperforms the classic model, it underperforms the standard CLoud model suggesting that critique generation is an important component.

   Model	Chat	Chat Hard	Safety	Reasoning	Average
   CLoud Critique 70B	97.21	58.22	83.99	93.50	83.23
   CLoud No Critique 70B	94.97	56.80	79.46	92.94	81.04
   Classic 70B	95.25	50.99	73.99	89.34	77.39

   We would further like to emphasize that training on critiques, even if not used at inference, is a novel aspect of our CLoud method. As such, the total advantage of our method is both the training and inference process which together represent a significant lift in reward modeling performance.

2. "If the critique SFT data is mostly self-generated (not on the fine-tuned one), would we still expect such high gain in downstream performance? This will be helpful for the case where we want to improve the largest model available with only binary human feedback."

   Investigating self-generated critiques is a very interesting avenue for future work. While we don’t want to speculate too much, our initial guess is that training the model to produce critiques through methods such as self-taught evaluators [1] would also improve reward modeling performance without requiring explicit critiques to train on.

3. "The generation of SFT data is using llama 3.1 405B but the training model is llama 3 instead of llama 3.1, what's the reason behind this? Would the gain of the same methodology appear weaker when it comes to stronger base RM?"

   We began the research before the llama 3.1 series of models was announced and had built the training stack around llama 3 models. As such, we only used the 3.1 series of models for data labeling but kept the same training stack that was optimized for llama 3 models. Using llama 3 models for training and 3.1 for labeling was simply an artifact of the project timeline and we believe that while the exact performance numbers might differ, the relative comparison would be the same had we used 3.1 for training.

4. "Does the gain mainly come from the diverse critiques generated, or from more clear instructions of what the RM should look like? For example, if we provide a very comprehensive fixed system prompt asking the response to focus on safety, helpfulness, comprehensiveness, (e.g. by modifying the prompt in Figure 9) and provide a combination of system prompt + user prompt + response to RM, would it achieve similar performance as user prompt + response + critique?"

   We agree that understanding what aspects of the critiques lead to improved reward modeling is an interesting research question that we hope to investigate in further work. We implement the proposed experiment of using a system prompt similar to that in Figure 9 on RewardBench for the 8B classic reward model. We find that focusing the reward model in such a manner does lead to a slight improvement in reward modeling performance. However, this method still significantly underperforms the CLoud model suggesting more focused instructions do not account for the total gain from CLoud.

   Model	Chat	Chat Hard	Safety	Reasoning	Average
   CLoud 8B	96.51	45.39	75.81	87.04	76.19
   Classic 8B	93.29	43.75	66.82	82.29	71.54
   Classic + Sys. Prompt 8B	94.41	46.71	67.84	81.20	72.54

References

[1] Wang, Tianlu, et al. "Self-taught evaluators." 2024

We would like to thank reviewer sPGi for taking the time to review our paper and for providing detailed feedback. We would also like to thank reviewer sPGi for valuing the motivation behind our work. Our responses to the reviewer’s feedback are listed below.

1. "The authors constructed their dataset based on UltraFeedback and UltraInstruct, requiring Llama-3.1-405B-Instruct to generate critiques for chosen and non-chosen responses. This seems to introduce additional annotation costs and complexities, which is a drawback of the method. How effective would this pipeline be in a pure self-critique setting?"

   While building a dataset of critiques does include additional overhead, we would argue that this overhead is not necessarily a detractor as it enables performance not achievable via classic reward models. Additionally, the increase in data overhead enables a reduction of other overheads such as model size, i.e., the 8B CLoud model closes most of the gap to the 70B classic model. While out of the scope of this work, we agree that investigating the self-critique settings is an interesting direction for future work, and other works such as “Self-taught evaluators” [1] suggest it is a promising direction.

2. "The authors test the method's effectiveness in the safety domain, where critique is currently widely used in LLM safety alignment scenarios. This is intriguing. However, the authors' safety evaluation of the method on RewardBench raises concerns, as my understanding is that RewardBench aggregates existing datasets with inconsistent safety classifications across different safety datasets. Consequently, the detailed assessments in safety and reasoning may lack credibility. I recommend that the authors evaluate the method on specific datasets for different categories, such as reasoning on GSM8K [5] and safety on BeaverTails [6], which are designed for specific LLM categories."

   The safety section of RewardBench consists of custom human-curated versions of XSTest ([2]) and Do-Not-Answer ([3]) and additional questions uniquely developed by the authors of RewardBench. These do not have overlapping prompts and the authors note the methods they have used to remove any remaining inconsistent classifications. Given these annotation pipelines and the fact that BeaverTails reports an average annotator agreement of only around 60% for their helpful/harmless splits, we expect the safety split of RewardBench to be a higher-quality signal than BeaverTails. As for reasoning, the reasoning split of RewardBench contains the Math dataset [4] which is a more difficult mathematics dataset than GSM8K and makes for a better evaluation as it is less saturated. The ArenaHard benchmark, which we evaluate the BoN policy on, is also heavily reasoning dominated with problems of difficulty greater than those of GSM8K.

3. "In validating the effectiveness of CLoud RM, apart from testing on RewardBench, the authors conducted only BoN experiments. What is the impact of Cloud RM on RLHF? This is a critical aspect that I would like to address, especially since Cloud RM adds the additional annotation cost of critiques. The authors claim that the scalar reward is more accurate but have not conducted any RLHF experiments [3]."

   Learning a policy (i.e., PPO) is not required for RLHF. RHLF is the process of defining a policy via a reward signal derived from human feedback. Best-of-N is one such method for defining a policy from a reward and it is actually the KL optimal policy given a reward function [5][6]. As we demonstrate that CLoud defines a strictly better Best-of-N policy on the ArenaHard chat benchmark as compared to a classic reward model (Figure 4), our work explicitly demonstrates that CLoud is better for RLHF and thus is useful to practitioners.

4. "This method inevitably increases the complexity and cost of the pipeline."

   CLoud does introduce overhead at inference and data collection as compared to classic reward models, however, the improvement in reward modeling performance justifies this overhead. Namely, suppose one was trying to maximize the performance of their reward model and they were already using the largest base model available. To improve reward modeling performance one would need to scale some part of the system, and as model size is no longer a valid lever, one would need to scale inference compute via critiques as done in CLoud. It’s more productive to think of the overhead of CLoud as a tradeoff in overheads. Similar to the case where you are using the largest existing base model, suppose you are using the largest reward model that fits within your machine’s memory constraints. If you want to produce a better reward model subject to such constraints one way is by adding critiques.

We would like to thank reviewer bVCe for taking the time to review our paper and for finding our approach “innovative” and agreeing that the method leads to “significant improvements”. Our responses to the reviewer's feedback are listed below.

1. "Criticism generation methods, while innovative, may introduce bias if the criticisms reflect some fixed bias of LLMs rather than true preferences"

   While we agree that it is an interesting direction for future work to investigate the biases in critiques, we don’t believe it is fair to say that critiques will a priori increase the bias of reward models. Namely, classic reward models must infer the implicit reasons behind a user's preferences while CLoud reward models are exposed to the user’s explicit preferences in language. As such one may expect CLoud models to be less biased as they are modeling the user’s explicit preferences. We would also like to emphasize that our method does not require using an LLM to generate the training critiques, and in fact we only use LLM training critiques as a proxy for human critiques.

2. "The process of generating criticisms can be viewed as a sort of distillation from the larger motdel, and the comparisons provided in the paper are slightly unfair compared to scoring directly using the reward model."

   We believe that the evaluation is a fair comparison as both the CLoud and classic models have access to the same preference data generated by the same labeler, the classic model is just incapable of leveraging such data. As we are introducing a new capability for reward models (generating critiques), our method fundamentally requires leveraging data that couldn’t be leveraged with previous models. The point of our method is that there is useful information to be learned from a labeler that can improve modeling performance and that our method is the first method capable of doing so.

3. "More ablation should be provided."

   We would be happy to provide further experiments for the camera ready if you could clarify exactly what ablations you would be interested in.

4. "Critic generation and inference practices lack rational supervision."

   We assume that the reviewer meant to ask about “rationale supervision”. If our assumption is wrong please let us know. We agree that adding additional methods to supervise the critique quality during inference is an exciting direction for future work. However, actually measuring and varying the quality of critiques along some axis has the scope of an entirely independent research project [1].

   As stated in the paper, we only use an LLM to generate oracle critiques to approximate human critiques. As such, our work was not concerned with supervising the quality of critiques to train on as our setup was meant to approximate the setting in which a human provides the critiques.

5. "For GenRM, further discussion should be proposed on the construction of the critique process."

   We would be happy to provide any requested details in the final version of the paper. Are there any specific questions that you have?

6. "The approach in the paper can be viewed as incremental compared to a general reward model, so performance increases do not fully substantiate its value"

   CLoud represents a substantial improvement over classic reward models both due to the significant performance increase and the new reward modeling methods enabled by generating critiques, each of which alone would justify CLoud as a substantial improvement. Namely, on RewardBench 8B CLoud almost matches 70B classic, and 70B CLoud is a 5.84 pp improvement over 70B classic. Furthermore, by introducing language generation to reward modeling CLoud opens up a new inference-compute scaling axis for improving reward model performance (Section 3.3) that we hope encourages future work.

7. "How does the performance of this type of operation compare to simply llm-judge to produce preference labeling?"

   To baseline how llm-as-a-judge performs, we can compare to the official RewardBench performance for llm-as-a-judge using Llama3 8B Instruct. It is important to state that while Llama3 8B Instruct has gone through significantly more general post-training than any of our models, it was not trained on the exact data that we trained our reward models on. As such it is not a completely controlled comparison. That being said, this is how llm-as-a-judge performed:

   Model	Chat	Chat Hard	Safety	Reasoning	Average
   CLoud 8B	96.51	45.39	75.81	87.04	76.19
   Classic 8B	93.29	43.75	66.82	82.29	71.54
   LLM-Judge 8B	85.50	41.60	68.0	64.8	65.0

   As can be seen, LLM-as-a-Judge with this setup significantly underperforms both the classic and CLoud models.

We would like to thank reviewer T6ho for taking the time to engage with and review our paper, and we appreciate that the reviewer finds the paper “well-written” and containing “comprehensive ablations”. Please see below for our responses to the reviewer’s feedback.

1. "One of the most critical applications of the Reward Model is RLHF, yet the authors have not verified the improvement of CLoud RM over traditional RMs in RLHF. I believe this would greatly limit the widespread application of this method."

   Learning a policy (i.e., PPO) is not required for RLHF. RLHF as a framework is the process of defining a policy via a reward signal derived from human feedback. Best-of-N is one such method for defining a policy from a reward and it is actually the KL optimal policy given a reward signal [1][2]. As we demonstrate that CLoud defines a strictly better Best-of-N policy on the ArenaHard chat benchmark as compared to a classic reward model (Figure 4), our work explicitly demonstrates that CLoud is better for RLHF and thus is useful to practitioners.

2. "It is not very clear what significant issues in Reward Modeling CLoud has addressed, such as reward overoptimization. From the experimental results, it appears to only enhance the reward model's ability to distinguish between binary responses"

   Improving the ability of reward models to assign preference scores to responses is the most fundamental improvement one can make to a reward model. Reward models are explicitly trained to model human preferences so surely improving the reward model’s capability to distinguish preferred responses is solving a significant issue. CLoud also solves the problem of reward models not being able to leverage additional inference compute, which we demonstrate improves reasoning performance in Figure 7. This provides an entirely new axis of scaling for improving reward quality. Additionally, we do not only look at reward model accuracies, but we also show that CLoud leads to improved downstream policies optimized using this reward (see first response and Figure 4).

3. "Based on my understanding from page 5, the authors' preference dataset and BoN dataset are derived from sampling Llama3-8B-Instruct. However, is it appropriate to use data generated by an 8B model for experiments with a 70B model? … The above point leads to my skepticism about the author's claim that "8B CLoud reward model even outperforms the 70B classic reward model."

   We believe this is an appropriate setup as the training setup is identical between the CLoud and classic model, so the tests still isolate for preference modeling capability. Additionally, common RM training benchmarks such as the original UltraFeedback dataset [3] contain responses from smaller 8B models.

   To demonstrate that our results are not just an artifact of using 8B responses, we have trained both CLoud 8B and 70B and classic 8B and 70B on the original responses in the Ultra datasets. For this experiment, we used critiques generated by GPT-4o, as we had created this dataset before the rebuttal and would not have time to generate critiques with Llama 405B. We find that CLoud models still significantly outperform classic models and that CLoud 8B recovers a large amount of the gap to the classic 70B model.

   Model	Chat	Chat Hard	Safety	Reasoning	Average
   Classic 8B	95.25	54.75	71.76	83.83	76.40
   CLoud 8B	95.44	58.26	77.52	87.63	79.71
   Classic 70B	95.81	69.30	82.70	88.47	84.07
   CLoud 70B	96.09	73.03	88.11	94.55	87.94

4. "The experiment on self-consistency is interesting, but the author's research is not thorough enough. If they could investigate how the quality of critique generation affects scoring accuracy, and combine this with methods like COT, TOT, etc., for further exploration, it would be more impressive."

   While we do already investigate when self-consistency is useful by looking at its relationship with the reasoning horizon of the problem (Figure 8), we agree that investigating the impact that critique quality has on self-consistency would be interesting. However, we believe it’s out of scope given the large ground that our paper already covers. While all experiments are already conducted in the setting of CoT critiques, we agree that methods such as ToT would be interesting to investigate as exciting future directions.

5. "I'm not particularly fond of the title "Results" for Section 3; terms like "Experiment" might be more appropriate."

   We will update the section title to be “Experiments” in the final version.

6. "RQ2 in Section 3 seems a bit odd because there hasn't been any prior mention of on-policy/off-policy concepts, which makes it difficult to understand."

   Thank you for pointing this out, we will explicitly state the distinction between on and off-policy in our methods section of the final paper.

Thank you very much for the detailed response provided by the author. In response, I will also provide the following detailed comment:

Learning a policy (i.e., PPO) is not required for RLHF. RLHF as a framework is the process of defining a policy via a reward signal derived from human feedback.

Which published papers did the author derive this statement from? If RLHF does not require learning a policy, then is the "RL" in RLHF not reinforcement learning, but rather reward learning?

Improving the ability of reward models to assign preference scores to responses is the most fundamental improvement one can make to a reward model.

Improving the discernment of responses is the ultimate goal of the BT reward model, but not of the task of reward modeling itself. Reward models can take many forms, such as more dense token-level supervision. The author mentions that Cloud utilizes the language capabilities of the language model itself to improve scoring accuracy, but this process still requires a more powerful model, such as GPT-4o. When accessing GPT-4o, there are many methods to improve the scoring accuracy of the BT RM. I do not understand the necessity of Cloud if it is merely an enhanced version of the BT RM. (The enhancement here is still questionable because it does not necessarily improve the performance of methods like PPO.)

Cloud does introduce overhead at inference as compared to classic reward models, however, the improvement in reward modeling performance justifies this overhead.

Instead of writing lengthy explanations, I would prefer to see the author use tabular data to present the differences in inference time consumption between Cloud RM and baseline methods.