Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

We thank the reviewer, but we respectively disagree with the main rejection reasons. See our responses below.

The proposed approach requires more computation (compared to their baseline Yuan et al). Hence, if the comparison is for a fixed compute budget, which is the best data augmentation method, one needs to redo the experiments (e.g., Fig3) to obtain compute-adjusted comparison results. I acknowledge that it's fair to control for "iteration" only if the goal is to solve the saturation problem in data augmentation, but that’s a different goal.

This is not accurate. Spending more compute in Self-Rewarding LLM does not necessarily lead to performance improvement. This is evidenced by the saturation observed in Self-Rewarding LLM in Figure 3, which shows that while we may aim to use more compute, it does not result in better outcomes. Even within each iteration, we rarely use the final checkpoint during checkpoint selection, indicating that allocating more compute for training Self-Rewarding LLM often leads to overfitting rather than improved performance. In contrast, Meta-Rewarding offers a more effective way to utilize additional compute per sample, significantly enhancing performance and mitigating the saturation problem.

If you can take off-the-shelf SFT models, and assume “no human annotator available”, why can’t you take off-the-shelf reward models under the same assumption.

Using an external reward model is a well-studied research problem but is outside the scope of our paper. Our method assumes a self-improving training setup where no external reward model or human data is available. Training an external reward model requires human annotation, which is costly to obtain. Furthermore, our setup is closely related to the challenge of superalignment, where the model reaches a regime in which humans can no longer provide reliable feedback. In such a regime, distilling a reward model from human feedback becomes infeasible. As a result, relying on off-the-shelf reward models inherently risks degrading the model's performance.

The setup assumes that the base LM is bad at judging (or otherwise, you wouldn’t need to train the judging ability) but good at meta-judging

It is incorrect to state that our assumption is that the base LM is "bad" at judging—especially since "bad" is difficult to define objectively. Instead, we assume that the model makes mistakes, which is a reasonable and realistic premise. We further assume that, when the model compares two judgments, it can identify some of these mistakes. The Meta-Judge operates based on a detailed rubric, evaluating each aspect of the judgment according to specific criteria, which makes it easier to detect flaws. Additionally, multiple judgments are compared in this way and ranked using the ELO metric, further improving the accuracy of the evaluations.

It is unclear what is the upper limit of self-rewarding [3] and meta-rewarding (this work) as the former only reports performance up to 3 iterations and this work only reports performance up to 4 iterations. If, after these iterations, the performance stops increasing, then this should also be reported so that the community can learn about the upper bound of self-rewarding mechanisms.

We have not performed more than four iterations due to various resource limitations. Of course, we do not expect performance to improve indefinitely. However, we have demonstrated that Meta-Rewarding scales better than Self-Rewarding LLMs. This is evidenced in Figure 3, where Meta-Rewarding enables more effective use of compute resources before performance plateaus. While it is possible to determine the exact iteration where performance saturates, that number would likely depend on factors such as the specific model, training prompts, and overall setup, which may not generalize to other scenarios. The general rule of thumb is simple: continue iterating as long as there is improvement and resources allow.

Epsilon equations

Right before the equation, we defined $\epsilon_n$ as the ELO score corresponding to judgement $n$. It is a scalar instead of a vector. While $\epsilon$ is the vector with $\epsilon_n$ as its elements.

Your work is also related to the growing literature on inference-scaling. As the prior work has shown, inference-scaling works.

Our method is fundamentally different from inference-scaling papers (most of which were published after or concurrently with ours) because it does not increase inference-time compute.

Missing citation

Thanks for the suggestions. We will take them into consideration in the related work.

This is evidenced by the saturation observed in Self-Rewarding LLM in Figure 3

The so called "saturation" in fig3 is within error margins. I am not ready to accept this as saturation with the current evidence. I would be convinced if you show the trends beyond iter4, say up to 10 iterations.

We have not performed more than four iterations due to various resource limitations

Can you elaborate on this? What form of costs are we discussing here? (you don't discuss these in the paper) To my understanding, here "cost" does not change much from one iteration to another. Given that you had resources to run this for four iterations, why is it impossible to run this for a few more iterations?

Also you completely ignored several of my comments. Am I correct that?

The proposed approach requires more computation (compared to their baseline Yuan et al).

If so, have you tried the following suggested plot? (if you have the cost statistics, it should be pretty easy to plot this)

Hence, if the comparison is for a fixed compute budget, which is the best data augmentation method, one needs to redo the experiments (e.g., Fig3) to obtain compute-adjusted comparison results.

You also did not address this comment:

This work only investigated ONE base model (llama3-8b-inst) on TWO different benchmarks (Arena Hard, Alpaca Eval 2.0),

We appreciate your review of our paper.

These limitations of the judgment process, bring serious concerns regarding the scalability of the Meta-Rewarding approach.

Yes, as demonstrated in the paper, there are areas where improvements can be made. One of them is the saturation of the judge score, which resulted from following the setup of Self-Rewarding LLMs. Using a different scoring system or adopting pairwise comparisons could help alleviate this issue. Additionally, as you pointed out, the Meta-Judge becoming biased is another challenge, which we leave as future work. We hope that addressing these issues will further enhance our method. Nevertheless, the current version of our method already demonstrates improvements at the 8B model scale and is likely to scale effectively to 70B models as well, given that Self-Rewarding LLMs have proven successful at that scale.

In Table 3, the judgment agreement increases in iterations 3 and 4, even though only the actor model is trained in these iterations. What explains this improvement?

While the judge is not explicitly trained, it does evolve because it shares weights with the actor, which is being trained. The actor is trained on general instructions, and judging tasks can be considered a specific instance of general instruction following. This allows for positive transfer from actor learning to the judge. This phenomenon was observed in the Self-Rewarding LLM paper (Table 4), where the judge improved despite not being explicitly trained. However, as you mentioned, Table 3 is not the most reliable metric. Instead, we should focus on the model's final performance, where significant improvements are observed.

Given the failure of improvement of judgments due to biases in meta-judge, does it make more sense to have two different LLMs one that mainly focuses on improving the judge and another that focuses on only improving the actor? Perhaps training both tasks simultaneously into a single model leads to biases in Meta-Judgements.

Yes, this is an interesting direction to explore. Using a separate model would mean that the Meta-Judge remains fixed due to the lack of training data for it, which could help mitigate increasing bias. Another approach could involve making the judge pairwise, similar to the Meta-Judge, so that training the judge could also contribute to reducing positional bias in the Meta-Judge. We plan to explore these directions in future work.

Even with the length conditioned training, the average length of responses keeps increasing up until iteration 3 in Tables 1 and 2. By virtue of the preference selection process, if the chosen responses are guaranteed to be smaller than rejected responses in DPO training, is there any intuition why the length still increases in Meta-Rewarding process?

The increase in response length is a general phenomenon observed in the preference alignment process and is not specific to Meta-Rewarding. Researchers are actively investigating this issue, and methods like ours have been proposed to help mitigate it. It is worth noting that, in our case, the chosen responses are shorter on average; however, this does not guarantee that a chosen response will be shorter than the rejected one for every prompt.

From Figure 3, it seems like the performance from Meta-Rewarding hasn't saturated on Alpaca Eval. I'm curious if authors tried more iterations of this process and if it helped.

No, we have not conducted additional iterations due to a lack of data—specifically, running out of prompts—as well as time constraints and resource limitations.

Thank you for reviewing our paper.

A logical concern arises regarding the authors’ approach of using the LLM as a meta-judge to improve its judging ability. If so, wouldn’t the meta-judge also need enhancement? Would this necessitate a meta-meta-judge?

Not necessarily. Even if the Meta-Judge’s performance remains constant, it can still improve the judge as long as it has good accuracy. This phenomenon was also demonstrated in the Self-Rewarding paper, where the judge was not explicitly trained yet still contributed to improving the actor''s performance. Our results further support this observation, as the Meta-Judge clearly enhances overall performance. While improving the judging capability of the Meta-Judge would likely lead to even greater performance gains, we leave that for future work.

The authors compare the Meta Rewarding and Self Rewarding methods, showing performance improvements. However, I am curious if computational costs were aligned in the comparison. This includes the overhead for generating Actor Data and Judge Data and the amount of data used for training after filtering. Especially given that the Meta Rewarding method requires additional Judge Data for training, clarifying this aspect is crucial to alleviate concerns about performance gains resulting merely from increased data construction and training costs.

As background information, naively increasing compute for alignment might reduce performance due to overfitting, making it more important to focus on how much compute can be used effectively before performance saturates. This is evidenced by the saturation of Self-Rewarding LLM shown in Figure 3, which indicates that while increasing compute may seem beneficial, it does not necessarily yield improvement. In contrast, Meta-Rewarding offers a better way to utilize more compute per sample, significantly improving performance and mitigating the saturation problem. Additionally, even within each iteration, we rarely use the final checkpoint during checkpoint selection, highlighting that allocating more compute for training the Self-Rewarding LLM does not improve performance, as the model begins to overfit.

We thank the reviewers for taking the time to review our paper.

The primary contribution of this paper is utilizing Meta-Rewarding to address performance saturation during iterative training, however, the judge score shifting results in less training iterations of the judge model than the actor model. This limitation not only results in the diminished performance of the judging model in the final iteration ...

The intended purpose of the paper is to improve the judge’s performance, as demonstrated by the fact that adding the Meta-Judge did enhance the performance of Self-Rewarding LLMs. Compared to Self-Rewarding, the Meta-Rewarding Iter4 judge achieves approximately 5-6% better agreement on GPT-4 Chosen Pairs. The only difference between them is the addition of the Meta-Judge. However, GPT-4 agreement is not a perfect metric, as GPT-4 can make mistakes. Instead, the final instruction-following performance should be used as the primary metric, where we observe significant gains.

The paper lacks sufficient discussion of the rationale for employing the model as its meta-judge in supervising the judging model. A critical question that emerges is the underlying mechanism behind the effectiveness of this self-supervision approach.

The rationale behind the Meta-Judge is similar to the LLM-as-a-Judge paradigm. Since LLMs are trained to follow general instructions, they can be prompted to perform evaluations, enabling the LLM to judge its own outputs. Similarly, LLMs can be prompted to assess the quality of their judgments, allowing the model to evaluate its own evaluations. This approach works because it breaks down the problem of determining “which judgment is better” into verifying whether each judgment adheres to the rules outlined in the detailed rubric. Analyzing different aspects of a judgment in this way makes it easier to identify flaws that may have been overlooked during the initial judgment generation. The effectiveness of this method is demonstrated in our experiments, which show significant improvements.

In the implementation of the judge model and meta-judge model, the judge model is a point-wise scorer while the meta-judge model is a pairwise evaluator. What is the underlying rationale for this differentiation?

For the Meta-Judge, we found that using a pointwise judge can result in high scores that barely differentiate between judgments, providing less signals for training compared to a pairwise judge. In contrast, the lack of separability is less problematic when training the actor, so we retain the same setting as the Self-Rewarding LLM for better consistency.

What would be the implications of adopting a consistent approach across both models - either exclusively point-wise scoring or pairwise evaluation?

If both models use pointwise scoring, there is not much to change in the algorithm, except that the performance may vary depending on the pointwise judging capability of the Meta-Judge. However, if both models use pairwise scoring, the Meta-Judge must evaluate a pair of "pairwise" judgments. In this case, it takes two responses, A and B, as input, along with two judgments with differing results (e.g., scores indicating A > B and A < B). The Meta-Judge then determines which judgment is better.

Regarding the evaluation of reward modeling, it would be valuable to conduct a comparative analysis between the Meta-Rewarding LLM and several established reward models, …

The goal of this paper is to enhance self-improving training, where the model cannot rely on external models or additional data. Specifically, we focused on improving the judging capability of self-rewarding LLMs. Comparing this approach to established reward models is not entirely fair, as those models are not trained in a self-improving manner and typically depend on extra human data or outputs from stronger models, such as GPT-4.

Recent self-improvement approaches, such as iterative DPO[6], SPPO[7], and SimPO[8], have demonstrated considerable performance gains in AlpacaEval 2.0 when utilizing external reward models (RM) ...

Our setup is closely related to the challenge of superalignment, where the model reaches a regime in which humans can no longer provide reliable feedback. In such a regime, distilling a reward model from human feedback becomes infeasible. Therefore, specific results in [6], [7], and [8] are not self-improving approaches, as they rely on external reward models. That said, those loss functions can still be incorporated into self-rewarding training, as we demonstrated with DPO in our paper. Whether using an external reward model is advantageous depends primarily on the quality of the specific model. Thus, there is no contradiction. We are not arguing that external reward models shouldn’t be used; some external models may perform better in certain settings. Consequently, there will be scenarios where using external rewards is the better choice.