Self-Taught Evaluators

Concerning the explanation provided in line 044 and Figure 2, could you elaborate on the methodology used to ascertain the contrastiveness of the generated synthetic preference pairs beyond the prompt instructions?

We extract the final verdict from the synthetic preference pairs to ensure they are contrastive pairs, i.e., the pair must have opposite final verdicts [[A]] and [[B]].

The definition of "similar" in Figure 2 appears ambiguous. Specifically, while x' is deemed similar to x, the term "similar" implies a high level of relevance without strict semantic identity. Clarification on this aspect would be beneficial.

We have updated the paper to improve the accuracy of our writing. Specifically, we now refer to x' as the "modified instruction" instead of a "similar instruction".

Several typographical errors, such as the presence of "?" in Figure 2 and line 241, have been noted.

Fixed.Thanks!

The citation format lacks consistency, as evidenced by variations between line 137 and line 139. It would be beneficial to standardize this aspect.

Thank you for bringing this to our attention. We have noted that other reviewers share the same concern. In response, we have thoroughly proofread our paper to eliminate such inconsistencies and ensure consistency throughout.

In terms of experiments, aside from Llama, have results been obtained using different models of varying sizes?

Please refer to Table 12 in the Appendix. We conducted experiments with Llama2-70B, Llama 3 (8B and 70B), and Llama 3.1 70B using our approach, limited to the first iteration. The results consistently show significant performance improvements across all models.

Could you provide insight into the interpretation of performance fluctuations observed in Chat and Reasoning across multiple iterations?

Our experiments with RewardBench revealed a competition between different categories. For instance, across iterations, we observed that the performance of the "chat" category improved while the performance of the "chat-hard" category declined. We speculate that this is due to the distinct capabilities required by each category; the "chat" category emphasizes stylistic and writing tone aspects, whereas the "chat-hard" category prioritizes factuality and correctness. We believe that such competitions may contribute to performance fluctuations in certain scenarios.

First we will go over questions, and then over suggestions for improvements:

Validation of Prompt Selection and Evaluation

It happens that the reasoning category is rather a vague term within LLM benchmarks. For example, coding and math prompts are part of the reasoning category in the rewardbench such as math-prm, hep-cpp, hep-go, hep-java, hep-js, hep-python, hep-rust datasets. This is why we consider this to be more challenging compared to other prompts such as creative writing and other chat-like prompts.

Judgment Selection Consistency

We included examples of inferior responses (Table 15 in the Appendix). In addition, all the data is planned to be released to analysis and use by the community. We did not make an extensive verification of the worse response quality w.r.t., a better one since we wanted to reduce the amount of human annotation signal that we might inject in this research design.

Majority Vote Implementation

Majority voting is implemented based on the fact that the final verdict (A vs B) is parsable. That is, judgements are generated N times, N verdict signals are extracted, and the final verdict corresponds to the most common one following the counts. This improves our model from 88.3 to 88.7 on RewardBench, as shown in Table1.

Synthetic Data Generation Process

We included some details about generations in Section 4.2. We will share our data as well as our code for data generation on Github.

MT-Bench Experiment

We had only used 1st turn data for MTBench evaluation since our training data does consider any multi-turn conversations. We will make this clear in our evaluation description!

Now we go over improvement suggestions:

Validation of Pipeline Components and Ablation Studies

We agree that more validation signals would help, but the best validation signal is the performance we got using the resulting model. Due to time constraints, we won’t be able to add more validation experiments, and we think it does not limit the overall results delivered in this work. As per ablation studies, we have included multiple ablations about different data sources that is one of the foundational pipeline settings (the source of prompts). We believe future work might be coming from ablating this pipeline and proposing more advanced versions of it.

Generalizability

We hope that our pipeline will be attractive enough and tested with other model families by other researchers.

Bias analysis

This is a valid point. We believe this direction is crucial and warrants our attention, even though it's not currently reflected in popular benchmarks. In this paper, although we haven't conducted specific bias analysis, we did try to measure position-consistent accuracy: the accuracy when the model consistently makes correct predictions, even when swapping the order of two response options. This metric is reported on the HelpSteer validation split in Table 3.

More baselines

We used mt-bench evaluators following their guidelines to keep our numbers reproducible w.r.t., the other results reported in literature. For rewardbench we have other baselines such as gemini model from Google and larger size llama models. We also compare to other LLM-as-a-judge models (most of them came out after our submission) as shown in Table 14 in the Appendix.

We hope our answers and manuscript revisions (please see updated PDFs with blue colored revisions) convince you to adjust your score. Thank you!

Errata: Thanks for spotting these errata.

We have fixed all of them. Please see the updated paper.

Overfitting on instructions and responses

While we acknowledge the risk of overfitting with iterative training, our approach differs in that the instruction and response pairs remain constant, but the training data has new preference pairs in each iteration. These preference pairs include chain-of-thought reasoning and final conclusions, which introduces additional diversity and complexity to the training data. This can help alleviate the overfitting issue to some extent, as the model is exposed to a wider range of preferences and reasoning patterns, encouraging it to generalize better and adapt to new information.

Since the model is being fine-tuned to re-produce its own previous outputs (with some filtering applied), how can it learn to generate better critiques? Where is the exploration coming from, and why would it favor generating good critiques over poor ones which still agree with the ground truth?

Our model iteratively improves its critique generation through a filtering mechanism, similar to rejection sampling methods used in large language models like Llama3.1. This approach ensures that the model is trained on high-quality data that aligns with the ground truth. Poor critiques are filtered out, which in our setting means those that incorrectly predict the better response. By refining its training data, the model can produce more accurate and reliable critiques over time.

Iterated training does not always help.

Our experiments with RewardBench revealed a competition between categories, where improvements in one category (e.g., "chat") may come at the expense of another (e.g., "chat-hard"). This is likely due to different categories requiring distinct capabilities. Additionally, we observed fluctuations in performance across iterations, which can be attributed to the limitations of self-iterative methods and the model's capabilities given the seed training data.

From table 1, the method is beaten both overall and in the "Safety" and "Reasoning" categories by other methods.

Our model outperforms all LLM-as-a-judge baselines in Table 1, achieving an overall performance of 88.3. In comparison, fine-tuning a LLM-as-a-judge model with the same data only reached 85.6. While we surpass Gemini 1.5 Pro overall, we lag behind in the reasoning category, possibly due to our starting point being llama3-70B instruct, which is not strong in reasoning.

The main motivation behind training a reward model is to provide a reward signal to optimise a downstream LLM's outputs to better reflect human values. The proposed method has not been evaluated on its ability to provide such a reward signal.

Such analysis would be interesting to see. On the other hand, unlike reward models that only output scalar numbers or rankings, our model generates an evaluation plan (CoTs) that outlines how to make a judgment. These CoTs can be utilized in various ways, such as refining the original model response, analyzing model mistakes, understanding model behavior, and more.

What happens if you train an LLM to optimise the resulting reward model, especially on prompts not in the initial instruction distribution of the LLM-as-a-Judge?

A comparison between LLM-as-a-judge and reward models would be insightful. Beyond performance, LLM-as-a-judge offers a unique advantage: it provides not only the decision but also the reasoning steps behind it, offering transparency and insight into the decision-making process. This can be valuable in various applications.

Do you know or have any idea what happens if you include response pair construction or instruction selection as part of the iterative process?

This is not included as part of the iterative process.

How does the proposed method avoid model collapse, as is implied to eventually happen with these iterative recursive schemes by Shumailov et al. (2024, https://www.nature.com/articles/s41586-024-07566-y)?

We haven't observed model collapse, possibly due to our approach of randomly sampling high-quality chain-of-thought judgments from previous iterations. This may introduce sufficient diversity and complexity into the training process, helping to prevent model collapse.

Multiple seeds and computing standard error/deviation.

Thank you for your understanding. This is not feasible given our current computational resources.

In table 2, it's not clear whether the GPT4 baseline model was using 32 sample majority vote.

It's worth noting that the GPT4 baseline in Table 4 did not utilize 32 sample majority votes. When we remove majority voting from our results, our performance drops to 78.9, which is very close to the GPT4-0125 score of 79.1. Therefore, we claim that our model achieves on-par performance with GPT4.

The use of Mistral 22Bx8 Instruct for generating the initial synthetic responses.

We utilized the Mixtral model to generate initial synthetic responses due to our existing infrastructure, which enabled rapid inference with this model. However, we emphasize that this choice is not necessary. As demonstrated in the Appendix (Table 10), replacing the Mixtral model with llama3.0 models for generating synthetic responses or judgements yields similar results, highlighting the flexibility of our approach.

Regarding the experimental section and the baselines, the authors mentioned some related work that sounds to be highly applicable in the studied setting, such as the Best-of-N method. Were there any attempts to compare against it?

For a detailed comparison with competing models, please refer to Table 14 in the Appendix. It is important to note that these competing models are built using a large amount of human-annotated datasets, whereas our model only requires synthetic datasets without any human labels.

Regarding the "West-of-N" paper, they do not report any results on RewardBench and have not released their code. As a result, it is currently challenging for us to compare our model with theirs. However, we will make every effort to provide a comparison whenever they share their numbers or code.

The proposed method contains many steps / components that contribute to the performance of the policy. It would be nice to understand in which degree each component affects the solution. In table 2 it seems that the performance on iteration 5 is worse than on iteration 4.

We identify high-quality synthetic data and iterative training as the two most critical components of our model. The effectiveness of synthetic data is demonstrated in Table 4, where we conduct ablation experiments with different datasets. Similarly, the benefits of iterative training are evident in Table 1, which shows that performance improves from 83.9 (first iteration) to 88.3 (5th iteration). Notably, we observe that further iterations beyond this point result in only marginal improvements, suggesting that the model may have reached an upper bound with the given training data under our iterative training framework.

When comparing against the human data, were the sizes of the datasets identical? Was the user instructions with human labels filtered in the same way (which seems to preserve more complex instructions) as the user instructions used to generate the synthetic pairs?

The size of our datasets can vary slightly from one iteration to another. This is because we use the model from the previous iteration to generate new reasoning traces and judgments, and then select the correct ones. We did not apply the same filtering process to datasets with human labels, as these datasets are already carefully processed and filtered by their authors, and are therefore expected to be of relatively high quality.

Another reason for this approach is to provide a fair comparison between our model and the baselines. By using datasets with human labels in the same way as other papers, we ensure that our results are directly comparable to those reported in the literature.

Putting brackets around citations would make it easier to read the text.

Thanks for pointing this out! We have fixed it.

Line 090: West->Best?

It’s actually “West”, please check out this: https://arxiv.org/html/2401.12086v2

For the understanding of the algorithm, I would like to see some examples of the "related but different" prompts and to see in what kind of answers they result.

Please see the examples included in the appendix (Table 15).

What proportion of the traces agree with the synthetic labels over the training iterations?

This is a good question, and we haven't conducted such an analysis yet. Intuitively, we would expect the agreement to increase over training iterations as the model becomes more refined and accurate.

Details about the model selection (line 225-227 are not very clear)

For model selection, we utilize a metric that aggregates two types of accuracies:

1. Accuracy when the model makes a correct prediction.
2. Accuracy when the model consistently makes a correct prediction even when swapping the order of two response options.

There are N samples generated and the majority vote is performed for the proposed method (line 244-247). Is the same done for the baselines?

No, this is not the case for baselines. As evident from Table 1, majority voting does not provide significant improvements (88.3->88.7), indicating that it may not be an effective approach in this context.

What is the conclusion from Table 6?

Mixing data for building a Large Language Model (LLM) as a judge is not a straightforward process. Although this result may be considered negative, we believe it's essential to share our findings with the research community to contribute to the ongoing discussion and exploration of LLMs.

I would consider revising the score if the paper includes a more comprehensive discussion of potential drawbacks, along with a comparison to other LLM-as-a-judge baselines.

Thank you for your kind comments. We have taken your feedback into consideration and enhanced the Limitations section to provide a more in-depth discussion of potential drawbacks. Additionally, we have included comparisons to other LLM-as-a-judge baselines and experiments on different models above, which are also temporarily appended at the end of the Appendix (Table 13, 14) for reference. Please refer to the updated paper for these changes.

In Table 13, we applied our approach to the LLaMA2, LLaMA3, and LLaMA3.1 models. Due to time constraints, we only present results after the first iteration of supervised fine-tuning. Our approach consistently demonstrates performance improvement across different models, even with just one iteration.

In Table 14, we present a comparison between our Self-taught evaluator and several other LLM-as-a-judge models. The numbers are taken from the RewardBench Leaderboard. Note some of the models came out after the submission of our paper. The state-of-the-art (SOTA) performance is achieved by [1], where they fine-tune the Llama-3.1-70B instruct model on a pool of various human-labeled preference datasets, totaling 80K pairs. The remaining models (except for our Self-taught evaluator) are built on top of different base models but all rely on human-labeled preference datasets. In contrast, our Self-taught evaluator is based on the Llama-3.0-70B instruct model and only 10k synthetic pairs. Despite this, it still achieves good performance, demonstrating its effectiveness as an evaluator.

[1] (Tu et al., 2024) Skywork Critic Model Series.

A more carefully crafted design for preference pairs could improve the model’s ability to distinguish subtle judgment differences.

Thanks for suggesting! The focus of this paper is trying to present a framework for building powerful LLM-as-a-judge models by using high quality synthetic datasets. We plan to further improve our model by refining our synthetic data creation pipeline as you suggested here.

Further discussion on the computational demands and any associated performance gains.

Our approach inherently requires additional computational resources due to the multiple iterations of training and evaluation on validation sets. To address this, we plan to optimize the efficiency of iterative training in our future work and provide a detailed analysis of the computational demands involved.

The quality of judgments has not been independently verified.

Determining the correctness of judgments remains a significant challenge. Investigating methods to derive accurate judgments is an intriguing research question that warrants further exploration. We believe that high-quality judgments have the potential to be even more beneficial, as our straightforward approach has already demonstrated promising results.

While the performance of the self-taught evaluator generally improves with each iteration, there are scenarios (e.g., reasoning ability for RewardBench doesn't increase as more iterations are gone through) where performance declines.

Through our experiments with RewardBench, we observed a phenomenon where different categories competed with each other. Specifically, we found that the performance of one category could improve at the expense of another. For example, the "chat" category might show improved performance while the "chat-hard" category experiences a decline. We speculate that this competition arises from the fact that different categories require distinct capabilities. The "chat" category focuses on stylistic and writing tone aspects, whereas the "chat-hard" category prioritizes factuality and correctness. This competition may contribute to performance declines in certain scenarios.

The paper lacks clarity on the amount of synthetic data needed per iteration and how performance might vary with different volumes of synthetic data.

We used about 10K synthetic data examples in our experiments. We have included such details. Please see the updated paper.

Please use \citep in place of \cite where appropriate to ensure citation consistency.

We have fixed this. Please see the updated paper.