JudgeLM : Fine-tuned Large Language Models are Scalable Judges

Thanks for the valuable comments and suggestions which help us improve this work. Following are our detailed responses to your concerns.

Question 1: The model seems not having a conversation on the model's judgment.

Response 1: We thank the reviewer for this question. The JudgeLM can have a conversation on its judgments. To prove that, we still take the question in Fig. 19 as an example and list the chat history below.

Human: Why not give the Assistant 1’s answer a score of 10?

JudgeLM: 10 would imply that the answer is perfect and there is no room for improvement. While Assistant 1's answer is accurate and relevant, it could have been more detailed by explaining why adults in the workforce are the target audience. For example, it could have been mentioned that these individuals often experience stress from work and may seek relief through a simple pleasure like eating ice cream. Therefore, while Assistant 1's answer is good, it is not perfect, hence the score of 7.

Human: Can you give a possible statement for the specific group mentioned in Assistant 2's response?

JudgeLM: Certainly, a possible statement for professional athletes competing in a championship game could be: "Recharge with a bowl of ice cream during your break from the intense game." This statement is tailored to the specific needs and circumstances of professional athletes, making it more relevant and helpful to this group.

Human: Thanks! I also want to get another target audience for the original statement, can you help me?

JudgeLM: Of course! Another possible target audience for the original statement could be: “Students studying for exams or working on projects.” This group might also appreciate a break with a bowl of ice cream, especially if they‘re feeling stressed or need a quick energy boost. This answer is also relevant and helpful.

Question 2: Given there is already a PandaLM paper, while there are still good points and methods proposed in this paper, the novelty is a little weaker.

Response 2: Thanks for the question about novelty. Compared with PandaLM, our method has these different contributions:

• We introduce a high-quality, large-scale dataset for judge models, enriched with diverse seed tasks, LLMs-generated answers, and detailed judgments from GPT-4, laying the foundation for future LLMs evaluating research.
• We analyze the biases inherent to LLM judge fine-tuning and introduce a series of methods to address them. Our methods significantly improve the consistency of the model in different cases, making the JudgeLM more reliable.
• We analyze the scaling ability of the language model judge and propose a series of JudgeLM, i.e., JudgeLM-7B, JudgeLM-13B, and JudgeLM-33B, for evaluating LLMs in open-ended scenarios.
• The proposed judging pattern, i.e., grading, judging, and reasoning, makes judging efficient, which only needs little time to grade and judge, and generates time-consuming reasons optionally.
• The proposed JudgeLM presents extended capabilities as shown in Fig. 1b, including grading single answers, judging multiple answers, judging multimodal models, and multi-turn chat.

Question 3: It would be good to have a comparison of training and testing with the old-fashioned chain-of-thoughts style.

Response 3: Thanks for your great suggestion. As mentioned in the Explanation-first Form part of the global response, we applied the JudgeLM with the explanation-first form, which brings a slight agreement drop and limited consistency improvement.

Question 4: How can the paper claim it can judge a multimodal model while the model itself cannot take vision data as input? This does not make any sense.

Response 4: Thanks for your question. As mentioned in the Generalization Ability part of the global response, the multimodal benchmark, i.e., MM-Vet, proposes to make judgments based on the question text, reference text, and LLM-generated answer using API-based LLM judges. The proposed JudgeLM can also make judgments based on these input texts. Furthermore, we hold the same viewpoint as yours, a better multimodal judge should also see the image. So, we plan to use multimodal LLMs, e.g., LLaVA, as the JudgeLM's backbone for better judging multimodal tasks. We leave it as a future work.

Question 3: Although incorporating reference answers does improve JudgeLM on validation data without reference, it could be better to conduct experiments where knowledge is injected under their retrieved form, instead of a well-organized answer.

Response 3: Thanks for your great suggestion. As shown in Table R11, we further inject original reference answers into paragraphs with different words, and evaluate JudgeLM-33B with the injected paragraphs as references in a 0-shot setting. It shows that JudgeLM-33B can use references in the retrieved form with a certain loss of agreement and consistency.

Table R11. Performance of JudgeLM-33B with injected paragraphs as references on JudgeLM val set.

Question 4: Yet among these methods, "reference drop" is basically equivalent to controlling the number of samples with reference support, and thus should be a hyper-parameter of the latter strategy instead of an extra method.

Response 4: Thanks for your question. We think the reference drop should be an independent method. At first, we argue that judging with or without references are two sub-benchmarks, which require judges to make judgments with internal knowledge or by comparing LLM-generated answers with a reference answer, respectively. The reference drop is not only a simple but effective hyper-parameter, but also an important method that bridges the two sub-benchmarks, which enables the JudgeLM to make judgments in different situations.

Question 5: Why is the data in Table 4 not aligned with the ones in Table 1?

Response 5: Thanks for pointing that out. As shown in 6.2 SCALING ANALYSIS OF JUDGELM, we analyze the scaling ability of the plain JudgeLM, which does not involve the proposed methods, i.e., swap aug, ref sup, or ref drop. We have underlined this in a revision of our paper.

Question 6: In section 6.1, Table 1 is referenced as "Table. 1", which is different from other table references.

Response 6: Thanks for pointing out the typo, we have fixed it in a revision of our paper.

Question 7: It could be better to compare the performance between this evaluation form and the explanation-first one JudgeLM itself instead of with the original PandaLM.

Response 7: Thanks for your advice. As mentioned in the Explanation-first Form part of the global response, we applied the JudgeLM with the explanation-first form, which does not bring significant agreement improvement. So, we propose to use a flexible score-first form. Furthermore, the proposed methods, i.e., swap augmentation, reference support, and reference drop, can improve the consistency significantly but bring no more inference costs.

Question 8: Using the same computation resources to compare JudgeLM and PandaLM.

Response 8: We thank the reviewer for the advice. As mentioned in the Efficiency Comparison part of the global response, we made the ablations with only 1 GPU, and the JudgeLM is 16.65 times faster than PandaLM when it does not generate reasons.

Thanks for the valuable comments and suggestions which help us improve this work. Following are our detailed responses to your concerns.

Clarification: JudgeLM aims to fine-tune LLMs as scalable judges. In this work, we study the biases in fine-tuning judges and propose methods to alleviate them. The three biases and corresponding metrics are defined as follows:

1. position bias: we use consistency to measure the judging results before and after swapping LLM answers, as shown in Table 1 and Table 5.
2. knowledge bias: as importing external knowledge helps to improve both objective judging ability and reliability, the agreement (w/ teacher) and consistency (w/ swap) mentioned before are all used to measure this, respectively, as shown in Table 1 and Table 6.
3. format bias: this bias indicates that fine-tuned judges perform poorly in different template formats, i.e., w/o reference v.s. w/ reference. So, the metrics in mismatched format are used to measure this, as shown in Table 6.

Question 1: The authors only compared the position bias between JudgeLM and GPT 3.5 or PandaLM, leaving the comparison on the two other biases within the JudgeLM category.

Response 1: We thank the reviewer for the question about comparison. We further analyze the three biases on PandaLM and apply the proposed methods to it, as shown in Table R8 and Table R9. We first transfer the 3.5K JudgeLM train samples into PandaLM format. Then, we train the PandaLM baseline, i.e., PandaLM*, with LLaMA-7B and the 3.5K train samples in PandaLM format. It can be observed that the PandaLM* also has position bias, which prefers the last answer. Besides, PandaLM* also faces knowledge bias and format bias, as shown in Table R9. Last, we apply the proposed methods to PandaLM*. As shown in Table R8 and Table R9, the proposed methods can also improve the performance of PandaLM*.

Table R8. Ablation study of PandaLM for the swap augmentation on our val set.

Table R9. Ablation study of PandaLM for the reference support and reference drop on our val set.

Question 2: Moreover, they did not report these metrics on GPT-4, the teacher of JudgeLM, to examine if their proposed data augmentation strategies can also help JudgeLM supersede GPT-4 over these aspects.

Response 2: We thank the reviewer for the suggested comparison with GPT-4. As shown in Table R10, we further list the metrics except for the agreement with GPT-4 itself. JudgeLM-33B achieves higher consistency than GPT-4, which demonstrates that JudgeLM-33B is more robust with position bias.

Table R10. Comparison between GPT-4 teacher and JudgeLM-33B on JudgeLM val set.

Thanks for the valuable comments and suggestions which help us improve this work. Following are our detailed responses to your concerns.

Question 1: Using GPT-4 generated data could limit the usage of the proposed method (as the GPT-4 license).

Response 1: Thanks for your important concern. We emphasize that the JudgeLM dataset is intended only for academic research and any commercial use is prohibited. Because the OpenAI's terms prohibit developing models that compete with OpenAI, the instruction-tuning datasets generated by the OpenAI's API, i.e., Alpaca, PandaLM, etc., all follow this rule. Thanks for your concerns, we have underlined this in a revision of our paper.

Question 2: The main evaluation of the judge system is the agreement with GPT-4, thus training on the GPT-4 generated judges may give the proposed method an unfair advantage compared to other methods.

Response 2: Thanks for your question. For a fair comparison, we also evaluate JudgeLM on the PandaLM test set in a zero-shot setting. The PandaLM test set is annotated by humans. As shown in Table 2, the zero-shot results of JudgeLM also outperform other judging methods, i.e., PandaLM, GPT-3.5, GPT-4. Furthermore, JudgeLM also achieves a superior 0-shot judging performance on the multimodal benchmark with human annotation, i.e., MM-Vet, as shown in Table R1.

Question 3: It would be nice if the paper could include some statistics on the type of question in the evaluation set.

Response 3: Thanks for your advice. We counted the distribution of questions in the JudgeLM val set as shown in Table R6, which has been added to a revision of our paper.

Table R6. Distribution of questions in JudgeLM val set.

Question 4: Since LLMs are currently still bad at certain reasoning tasks (such as counterfactual), how can we trust the evaluation results of such a judging system built by LLMs?

Response 4: Thanks for your concern. We agree with your concerns about LLM's reasoning performance at certain tasks. Nowadays, NLP researchers are still struggling with proposing LLMs with superior reasoning abilities. JudgeLM also needs the proposed reference sup method to enhance the judging ability for out-of-domain or counterfactual tasks, as shown in Fig. 11 and Fig. 16. Notably, the proposed JudgeLM can benefit from stronger foundation LLMs, e.g., the LLaMA2-7B-Chat-based [1] JudgeLM outperforms the original JudgeLM-7B on all metrics, as shown in Table R7. We hold the same viewpoint as Reviewer rHPM, the research of judge models is critical for the development of LLMs and can benefit from advanced LLMs, establishing a positive cycle.

Table R7. Comparison of different base models for JudgeLM-7B on JudgeLM val set.

[1] Llama 2: Open foundation and fine-tuned chat models

Thanks for the valuable comments and suggestions which help us improve this work. Following are our detailed responses to your concerns.

Question 1: The judge model lacks the granularity of its judgments and can only output an overall score. However, there may be many aspects to grade an open answer, e.g., factuality, fluency, novelty, and helpfulness. These fine-grained aspects should be considered in this work.

Response 1: Thank the reviewer for this suggestion. We further ask the JudgeLM to output the fine-grained results through the modified template. We also evaluate the JudgeLM-33B on the JudgeLM val set, as shown in Table R4.

Table R4. JudgeLM fine-grained evaluation results on JudgeLM val set.

Question 2: Is there any result of the performance in judging multiple answers?

Response 2: We thank the reviewer for the advice. As shown in Table R5, we calculate the consistency results in the judging form of answer pairs and multiple answers. We first generate answers on JudgeLM val set through 3 LLMs, i.e., Vicuna-13B, LLaMA-7B, and alpaca-7B, for evaluation. Then we use pairwise judging and multiple judging to grade answers and rank them, respectively. Last, we compute the consistency between the two ranking results. Please note that Error Rate@2 indicates the position orderings of two answers are different between the result of paired judgment and the result of multiple judgment, and Error Rate@3 means the position orderings of three answers are different.

Table R5. Performance of JudgeLM-33B in judging multiple answers on JudgeLM val set. We calculate the consistency between the pairwise judging results and multiple judging ones.

Question 3: The paper has mentioned the format bias resulting from mismatched prompt format. However, the prompt template of judging multiple answers also mismatches that of judging two answers.

Response 3: We thank the reviewer for this question. The judging of multiple answers is not misled by the format bias. We hold the viewpoint that judging multiple answers is an easy extension for JudgeLM, which does not change the basis for judging. As mentioned in 4 Inherent Biases - Format Bias, format bias means the model judging basis changes from pre-trained knowledge to reference, or vice versa. So, judging in mismatched situations faces format bias, as shown in Table 6, but judging multiple answers does not receive a significant performance drop, as shown in Table R5.

Question 4: One critical issue of the proposed method is its generalization ability to unseen tasks, which is important for an LM to be a general evaluation toolkit.

Response 4: Thank the reviewer for the question. As mentioned in the Generalization Ability part of the global response, we apply JudgeLM to different benchmarks following the 0-shot setting and it achieves superior performance.

Question 5: The main difference is that JudgeLM generates scores first followed by explanations and thus stopping generation before explanations can save time.

Response 5: Thanks for your question. As mentioned in the Efficiency Comparison part of the global response, we launched the ablations with only 1 GPU, and the JudgeLM still has superior efficiency when stops to generate reasons.