JudgeBench: A Benchmark for Evaluating LLM-Based Judges

We thank the reviewer for your valuable feedback.

On judges performing worse than random guessing. Please see "Response to common comments" for our response to this. Also, we provide a failure case analysis in Appendix A.2 in which we explains why many of the fine-tuned judges performs poorly. Please refer to "On failure cases" above for more details as well.

On key insight not verified carefully. Please see "Response to common comments" above for how this insight is verified through Figure 4. Our key insight states that if a model cannot generate consistently correct responses (sampled with a temperature of 1.0) across k trials, then the same model will struggle to distinguish between these k responses. Table 4, on the other hand, pertains to a different experimental setting in which we study the performance of several models on solving (greedy decoding, one trial) the set of questions identified via our key insight (i.e., these were only questions which GPT-4o struggled to consistently answer correctly). In this context, the solver's accuracy is not indicative of the difficulty in distinguishing between response pairs. Instead, the key takeaway from Table 4 is that idetifying a correct response in JudgeBench is highly correlated with, and nearly as difficult as, solving the underlying problem itself. This reinforces the challenging nature of our dataset.

On reference-based judges. Our study already incorporates reference-based judges in the form of the Arena-hard judge, which is one of the two prompting methods we evaluate: Vanilla judge and Arena-hard judge. The Arena-hard judge, detailed in Appendix A.5 L1080, operates exactly as the reference-based judge mentioned in the feedback. It first generates its own reference solution before making judgments, unlike the Vanilla judge, which directly outputs a judgment. When comparing Arena-hard judge to Vanilla judge (See Table 1), we see a notable accuracy improvement (from 50.86% to 56.57%), showing that reference-based judges do improve performance on reasoning tasks.

On conclusion from L426 being generic. We appreciate the feedback regarding the need for more detailed discussions on improving judges' reasoning ability. While we leave improving the LLM-based judges as future work, we expand here on some potential approaches informed by the insights gained from JudgeBench.

• Agentic judges with tool-use ability. For complex tasks like math and coding, incorporating external tools such as calculators or code interpreters can facilitate more accurate verification. The challenge lies in designing AI agents that can extract intermediate results from prover responses, apply external tools effectively, and synthesize the tool outputs into final judgments. This approach could significantly enhance reasoning accuracy.
• Adversarial training for verifiers. Our dataset can be viewed as "adversarial", in a sense that we explicitly design it to produce response pairs that are difficult to distinguish. A promising direction involves adapting adversarial machine learning techniques to train robust verifiers. For instance, recent work [1] has explored training provers to generate both valid and misleading proofs, with verifiers trained to distinguish between them. While the focus of this work is on improving the legibility of prover's response and only evaluates on the GSM8k dataset, the idea could be applied to train a stronger verifier on general reasoning problems.

We will integrate these discussion into the section.

[1] https://arxiv.org/pdf/2407.13692

On evaluating positional bias. While we do not explicitly measure positional bias, our evaluation method inherently accounts for it. As described, we aggregate the judge's decisions across two independent trials with swapped response orders. If the judge's responses are inconsistent (e.g., A > B in one trial and B > A in the other), the aggregate decision is recorded as A = B, which is deemed incorrect. This discrepency is thus directly discredited in our accuracy metric. Furthermore, in our failure case analysis (Appendix A.2), we measure the ratio of inconsistent responses from judges. This metric can indeed serve as an indicator to quantify positional bias, as you suggest.

We thank the reviewer for your valuable feedback.

On the size of JudgeBench. Please see "Response to common comments" above.

On judges performing worse than random guessing. Please see "Response to common comments" above.

The key insight not empirically verified. Please see "Response to common comments" above.

On extensive validation. We employ many strategies to ensure the quality of our pipeline represents truly challenging cases. First, as detailed in Section 3 (Data Filtering and Selection), we perform a meticulous filtering process. We utilize GPT-4o-mini as a solution verifier to remove responses that are incorrect solely due to formatting issues. Upon manual inspection, all filtered-out responses were confirmed to contain only formatting errors. For coding problems, incorrect responses are further validated using a coding checker to ensure errors are not due to formatting. This process guarantees that incorrect responses are genuinely incorrect and not artifacts of formatting.

Second, we did extensive experiments to evaluate different LLM models on JudgeBench. The relative rank of the models are, in general, aligned with ranks from other reasoning benchmarks (e.g. GPQA, MATH), where larger models (e.g. Llama-3.1-70B-Instruct vs Llama-3.1-8B-Instruct) and models with stronger reasoning ability (e.g o1-preview, Claude-3.5-sonnet) in general performs better on this task. This indicates that our benchmark is not artificially designed to be challenging to models (for example, one could ask LLM to guess a random coin flip's result and all LLM's accuracy wouldn't surpass 50%), but indeed truly reflects the reasoning ability of the judges.

Finally, our in-depth analysis of failure cases (see Appendix A.2) provides insights on why some finetuned judges underperform random guessing. See "On failure case analysis" from "Response to common comments" above for more details. These observations further validate the reliability of JudgeBench in assessing judges' reasoning ability.

On addressing data contamination. Thank you for raising this important point! Our work indeed takes data contamination into account. The pipeline we propose in JudgeBench is very flexible and can be adapted to transform any challenging dataset with verification mechanism into challenging response pairs. In fact, many of the datasets we use in JudgeBench are already "live" (e.g., LiveBench, LiveCodeBench). These datasets are specifically designed with contamination in mind and are regularly updated to mitigate this issue. As LLMs continue to evolve, our pipeline can be applied to transform more challenging datasets (e.g., Olympiad Math) into this format, ensuring that the difficulty of our dataset evolves over time.

Thank you for your valuable feedback.

On the dataset size Please refer to "Response to common comments" above.

On failure cases. Please refer to "Response to common comments" above.

On overlooking Principle 1 and 3. While we do not cover stylistic preference (Principle 3), our work can be viewed as covering both Principle 1 and 2. This is because to answer the question correctly, the model has to first follow the instruction and answer that question, and failure to follow instruction would definitely result in a wrong answer. With that being said, we would like to highlight that the main emphasis of our work is indeed Principle 2. This is because Principle 1 and 3 has both been previously studied (e.g., LLMBar and MTBench), but Principle 2 is rarely looked at. Our work is complementary to these prior benchmarks and aim to fill the gap for Principle 2, which is missing in the current literature.

On accuracy drop for self-evaluation. Thanks for raising this interesting point. LLM's ability to self-evaluate is an open research question, with some evidence suggesting they can [1,2] and other evidence suggesting they cannot without some form of external feedback (e.g., tools or oracle labels) [3,4]. Our results do not contradict the positive findings, since despite the observed drop in performance, LLMs in our study can still distinguish their own correct and incorrect responses a certain percentage of the time. This indicates that refinement based on self-evaluation is indeed possible. However, based on our results, leveraging another model with similar overall capabilities may prove to be a more effective approach for evaluation.

[1] https://arxiv.org/abs/2303.17651 [2] https://arxiv.org/abs/2305.11738 [3] https://arxiv.org/abs/2402.11436 [4] https://arxiv.org/abs/2310.01798

Thank you for your valuable feedback.

On the lack of technical novelty. A significant, but overlooked, contribution of this work is the novel pipeline used to generate the JudgeBench benchmark, which can transform any existing dataset for evaluating LLM performance into a dataset for evaluating the performance of LLM-based judges. This is critical, as there exist far fewer meta-evaluation benchmarks than standard LLM benchmarks, despite recent research supporting the importance of strong judges/verifiers for training more reliable models [1,2] and for effectively scaling inference-time compute [3,4].

On length bias. Thanks for raising this point. Because each of our pairs contains two responses sampled from the same model, rather than from two different models, the responses tend to be of similar length. On average, across all instances of JudgeBench, correct and incorrect responses contain 562.29 and 561.16 tokens, respectively, using the GPT-4o tokenizer. This negligible difference demonstrates that the construction of JudgeBench effectively mitigates length bias, allowing LLM-based judges to be evaluated without this confounding factor.

On the strong performance on math. This is a rather interesting observation! We sourced the math and reasoning questions from LiveBench, whose leaderboard [5] indicates that LLMs tend to perform better on the reasoning category than the math category. We observed this too, with 3 of the 4 models in Table 5 achieveing higher accuracy on solving reasoning questions than math questions. Yet, when it comes to judging, these same models tend to exhibit higher accuracy on the math pairs than the reasoning pairs (see Table 5). Given this, we hypothesize that the strong performance we observed on the math subset of JudgeBench is because LLM-based judges are better equipped at verifying responses to math questions than other types of questions (e.g., reasoning), and not because the math questions themselves are less challenging.

On the granularity of the results. The existance of subcategories within JudgeBench is dependent on the source dataset. For example, the MMLU-Pro dataset from which we derive the knowledge category contains 14 subcategories (e.g., Law, Engineering, and Economics). Though JudgeBench includes an equal number of pairs derived from each subcategory, there are likely not enough pairs per subcategory to draw robust conclusions. Other source datasets include no natural subsets, e.g., our code category is derived from LiveCodeBench which is entirely in python.

[1] https://arxiv.org/abs/2407.13692 [2] https://arxiv.org/abs/2305.20050 [3] https://arxiv.org/abs/2408.15240 [4] https://arxiv.org/abs/2407.21787 [5] https://livebench.ai/

On the size of JudgeBench (bwzk, BxhQ, VLGm). First, JudgeBench is of similar size to related benchmarks. For instance, FairEval [1] contains 80 unique questions, LLMEval-2 [2] contains 480, MT-Bench [3] contains 80, and LLMBar [4] contains 419. RewardBench [5] is larger, but it's an aggregration of existing benchmarks, including MT-Bench and LLMBar.

Importantly, JudgeBench novel pipeline enables us to transform any existing dataset for evaluating LLM performance into a dataset for evaluating the performance of LLM-based judges. To demonstrate the flexability of our pipeline and the robustness of the results, we augmented our "knowledge" subset, increasing the number of response pairs from 154 to 770. We evaluated several LLMs using the Arena-Hard prompt, and observed that the relative rankings among these judges were the same between our original set and the augmented set, despite small variations in the scores themselves (see the table below).

[1] https://arxiv.org/abs/2305.17926 [2] https://arxiv.org/abs/2312.07398 [3] https://arxiv.org/abs/2306.05685 [4] https://arxiv.org/abs/2310.07641 [5] https://arxiv.org/abs/2403.13787

On judges performing worse than random guessing (BxhQ, VLGm). We explain our evaluation method in Section 4 (L286). Each response pair is evaluated twice, swapping the order of the pairs in the second trial. Judges' decisions are aggregated across both trials. For instance, if the decisions are (A1 > B1 and A2 = B2), the final aggregate decision is A > B. This final aggregate decision is what we use to compute the judge's accuracy on JudgeBench. Inconsistent decisions (e.g., A1 > B1 and A2 < B2) are interpreted as an aggregate of A = B and deemed incorrect.

This evaluation method is specifically designed to penalize random guessing. If a judge cannot produce consistent responses across trials, it indicates an inability to distinguish between the response pairs, equivalent to random guessing, which is marked as incorrect. Appendix B.2 (Tables 5 and 6) provides detailed statistics on fine-tuned judges' decisions, revealing a high ratio of inconsistent decisions as a major contributor to poor performance.

We would also like to highlight that, when comparing JudgeBench to other benchmarks, we apply the same evaluation methods to all benchmarks, so our benchmark does not gain any advantage from the evaluation method. Under this consistent setup, JudgeBench demonstrates higher challenge and separability than prior benchmarks.

On failure cases (bwzk, VLGm). Our work does provide failure case analysis, which is detailed in Appendix A.2 and referenced in Section 4.2 (L397). In A.2, we provide detailed statistic of fine-tuned judges decisions, and an in-depth analysis on their poor performance. One main reason for the poor performance is inconsistency between judgements. Table 6 shows that fine-tuned judges that fall below the random baseline all have pretty high ratio of inconsistent judgements (29.14%-59.71%), this indicates that they are incapable of distinguishing the correct response and are just doing random guessing. Other than inconsistent judgements, some judges (e.g. Prometheus2-bgb-8x7b, PandaLM-7B) frequently output invalid decisions (e.g. Neither A or B, 10/10, 3), which also contributes to the high error rate.

The key insight not empirically verified (BxhQ,VLGm). Our main insight posits: "If a model fails to generate correct, coherent responses, it will inherently struggle to distinguish between its own responses as a judge." This is empirically verified in Section 4.4 (Figure 4). We use Claude-3.5-Sonnet as the base model to construct datasets and evaluate accuracy across five representative LLMs. As predicted, Claude-3.5-Sonnet finds its own generated response pairs particularly challenging, with accuracy dropping significantly from 64.3% to 44.8%, ranking below GPT-4o and Llama-3.1-70B-Instruct. Importantly, the relative ranking of other models remains unchanged.

Meanwhile, the dataset's difficulty is transferable across models. When a stronger reasoning model (Claude-3.5-Sonnet) generates response pairs, they become more challenging for other models. All four other models show accuracy drops relative to GPT-4o pairs. This validates our insight and further demonstrates the robustness of JudgeBench.