Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?

Thank you for the well-structured review! Please find below our responses to each of the weaknesses (W) and questions (Q) raised in your review. All references refer to the updated manuscript, where we highlight relevant changes in blue.

W1: One con for this work is the insufficiency of experiments to show the accuracy / reliability of specific components of the framework. For example, how reliable is the “Initial domain assessment” component for routing responses to specific tools?

The different datasets from varying domains aim to evaluate the reliability the individual tools (i.e., LongFact for the fact check, GSM8k for math check, and APPS for the code check tool).

However, we agree that more could be done to evaluate the impact of the other parts of the evaluation agent system, especially the initial domain assessment component. To test the impact of the other components on reliability, we run and share another set of experiments where we remove the "agentic" components (initial domain assessment, final decision and in-tool scaffolding) and simply use OpenAI's assistant API with a code interpreter and search tool. We test adding tools in this way to two of our baseline LLM-as-a-Judge methods: the simple pick-best and overall best-performing ArenaHard baseline. We the following observations: (1) adding access to tools without additional scaffolding does not notably improve performance of the baselines across any of the tested datasets and LLM-as-a-Judge configurations (often it decreases performance); and (2) adding tools reduces the output reliability of GPT-4o-based ArenaHard baseline.

We consider these new observations a strong indication that the non-tool components of our framework help the reliability of our system. See a more detailed discussion of these results in Appendix B of our updated manuscript.

W1 (1): In addition, showing the robustness of the framework as new tools are added to the agent can help strengthen the use-case of this framework.

The effect of adding new tools on the robustness of the framework depends on two factors: the configuration of the initial domain assessment and the impact of the tool on annotation performance when activated. For each tool, the initial domain assessment determines if the given annotation task is in-domain for that tools, by asking an LLM a series of questions about the annotation task. By limiting domains where the tool activates to domains where we have high confidence of the tools efficacy, we improve robustness. Additional care is required when two tools activate on the same annotation domain. Then, we recommend running ablation experiments to validate that joint tool usage is more effective than single-tool usage. Thus, the effect of adding a new tool can be controlled using these configuration parameters.

W2 (1): Lack of motivation as to why the specific tools (SAFE, OpenAI code, OpenAI math) were chosen for each component. Were there any other components tested?

Thank you for raising this point! We have clarified our motivation in Section 3 of the updated manuscript but also summarize this clarification as well as more details on the development process below.

Based on the chosen annotation domains (long-form factual, math and code tasks), we initially selected two categories of tools to evaluate: fact-checking tools and code-interpreter-based tools. For fact-checking, we quickly determined the SAFE method to be effective. Our development effort primarily focused on determining the domains where fact-checking works well. For math and code tasks, we initially tested a single code-interpreter tool but found it to not work well across code and math tasks. Thus, instead we created two separate code-interpreter-based tools with custom prompts and domain assessment, one for code tasks and one for math tasks. We updated Section 3 of our manuscript to better reflect this motivation.

W2 (2): How sensitive are the reported results to these tools?

Generally, as our baseline experiments and the AlpacaEval annotator leaderboard [1] indicate, LLM-as-a-Judge systems can be very sensitive to the specific prompting setup used --- even if the same base model is used (e.g. GPT-4o). In our experience, this sensitivity also applies to tool-use within our framework. As such, each tool needs to be carefully configured in terms of domain and output to ensure that the provided information is effectively used by the model.

[1] https://github.com/tatsu-lab/alpaca_eval/tree/main/src/alpaca_eval/evaluators_configs

Thank you for these clarifications and for updating the manuscript. As it stands, the manuscript has improved. I have also read the other reviews and agree with Reviewer Z3LQ on his concern about novelty. However, I appreciate the systematic approach the manuscript has done in testing these different tools. I have adjusted my rating to 6.

Thank you for your in-depth review! Please find below our responses to each of the weaknesses (W) and questions (Q) raised. All references refer to the updated manuscript, where we highlight relevant changes in blue.

W1: My main concern is novelty. Several highly related (i.e., tool-augmented AI feedback), published papers have not been cited and clearly discussed. "Novel framework" sounds overclaim.

Thank you for bringing this relevant work to our attention, we agree that this work is related to ours and have added a corresponding discussion to our related work section (Section 5). However, we also emphasize that there are notable differences between the referenced work and our framework. Below we provide a more detailed discussion of the referenced works and how they differ from our work:

Themis. The Themis framework introduced by Li et al. (2023) investigates the integration of tools into reward models. Similar to our work, this framework aims to give AI annotators access to a number of external validation tools to improve their annotation quality, including code interpreter and web search tools. The critical difference to our work is the modeling choice: the Themis approach by Li et. al (2023) combines generative language and conventional scalar reward outputs to create a tool-using reward model. Their setup requires a custom model with separate final layers for language- and reward-modeling. Such an approach requires customizing a model's architecture and fine-tuning a model. Our approach, on the other hand, following the general LLM-as-a-Judge direction, works with any generative language model without need for fine-tuning, including models only accessible via APIs. Our method not requiring fine-tuning has two main benefits: (a) easier and cheaper deployment compared to custom fine-tuned models, which are typically more expensive to train and deploy in terms of development time and compute cost; and (b) ability to use state-of-the-art closed-source LLMs that cannot be customized or fine-tuned as required for Themis. For example, OpenAI's GPT-4o or Anthropic's Claude-3.5-Sonnet could not be used directly with the Themis framework. We consider the ability to use state-of-the-art models critical when trying to annotate the kind of advanced tasks that our paper considers. Further, it is worth noting that anybody with the relevant API key(s) (model and search API) can use our framework with minimal development effort.

Beyond the general modeling differences, the code interpreter implementation in Themis also differs notably from ours: instead of working with any preferences pair containing code, Themis' code interpreter additionally requires the availability of a list of unit tests. This detail is shown in Figure 9 of the Themis paper and we confirmed this in their public implementation. Whilst such tests may be available for certain benchmark tasks, this is not generally the case and limits the applicability of this tool to new data. Our code interpreter tool enables the model to write such tests itself, and thus can be used with new data where such tests are unavailable.

CRITIC. The CRITIC framework by Guo et al. (2023) aims to use tool-based self-critiques to improve model outputs. Similar to our work, this framework considers the use of external validation tools to improve model outputs, including web-search and code-interpreter tools. However, CRITIC is intended to directly improve generated responses rather than providing general (pairwise) feedback that can be used for downstream training or evaluation. As such, it is more similar to general prompting techniques like Chain-of-Thought (CoT) [1], rather than LLM-as-a-Judge. Experimentally, the method is validated on general generation tasks such a free-form question answering, mathematical program synthesis, and toxicity reduction. The method is compared to general prompting technique baselines like CoT rather than more specific LLM-as-a-Judge approaches. In general, this work provides valuable information about integrating external validation tools as part of a prompting strategy but cannot be directly compared to our framework as it is intended for general response generation and refinement rather than for providing feedback in an LLM-as-a-Judge setting.

• [1] https://openreview.net/forum?id=_VjQlMeSB_J

Thank you for your constructive review, especially for highlighting potential issues with GSM8k! Please find below our responses to each of the weaknesses (W) and questions (Q) raised. All references refer to the updated manuscript, where we highlight relevant changes in blue.

W1: While this paper shows strong results on annotation accuracy, it is unclear how well this improves downstream performance. I don't think this is a hard requirement for this work, but I'd be interested to see how model performance changes using this method to either generate preference data, or do best-of-n ranking for model outputs. I do not think this is required for this paper to be accepted, however.

We agree that looking at effects on downstream performance would be a very interesting direction for future work! We consider the experiments included in our paper, that are based on the annotation of diverse set of preference datasets, as the most direct evaluation of our method's capabilities. Yet, testing downstream fine-tuned model performance would help obtain a more comprehensive understanding of the effects of our method. Unfortunately, given the limited time available, we were unable to conduct such a more resource-demanding experiment. We would be excited to see future work explore the downstream use-cases, both in terms of model fine-tuning and evaluation.

Q1: Did you check the correctness of the GSM8K hard answers? GSM8K has a small but noticeable subset ($<5\%$) that have incorrect labels, so without any validation, the instances that GPT4o gets "wrong" may be mislabeled. I'd recommend checking this, and if some are mislabeled, this may be the source of the mixed results you see on math reasoning. If so, I'd recommend thinking about harder math datasets (like MATH), though this may be more complicated for code execution.

Thank you for highlighting this potential issue! We had previously manually checked some of the datapoints, but following your suggestion we have done a thorough manual check of the validity of all GSM8k hard datapoints.

Process. We first compared our results to errors in GSM8k that were publicly reported [1][2]. We found two incorrect datapoints included, approx. $2\%$ of the dataset. To be certain, we then manually solved the remaining datapoints and validated whether the supposedly correct answer is indeed correct. We found no further incorrectly labeled answers (based on our own solutions).

Results. Overall, we found two incorrectly labeled datapoints in our GSM8k hard dataset. For both samples, we observe that our agent models consistently prefer the (actually correct) GPT-4o generated datapoints (rather than the incorrect golden perferences), whereas the baseline models only sometimes prefer the golden datapoints. This effect indeed, as you suggested, may slightly inflate the performance of baseline models, but by less than $2\%$. Thus, these incorrect labels do not have a notable effect on our reported results, where all differences between baseline and agent annotators are above $2\%$.

Further, we note that as part of this inspection we also found a small typo in our dataset description (L299): our GSM8k hard dataset consists of 116 rather than 117 datapoints. This error has been corrected in the updated manuscript and does not affect our results.

Overall, thank you again for helping us make these results more robust!

• [1] https://huggingface.co/datasets/Cleanlab/bad_data_gsm8k_svamp.csv/
• [2] https://github.com/openai/grade-school-math/issues

Q2: I'd be interested to see how this affects best-of-n ranking when using LLMs as a judge for ranking n model outputs -- I'd assume this would noticeably help performance on the domains tested. This may be expensive depending on the setup though, so this is also a reasonable follow up work instead.

See our response to W3. We further add that we agree that we would also expect the impact on downstream model performance to be primarily visible in similar task domains as covered in our current experiments. As before, we would be very excited to see this line of investigation in future work.

Thank you for your detailed review! Please find below our responses to each of the weaknesses (W) and questions (Q) raised. All references refer to the updated manuscript, where we highlight relevant changes in blue.

W1 (1): While the use of toolings for AI annotators is interesting, in the current iteration of the work, it is not very clear if it will scale with more custom toolings.

We intentionally designed our framework such that it is able to scale with additional and more custom toolings. In particular, the initial domain assessment step of our method makes it possible to restrict the domain of any new tool to a well-defined task domain where the developer is confident that the tool improves performance. Thus, to add a new tool, we recommend two steps: (1) running the annotator with the tool on a broad range of annotation tasks where the tool may help; and (2) writing questions for the initial domain assessment that reliably classify tasks where the tool is empirically shown to help. This way, the initial domain assessment enables adding custom tools whilst limiting negative impact on out-of-domain tasks. We have added a clarification to Section 3.1 highlighting this motivation.

W1 (2): In the agent evaluator discussed in the paper, eventhough it defaults to existing annotations for the no-tool use cases, the system shows a degradation in performance for RewardBench, the only OOD dataset evaluated. This makes me concerned about the generalizability of the system.

Thank you for raising this concern! We would like to clarify two aspects:

1. RewardBench is more than one dataset. Whilst accessible via a single download, RewardBench is an aggregate dataset combining a large set of existing datasets. In particular, the OOD subset we used consists of 16 separate sub-datasets covering a diverse set of domains [1].

2. RewardBench OOD is a challenging task. In the construction of this dataset, we explicitly omit all RewardBench subsets from categories where we would expect our method to potentially improve performance (math and coding tasks). We have clarified the construction in Section 4.4. As such, the dataset presents a very challenging task, where our tools and framework most likely cannot improve performance. Further, many of the remaining subsets of RewardBench are fairly saturated. On many datasets included in RewardBench our baseline annotators achieve $90\%+$ agreement. On such saturated datasets, the primary goal is preserve performance as well as possible --- rather than improve performance, as there are, relatively speaking, few datapoints to improve performance on. Overall, we note that this OOD test is likely a more challenging setting than in the real-world, where we would expect more of a mix of tasks that are similar to our own datasets, RewardBench datasets and other tasks.

That said, we agree the OOD performance is an important area for future improvements and we would be excited to see future work in this area!

• [1] Full list of included datasets: AlpacaEval Easy, AlpacaEval Length, AlpacaEval Hard, MT Bench Medium, MT Bench Hard, LLMBar Natural, LLMBar Adver. Neighbor, LLMBar Adver. GPTInst, LLMBar Adver. GPTOut, LLMBar Adver. Manual, Refusals Dangerous, Refusals Offensive, XSTest Should Refuse, XSTest Should Respond, Do Not Answer. See Table 1 of RewardBench paper (https://arxiv.org/abs/2403.13787)

W2: Two of the proposed benchmarks don't have baseline human annotation scores, making it hard to quantify the degree of hardness of the datasets.

We had limited capacity for human annotation collection within this project. Given this limited capacity, we decided that long-form fact-checking would be the most interesting to benchmark against human annotators, but agree that future work providing human annotations across all datasets would be insightful.