From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set

Dear Reviewer, thank you very much for your careful and comprehensive review of our paper. We appreciate that you recognize our sound experimental setup and clear presentation, as well as convincing results and parameter studies. We would like to address your concerns about the practicality of the Specialist method.

I find the practical value of the proposed method is hindered by its inherent requirements and costs of having full access to the evaluation set of examples.

It is true that the Specialist method specializes to a fixed test set by design. By removing the constraint that the metric generalize to new test sets, we achieve large gains in the metric's ability to generalize to new systems, and this is often a tradeoff that is worth making. It is common to hillclimb for many years on a fixed test set (e.g., WMT for machine translation, XLSum for Summarization, SQuAD for question answering). Eventually these benchmarks saturate and must be refreshed, but benchmarks for core NLG tasks are often carefully curated to measure certain capabilities of interest and are repeatedly used to allow for fair comparison against previous work (including during the model development process). Thus, the Specialist method provides a paradigm for joint development of Automatic Metrics and Test Sets (which are typically developed independently, but don't need to be, and we get quality gains if they're developed jointly).

I think this would make it inappropriate to rate specialized models and applications where successful translations can be hard to obtain or costly (e.g., underrepresented languages, long passages).

We interpret this concern to relate to the cost of collecting outputs from multiple models on the given test set, for the sake of then collecting human annotations to construct Specialist ICL examples, but please correct if this is a misinterpretation. If "successful translations" are hard to obtain, this would apply equally to the collection of outputs for the creation of ICL examples as it would at test time. Thus, this begs the question of whether this task should be used for evaluation in the first place, or would be a bottleneck at every evaluation.

The above point aside, it is true that for some NLG tasks, there are not that many different models/systems which can perform the given task (and from which to collect ICL examples for the Specialist Autorater). However, as we show in the ICL example scaling experiments in Section 5.3, only 3 ICL examples (i.e., ratings of outputs from 3 other systems) are needed to outperform the state-of-the-art. For many NLG tasks, including those tracked most closely on LLM leaderboards, we do indeed have easy access to outputs from multiple systems.

Moreover, as discussed in the "Specialist Algorithm" paragraph in Section 3, while "the different translations for each input example come from different translation systems [in this work], they could in principle also be sampled from a single model (e.g., using a diversity promoting sampling algorithm)." This is a promising direction for future work, especially for tasks which are not well represented in available models.

The scalability is also a potential concern, as with more comprehensive and larger test sets, there will be linearly scaling costs.

This work explores how to use human annotations to enhance the quality of automatic metrics and, while the cost of collecting Specialist annotations for a test set does scale linearly with the size, evaluation using automatic metrics is much cheaper than directly depending on human annotations for evaluation, without sacrificing quality (as shown in the comparison against inter-annotator agreement in Section 5.5).

Moreover, with larger test sets, the imperative to depend on automatic metrics (rather than human evaluation) increases too. One-off collection of human ratings on a large(r) test set to build an automatic metric is much more tractable than repeated human evaluations on this test set. Our hope is that the results in our paper showing the effectiveness of the Specialist method will provide motivation to the broader NLG community to acquire human annotations and build Specialist metrics for other tasks and datasets.

We hope our responses and clarifications have allayed your concerns about our method's practicality, and that you will consider increasing our score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.

Dear Reviewer, thank you very much for your feedback, which has helped us to improve our paper. In response to your primary concern, namely that the Specialist method's "effectiveness on tasks with fundamentally different characteristics (e.g., open-ended generation, [...]) remains uncertain", we have added a new experiment showing the effectiveness of the Specialist method on the HANNA benchmark (https://arxiv.org/pdf/2208.11646), which evaluates story generation. For additional details of our experimental setup and results, please see our response to Reviewer inJi. As shown in the table below (which reports story-level Pearson and Spearman correlations), our Specialist method demonstrated substantial gains on HANNA, outperforming the shuffled baseline across both tasks and all meta-evaluation metrics.

We also appreciate that, despite your concerns, you appreciate the soundness of our experimental design and the rigorous ablations which "ensure thorough validation of the effectiveness of same-source ICL specialization". We would also like to address your concerns about our method's novelty and practicality:

• Novelty: Your review states that "adjusting ICL demonstrations to enhance or analyze the performance of specific tasks has already been extensively studied". This is a broad field of study in its own right, and no supporting citations were provided. The "Essential References Not Discussed" section is also empty. If you believe essential references are missing, we would like to include them, but cannot act on this incomplete comment. Moreover, our contribution is not limited to showing that our proposed method for constructing ICL examples enhances performance of LLM-as-a-Judge metrics. More broadly, our work proposes a novel technique for using human evaluation to improve automatic metrics (rather than using human evaluation as the de facto evaluation method itself) via test set specialization, and includes an extensive study of how rater behavior affects performance of LLM-as-a-Judge metrics prompted with human ratings. Research on how to use human annotations to enhance the quality of automatic metrics is an important research area in its own right. To the best of our knowledge, our Specialist method has never been proposed, and yields enormous gains over the existing SOTA in MT evaluation (54% and 119%, respectively, on WMT'23 and WMT'24 test sets). While we acknowledge the simplicity of the method, its dramatic effectiveness is evident from the results presented in the paper.
• Generalization to real-world use cases: We would like to address your concern that the WMT benchmark, despite being "standard and widely used", may not "adequately reflect real-world translation scenarios." The submissions to WMT (submitted by researchers from institutions across both academia and industry) include a diverse collection of systems of varying quality, including both LLM-based and non-LLM-based (e.g., small encoder-decoder) systems. As shown in Figure 2 in our paper (and discussed in Section 5.2), the Specialist metric's outperformance is consistent for translation systems across the quality spectrum, and cannot be explained by gains only for a certain quality tier. These results imply that the Specialist metric has better generalization to new systems, when removing the constraint that it must also generalize to new test sets.
• Cost: The review states that "evaluation latency, cost implications, and practicality [...] have not been discussed." Please note that these considerations are in fact discussed in Section 3 of the paper, in the paragraph entitled "Specialist Method in Practice". The review also states that "large-scale deployment of prompted LLM-based metrics can be computationally costly". Once deployed, our proposed Specialist metric does not have any additional computational overhead relative to standard LLM-based metrics. Moreover, large-scale deployment of LLM-based metrics has been widely adopted and, for many real-world applications, is more practical, efficient, and cost-effective than running human evaluations. In fact, "rapid model iteration" is virtually impossible without relying on automatic metrics.

We hope the additions we have made to our paper (especially the new results on the HANNA benchmark for story generation evaluation) allay your concerns about our method's generalizability to other NLG evaluation tasks, and that you will consider increasing our score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.

Dear Reviewer, thank you very much for your thoughtful and detailed review of our paper. We will address your concerns below.

Q1: Could you clarify how their method relates to, differs from, or improves upon BatchEval: Towards Human-like Text Evaluation? (same as Weakness 3)

BatchEval is only related to our work through our use of in-context learning (ICL), as both approaches involve exposing the model to predictions from either itself (BatchEval) or the ground truth (Specialist) on different examples. BatchEval exposes the model to its own predictions on a batch of unrelated other samples. In this work, we expose the model to ground-truth predictions on a batch of related (same-source) samples. The approaches are not in competition with each other: one could provide multiple same-source historical annotations as ICL examples (our work) while asking the evaluator model to evaluate a batch of different outputs from the same source in one or more stages (BatchEval). We will add this BatchEval reference to the Related Work section in the final version of our paper.

Weakness 1: The method's reliance on historical annotations might limit applicability in settings without existing annotation resources.

It is true that access to (an offline collection of) human annotations is inherent to our work, which proposes a novel technique for using human evaluation to improve automatic metrics (rather than using human evaluation as the de facto evaluation method itself) via test set specialization. This work contributes to research on how to use human annotations to enhance the quality of automatic metrics, which we believe is an important and under-explored field, and our hope is that the results in our paper showing the effectiveness of the Specialist method will provide motivation to the broader NLG community to acquire human annotations and build Specialist metrics for other tasks and datasets.

Q2: Could you validate the method across other widely-used NLG evaluation tasks such as Topical-Chat, FED, HANNA, or QAGS? (same as Weakness 1)

Thank you for pointing us to a suitable non-MT NLG evaluation dataset, which has enabled us to improve our paper by validating the generalization of our method. We have added a new experiment showing the effectiveness of the Specialist method on the HANNA benchmark (https://arxiv.org/pdf/2208.11646; one of your recommendations), which evaluates story generation. Some additional details of our experimental setup are below:

• We evaluated both with and without human stories included. The HANNA paper excluded human stories, as these were considered outliers, while some follow-up papers included them.
• The HANNA dataset includes 6 criteria: relevance, coherence, empathy, surprise, engagement, and complexity. Due to time constraints, we report results on the first criterion (Relevance) and on the average over all criteria (where the averaged score represents an overall indication of the story's quality).
• We follow the same experimental methodology as described in Section 4.4 of our paper, and construct Specialist ICL examples per each test example via hold-one-system-out prompting. This simulates the real-world use case of evaluating a new system.

As shown in the table below (which reports story-level Pearson and Spearman correlations), our Specialist method demonstrated substantial gains on HANNA, outperforming the shuffled baseline across both tasks and all meta-evaluation metrics. On the Relevance task (REL), the Specialist achieved a Pearson of 49.0 (excl. human stories), compared to 43.7 for the shuffled baseline, which already outperformed the winning metric (BARTScore; Pearson=42.6) reported in the HANNA paper. The gap in performance between the Specialist and shuffled baseline is even wider when evaluating on the average score, with Specialist Pearson of 46.3 (66.2 incl. human stories), compared to 36.7 (54.3 incl. human stories) for the shuffled baseline.

We believe this new experiment has strengthened our paper substantially by showing that the Specialist method generalizes to a long-form text generation evaluation task (with very different characteristics than machine translation evaluation).

We hope the additions we have made to our paper (especially the results on the HANNA benchmark and the comparison with the BatchEval method) allay your concerns and will be taken into account when assigning your final score. Please let us know if you have any further questions or if we can provide any additional clarifications to help finalize your assessment of our paper.

Dear Reviewer, thank you very much for your positive and thorough review of our paper. We will address your outstanding concerns below, starting with the question which you stated as most important.

Can the authors explain to me what they are trying to achieve in 5.4.1 and what the results mean?

Sections 5.4.1 and 5.4.2 both address the question of whether the Specialist model is simply copying errors in the ICL examples, but from two different angles, to provide further evidence for the claim that the success of the Specialist cannot be attributed to naive copying behavior. In Section 5.4.1, we compare copying behavior of prompted Autoraters using Specialist vs non-Specialist (and fewer Specialist) ICL examples, while in Section 5.4.2, we compare the Specialist prompted Autorater with a non-model-based baseline, which makes predictions by directly copying ICL example errors. While the gap in precision between the Specialist Autorater and the Parrot baseline in Section 5.4.2 shows that the Specialist metric is not copying all of the errors that it could from the ICL examples, this alone does not establish that the Specialist ICL examples are actually teaching the model which errors to abstain from predicting. This latter claim is substantiated in Section 5.4.1, where we contextualize the Specialist's copying behavior with respect to that of other prompted Autoraters, and show that providing more Specialist ICL examples results in a decrease in copying behavior with respect to these examples. We hope this has clarified the difference between Sections 5.4.1 and 5.4.2, but if anything remains unclear, please let us know.

Section 5.6: Why did you use an alternative prompt, instead of directly converting the annotated MQM errors into MQM scores?

In Table 5, we are actually directly comparing these two settings. That is, to compute Acc23 for the AutoMQM Specialist, we are indeed converting the predicted MQM errors into scores. As shown in the table, this outperforms the Score Prediction Specialist, which directly prompts the LLM to generate a float quality score on a scale from 0-100 (aligning with standard direct assessment prompting techniques traditionally used for LLM-as-a-Judge evaluation).

As for the meta-evaluation metrics used, and in particular your concern regarding leniency to error classification and severity, note that the character-level F1 does indeed penalize incorrect error severities. See Section 4.5 of the paper: "Partial credit of 0.5 is given if the predicted rating correctly marks a character as an error but predicts the incorrect severity." To address your concern about leniency towards error classification, we report the "binary error classification F1" meta-evaluation metric below, which computes the F1 of the rating-level binary error vs no-error decision. This complements Table 1 in the paper, and we see that the Specialist metric outperforms all others according to this evaluation too.

[...] this is only useful when one has existing human evaluation on a dataset and one is using "Specialist" to obtain accurate, explainable evaluations for future systems developed on this test set.

It is true that the Specialist method specializes to a fixed test set by design. By removing the constraint that the Autorater generalize to new test sets, we achieve large gains in the metric's ability to generalize to new systems, and this is often a tradeoff that is worth making. Many benchmarks (e.g., WMT for machine translation, XLSum for Summarization, SQuAD for question answering) are carefully curated and repeatedly used to allow for fair comparison against previous work (including during the model development process). Moreover, the Specialist method provides a paradigm for joint development of automatic metrics and test sets (which are typically developed independently, but don't need to be, and we get quality gains if they're developed jointly). Thus, when seeking to evaluate on "an arbitrary new dataset where no prior human evaluation is available", the metric development process which accompanies development of this new test set would involve collection of a small set of human annotations of system outputs on this test set.

[...] resemblance to the spirit of "test-time training" (https://arxiv.org/pdf/1909.13231)

We agree that Test-Time Training has some similarity in spirit with our work, in that we both seek to specialize a model to a particular test set example using data related to that example. While they use related examples from a simpler auxiliary task with automatic labels (image rotation), we use related examples from the same task with ground-truth labels. Additionally, our method does not involve parameter updates.

Thank you!