Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models

Thank you for your review. We address your comments and questions below.

The generated data are not analyzed or explained in any depth. The paper does not tell the reader what kind of prompts were created, how well the generated references complete these tasks, or provide any statistic other than the number of examples generated per task (500). Aside from translation abilities, it is unclear what abilities this methodology really evaluates. [...] What are the tasks generated and what are their basic statistics? [...]

Thank you for pointing this out. In Tables 5 to 17 we detail the distribution of meta-prompt attribute values, providing some insight into dataset variety. Also, in line 81, we state that, with our meta-prompt template, it is possible to generate 9832320 unique data instances. We also provide the meta prompts used for data generation in Appendix A, given some insight on the type of data that will be generated, and what it is meant to evaluate.

To further address this issue, we have classified all our LLM general capabilities instances in terms of complexity, and placed them into 10 topic categories. We also do this with the ArenaHard data, to put our datasets in perspective with human-derived data. Please refer to our official comment for more details on this. We will include this analysis in the final version of the paper.

Furthermore, we will release our datasets upon publication, facilitating further analysis and scrutiny by the community. Please let us know if this alleviates your concern.

The methodology has several limiting requirements: 1) strong instruction-following model capable of generating fluent text, 2) a strong judge LLM capable of giving meaningful direct assessments, 3) tasks that can be assessed meaningfully via direct assessment. These requirements mean that for many languages and tasks, this methodology is likely to perform poorer than reported here. It would strengthen the paper to include experiments on an under-resourced language or structured prediction task to determine if this methodology is able to deal with adverse conditions.

Respectfully, we believe these requirements are not very limiting. More and more models that satisfy 1) and 2) are becoming widely available and cheaper to run (examples are the Gemma3 and Qwen3 family of models). Indeed, one of the main advantages of ZSB is precisely that it is model agnostic. As open models improve and become more accessible, so will ZSB benchmarks become better and cheaper to create. For example, since the time of writing, in less than 6 months, a wide variety of extremely capable open models have come out (e.g., Qwen3, Gemma-3), so it is reasonable to expect that requirements 1) and 2) will be even easier to fulfill as time goes by.

Requirement 3) is not a hard requirement of ZSB. The framework requires the creation of a judgment prompt, but this need not be DA. In fact, we show in Table 4 that different evaluation methodologies (e.g., pairwise comparison) can be used to achieve similar pairwise accuracy. We chose to frame the tasks in a DA fashion because it was a simple yet effective approach (see Table 4), but extending the framework to other evaluation settings and rubrics requires only adapting the judgment prompt. We will make this clearer in the final version.

Finally, we agree that it is relevant to evaluate ZSB on under-resourced languages. In fact, we included lower-resourced language pairs (LPs) in our machine translation evaluation suite: English-Icelandic and English Ukrainian. Interestingly, ZSB outperforms other MT metrics on both cases (1.0 vs 0.83 PA, and 0.83 vs 0.66, respectively). We realize these LP-specific results were not included in our submission, and we will include them in the final version.

Does the inclusion of the meta-prompt attributes actually lead to the creation of corresponding tasks examples? It seems that this aspect has not been evaluated. [...] the percentage of meta-prompt attributes that are correctly modeled in the task

This is a great question! We prompted Claude about whether each instance of all our LLM general capabilities benchmarks (across all languages) abided by the attributes passed in the meta prompt (i.e., binary classification). The results are the following:

The table shows the vast majority of generated instances model the meta-prompt attributes correctly on all languages.

We are happy to include this analysis for the rest of our benchmarks in the final version (as well as to release the prompts and the code used). Thank you for raising this.

Do the generated reference answers correctly complete the tasks? [...] the percentage of references that correctly complete the task Thank you for raising this. We ran Qwen2.5-72B-Instruct as a judge on top of Claude’s references for all our LLM General Capabilities benchmarks. In the following table, we show the percentage of times the reference receives the highest and the second highest scores:

While the vast majority of references are high-quality, few are perfect, according to this judge. This perhaps explains why the gains in PA of including a reference during evaluation are not very high (see Figure 2 b)). This is an interesting finding, which we will include in the final version. Thank you for raising this.

If I've understood correctly, you compare system rankings created by judge LLMs on your generated data to system rankings created via human/metric judgments on other datasets. Is this really a useful comparison? Shouldn't you instead compare system rankings created via human judgments on your generated data to perform a true metaevaluation?

Comparing system rankings created via human judgments on our generated data with our predicted system rankings would be evaluating solely the evaluation component of ZSB, rather than the entire framework. Note that ZSB yields an entire benchmark (i.e., test data + an evaluation method), so, as such, we must compare it directly with other benchmarks as a whole. Your suggestion is indeed valuable in the context of evaluating the evaluation method, but it would not take into consideration the test data generation component of ZSB.

Similarly to other prominent efforts in this area (e.g., Arena-Hard-Auto [1]), we pose that the most appropriate way of doing this evaluation is thus to choose a reliable gold ranking for a given task/set of capabilities (we describe our choices in Section 3.2), and to compare those rankings against system rankings obtained by different benchmarks. For example, M-Arena-Hard sources the test data from the Chatbot Arena and yields rankings according to a LLM-as-a-Judge evaluation via pairwise comparison (the final model scores are obtained via the Bradley & Terry model); ZSB, on the other hand, sources the test data from LLM-generated answers to a human metaprompt and yields rankings according to a LLM-as-a-Judge evaluation (we found a direct assessment rubric to be the most effective). As such, when we compare ZSB to Arena-Hard-Auto, we are indeed comparing the full “pipeline”, including test data sourcing and evaluation method. We hope this clarification answers your questions.

[1] Li, T., Chiang, W. L., Frick, E., Dunlap, L., Wu, T., Zhu, B., ... & Stoica, I. (2024). From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline. CoRR.

The paper you cite for Pairwise Accuracy (Thompson et al., 2024) criticizes many aspects of Pairwise Accuracy and proposes a better solution: Soft Pairwise Accuracy. Why don't you use this measure instead of PA?

Thank you for pointing this out. Soft PA improves upon PA by evaluating metrics not only on whether they rank systems accurately, but also on whether they do it with the same confidence as that suggested by the gold rankings. However, in order to do this, soft PA requires instance-level scores. These scores must be obtained from the same instances, otherwise they are not comparable. This is the case in WMT, where Thompson et al. (2024) propose the metric, since gold and predicted rankings are obtained from the same data, just different metrics (humans for the gold rankings, and automatic metrics for predicted rankings). With ZSB, we create our own instances, so the metric we use must be instance-agnostic, justifying choosing PA over soft PA. We hope this clarifies this matter.

Diversity: The authors have not reported how diverse the datasets are

Thank you for pointing this out. In Tables 5 to 17 we detail the distribution of meta-prompt attribute values, providing some insight into dataset variety. Also, in line 81, we state that, with our meta-prompt template, it is possible to generate 9,832,320 unique data instances. We also provide the meta prompts used for data generation in Appendix A, giving some insight on the type of data that will be generated, and what it is meant to evaluate.

To further address this issue, we have classified all our LLM general capabilities instances in terms of complexity, and placed them into 10 topic categories (as discussed in our first official comment at the top of the forum). We also do this with the ArenaHard data, to put our datasets in perspective with human-derived data. Please refer to our official comment for more details on this. We will include this analysis in the final version of the paper.

Furthermore, we will release our datasets and all prompts (meta-prompts and judge prompts) upon publication, facilitating further analysis and scrutiny by the community.

Authors mentioned this technique to be used as to complement the human curated dataset, to remove some bias. then this approach will not be a zero-shot generation process.

As mentioned in our Ethics statement, we believe all automatic evaluation should ultimately complement, rather than substitute, human evaluation. By design, ZSB is a zero-shot process. However, its flexibility is such that it is possible to integrate other signals during data creation, including human ones. We see this flexibility as an advantage of the framework, not a limitation.

The authors may explore the limits and dependencies of LLM judges- the framework relies fundamentally on LLMs acting as evaluators. While the paper demonstrates that open models can be strong judges, especially larger ones, a deeper discussion or analysis of the limitations of LLM judges (e.g., their potential blind spots, susceptibility to adversarial attacks on responses, or how task complexity affects their performance) would strengthen the paper.

As we mention in our Introduction and Related Work sections, LLM-as-a-judge is not a novel paradigm, but rather one that has been increasingly popular because of its effectiveness. As such, there is a considerable body of recent work on the limitations and biases of LLM Judges [4,5,6,7]. Thus, we respectfully point out that studying their limitations would likely overlap significantly with existing literature, and would not be central to our work, whose fundamental premise goes way beyond the step of evaluation. We show that ZSB can be used to create benchmarks that strongly correlate with human rankings by mostly using an LLM judge “out-of-the-box”.

We believe to have addressed all your main questions, please let us know if any further clarification is necessary. In case our answer alleviates your concerns, we kindly ask you to consider updating your recommendation score.

[4] Thakur, A. S., Choudhary, K., Ramayapally, V. S., Vaidyanathan, S., & Hupkes, D. (2024). Judging the judges: Evaluating alignment and vulnerabilities in llms-as-judges. arXiv preprint arXiv:2406.12624.

[5] Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z., ... & Liu, H. (2024). From generation to judgment: Opportunities and challenges of llm-as-a-judge. arXiv preprint arXiv:2411.16594.

[6] Szymanski, A., Ziems, N., Eicher-Miller, H. A., Li, T. J. J., Jiang, M., & Metoyer, R. A. (2025, March). Limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference on Intelligent User Interfaces (pp. 952-966).

[7] Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., ... & Lewis, P. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.

Thank you for your review. We address your comments and questions below.

Inheriting and exaggerating bias: The LLM may have biases on certain topics, those biases will likely be inherited and potentially exaggerated in the synthetic data. This can lead to models trained or evaluated on such benchmarks to perpetuate or even amplify those biases in real-world deployment

Overfitting to synthetic data: Models optimized or benchmarked against synthetic data might "overfit" to the specific characteristics, patterns, or even flaws of the generated data.

Address biases in synthetic data more comprehensively- the paper acknowledges that synthetic data may encode biases present in the models used to create it. While safety scores are included for transparency, they are explicitly stated as "not a complete solution". Strengthening the paper could involve exploring more rigorous methods for detecting and mitigating biases within the generated datasets, perhaps through explicit fairness metrics or analysis of how different generator models introduce different types of biases. We respond here similarly to our response to reviewer D9Dp.

We want to highlight that we are aware of these risks, and we openly discuss them in our Ethics statement. Furthermore, as we also discuss in our official comment above, we produce an additional column with the complexity of each instance generated, and we split the data into 10 topic categories to provide further insight on what exactly our datasets are evaluating. We also do this with the ArenaHard data, to put our datasets in perspective with human-derived data. We will include this analysis in the final version of the paper.

It should be noted that humans are not free of biases, and, as such, methods that rely exclusively on human-annotated data are not either. An excellent example are the state-of-the-art machine translation evaluation metrics we report in Table 2. They are trained on large amounts of human data, and yet there is a large body of work extensively studying their biases [1,2,3] (e.g., gender bias, self-preference bias). In our work, we provide safety scores to mitigate some of them, and, now, motivated by reviewers’ comments, we additionally obtain complexity scores, and categorize our data into more insightful buckets (see our official comment for details on this). Furthermore, at time of publication, the datasets will be made available for further exploration by the community, along with the metaprompts that were used to generate every single instance of the ZSB benchmarks.

A more complete solution to address these issues, including the exploration of methods for detecting and mitigating biases through explicit fairness metrics (as the reviewer suggests) is a very interesting direction for future work, though the breadth of the topic deserves a paper on its own—such an endeavour lies outside the scope of this paper. We see ZSB as a “live” benchmark which facilitates this kind of research.

[1] Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019, July). Evaluating Gender Bias in Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1679-1684).

[2] Zouhar, V., Chen, P., Lam, T. K., Moghe, N., & Haddow, B. (2024, November). Pitfalls and Outlooks in Using COMET. In Proceedings of the Ninth Conference on Machine Translation (pp. 1272-1288).

[3] Pombal, J., Guerreiro, N. M., Rei, R., & Martins, A. F. (2025). Adding chocolate to mint: Mitigating metric interference in machine translation. arXiv preprint arXiv:2503.08327.

Thank you for your review. We respond to your comments and questions below.

A significant concern is the reliance on a large-scale, proprietary model (Claude-Sonnet-3.5) for optimal performance in both data generation and judgment, as demonstrated in the primary experiments. While ablations suggest viability with open models, the observed performance degradation is considerable.

We show that using Qwen2.5-72B for both data generation and evaluation outperforms standard benchmarks (averaging), the most popular baseline considered, across all languages on the LLM General Capabilities task (this can be concluded from Tables 1, 3, 24, 25, and 27; see also our response to reviewer hRX1, and our official comment). The claim that observed performance degradation is considerable with open-models is unfair. ZSB produces benchmarks that are highly correlated with human judgments—higher even than the most widely used baselines—with an open model

Furthermore, and perhaps more importantly, as we elaborate in our response to reviewer hRX1 and in our second official comment, ZSB is a model-agnostic framework, so it does not rely on any particular type of model (proprietary, or otherwise). If anything, our experiments with open models show that it is possible to build strong benchmarks with open models (see L.228).

As open models improve, so will the performance of ZSB. Indeed, we ran Llama 4 Maverick—an open frontier model not available at time of writing—as a judge on our English LLM General Capabilities benchmark and obtained 0.844 PA. This outperforms the LLama-3.3 model we used, and is much closer to Claude. We are happy to include these experiments, and experiments with more frontier open models to better illustrate the point that ZSB is model agnostic and can improve over time.

The framework's complete reliance on LM-generated data and LM-based judgments introduces inherent risks. The quality of synthetic data—its fidelity to real-world complexities, potential inheritance of biases from the generator LM, and effective diversity—is paramount. While safety scores are provided, a more thorough analysis of the qualitative aspects of generated instances versus human-derived data would strengthen the paper's claims.

We want to highlight that we are aware of these risks, and we openly discuss them in our Ethics statement. Furthermore, as we also discuss in our official comment, we produce an additional column with the complexity of each instance generated, and we split the data into 10 topic categories to provide further insight on what exactly our datasets are evaluating. We also do this with the ArenaHard data, to put our datasets in perspective with human-derived data. We will include this analysis in the final version of the paper.

It should be noted that humans are not free of biases, and, as such, methods that rely exclusively on human-annotated data are not either. An excellent example are the state-of-the-art machine translation evaluation metrics we report in Table 2. They are trained on large amounts of human data, and yet there is a large body of work extensively studying their biases [1,2,3] (e.g., gender bias, self-preference bias). In our work, we provide safety scores to mitigate some of them, and, now, motivated by reviewers’ comments, we additionally obtain complexity scores, and categorize our data into more insightful buckets (see our official comment for details on this). Furthermore, at time of publication, the datasets will be made available for further exploration by the community, along with the metaprompts that were used to generate every single instance of the ZSB benchmarks. A more complete solution to address these issues, including the exploration of methods for detecting and mitigating biases through explicit fairness metrics is a very interesting direction for future work, though the breadth of the topic deserves a paper on its own—such an endeavour lies outside the scope of this paper. We see ZSB as a “live” benchmark which facilitates this kind of research.

[1] Stanovsky, G., Smith, N. A., & Zettlemoyer, L. (2019, July). Evaluating Gender Bias in Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 1679-1684).

[2] Zouhar, V., Chen, P., Lam, T. K., Moghe, N., & Haddow, B. (2024, November). Pitfalls and Outlooks in Using COMET. In Proceedings of the Ninth Conference on Machine Translation (pp. 1272-1288).

[3] Pombal, J., Guerreiro, N. M., Rei, R., & Martins, A. F. (2025). Adding chocolate to mint: Mitigating metric interference in machine translation. arXiv preprint arXiv:2503.08327.

I question the consistency and reliability of Direct Assessment (DA) using a 6-point Likert scale. The paper could benefit from a more detailed investigation into the sensitivity of DA results to rubric phrasing and judge model calibration. [...] Regarding the Direct Assessment methodology: How sensitive are the results to the specific phrasing and definitions within the 6-point Likert scale rubric? Was any form of judge model calibration undertaken to ensure consistent application of the scale?

Thank you for pointing this out. We used DA for all tasks considered, and performance was strong, which is, in itself, a testament to the consistency and reliability of DA. However, we agree that it is interesting to assess the sensitivity of the method to rubric phrasing. Thus, we obtain results for 2 additional rubric rephrasings of our judge prompt for the LLM general capabilities benchmarks. The results are the following:

The standard deviation of PA is very low in general (in French it is a bit higher), indicating DA is robust to rubric rephrasing. We will include these experiments in the final version of the paper.

In the VLM benchmark generation, the framework utilized pre-existing images paired with LM-generated text prompts, rather than fully synthetic image generation. While the authors acknowledge synthetic image creation as future work, this current implementation somewhat limits the "zero-shot" aspect for the visual modality.

We respectfully disagree that using existing images for the VLM benchmark is a practical limitation, due to the wide availability of images on the internet. The burden of collecting a small set of images is virtually zero, and can be viewed as part of the process of creating the data generation meta-prompt. Creating synthetic images, on the other hand, is potentially more burdensome, as it would require the creation of one additional prompt (the one for image generation). We also want to highlight that our framework is not tied to fully synthetic data generation—in fact, additional (including human-driven/-annotated/-collected) inputs such as external knowledge bases, benchmark examples, images, etc. can easily complement the data generation process (as illustrated in Figure 1).

The use of M-ArenaHard (queries derived from the Chatbot Arena, the source of the gold-standard human rankings) evaluated using the ZSB judging framework as an "upper bound" warrants careful interpretation. While intended to isolate the impact of test data provenance, the shared origin of M-ArenaHard queries with the gold standard inherently advantages this baseline, potentially making comparisons with ZSB less direct than implied.

We are aware of the inherent advantage of M-ArenaHard, and chose it precisely because it is an extremely challenging baseline. An upper bound is useful because it puts the performance of our method in perspective, and gives insight into how much we can still improve.

We respectfully disagree that M-ArenaHard and ZSB are not comparable. They are two benchmarks for general capabilities, just as ZSB is comparable with standard general capabilities benchmarks like, MMLU-Pro, or MUSR, even though they are very different in nature. These baselines are used for the same purpose: to rank systems automatically and measure progress. We are not comparing specific components of each method, but rather their effectiveness for automatic evaluation.

The claim of task-agnosticism may be challenged by the practicalities of prompt engineering. Developing effective meta-prompts for data generation and judgment prompts for evaluation, particularly for entirely novel or highly specialized tasks, could still represent a substantial engineering effort, potentially limiting true "zero-shot" applicability in all scenarios.

We understand your point. What we mean with “task-agnostic” is that ZSB is not tied to a particular task, but allows flexible incorporation of diverse tasks with minimal effort. Indeed, we evaluate ZSB on a suite of tasks covering different languages and modalities. In total, we cover 3 tasks, 4 languages, 11 language pairs, and 2 modalities. As discussed in Section 3.2, we choose these particular tasks because of their importance in LLM development, their diversity, and the availability of reliable gold rankings. It would be ideal to experiment with novel and highly specialized tasks, but it is hard to obtain gold rankings. We believe we made significant effort to support our claim of task-agnosticism, and encourage the reviewer to re-consider their assessment.

In any case, we respectfully disagree with the claim that prompt engineering effort limits the applicability of our framework. We state, when describing the framework in Section 2 and Figure 1, that the only human effort ZSB requires is the creation of two prompts: one for data generation and another for evaluation. Some tasks, regardless of their novelty and specialization, may require more complex prompts, depending on their specific requirements. This is expected within the framework. Moreover, as illustrated in Figure 1, ZSB is receptive to additional inputs that can further guide the data generation process and may be particularly helpful for novel or highly specialized tasks (e.g., human-annotated examples, knowledge bases, etc.).

The ablation study indicates that judge models may perform better on data generated by models from the same family. Does this suggest a potential "self-preference" or "model-family" bias, and what are the implications for benchmark fairness when evaluating a diverse range of models?

We believe there may have been a misunderstanding of this ablation, which we will make sure to clarify in the final version of the paper. ZSB has three stages where models come up: A) the data generation phase; B) the response phase; C) the evaluation phase, where responses from models used in stage B on the data created in stage A are evaluated.

There is no self-preference concern when using the same models for phases A and C, which is what we explore in our ablations. In fact, we find that there seems to be some synergy in using the same models or models of the same family for both data generation and judging.

Issues of benchmark fairness may arise when using the same models for stages B and C, for example. Indeed, this has been the focus of previous work that we mention in line 101 [4,5]; in fact, some solutions proposed in these works (e.g., using more than one model for judging) can be seamlessly integrated into ZSB. This topic, however, is not directly related to our ablations. Please let us know if this explanation clears up the matter.

[4] Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., ... & Lewis, P. (2024). Replacing judges with juries: Evaluating LLM generations with a panel of diverse models. arXiv preprint arXiv:2404.18796.

[5] Zhao, R., Zhang, W., Chia, Y. K., Xu, W., Zhao, D., & Bing, L. (2024). Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions. arXiv preprint arXiv:2405.20267.

Given the identified importance of dataset variety, what strategies beyond increasing meta-prompt attributes were considered or could be integrated into ZSB to enhance diversity and challenge? For instance, could external knowledge sources or adversarial generation techniques be incorporated?

As we point out in Figure 1, yes, external sources of data (e.g., pre-existing data, images) can be used to enhance ZSB. In this work, we chose to focus on extensively ablating components of the meta-prompt itself, because it is a central part of the framework, and where it is easiest to make adjustments. We believe it is appropriate to leave the study of other enhancements to future work.

We believe to have addressed all your main questions, please let us know if any further clarification is necessary. In case our answer alleviates your concerns, we kindly ask you to consider updating your recommendation score.

Thank you for your review. We will address your comments and questions below.

There's no clear generalizable guidance on which model to choose for zero-shot benchmarking for open weight models (Lines 238-239). The pairwise accuracy can large differ across different languages/datasets (Table 22)

Thanks for asking this question. In fact, one of the main advantages of our framework is that ZSB is agnostic to the model used, which allows for the quality of benchmarks to improve as existing models improve.

Nevertheless, to illustrate this point, we ran Llama-4-Maverick as a judge (an open frontier model not available at time of writing) on our English LLM General Capabilities dataset, and obtained 0.84 PA, which vastly outperforms the Llama-3.3 model we reported in the paper, and is competitive with Claude.

Thus, the guideline for choosing an open model is to experiment with the strongest models available at any given time (that fit compute restrictions), and choose the one whose generated data/evaluations fit best the requirements of the task of interest. Similar reasoning applies to closed models. We can make this more clear in the final version of the paper.

At the time of writing, we wanted to assess whether using existing open models would work well for both data generation and evaluation. Remarkably, when analyzing Tables 1, 3, 24, 25, and 27, despite the variance across languages you mention, using Qwen2.5-72B-Instruct for both ZSB steps (bottom right corner of Tables 3, 24, 25, and 27) outperforms standard benchmarks (averaging) across all languages. However, even if it did not, and perhaps most importantly, the merit of ZSB as a model-agnostic framework would not be compromised.

We are happy to run the rest of our benchmarks with Llama 4, and other recent open models, and include those experiments in the final version of the paper.

We also address this issue in our official comment at the top.

We are not entirely sure how your mention to Table 22 relates to this issue. The table shows the performance of individual standard benchmarks used as baselines (not our method). We will make this more clear in the final version.

The analysis on using open weight models for data generation is limited since the main model used to generate data is Claude Sonnet 3.5 [..] Figure 2: Is Claude Sonnet 3.5 the model used to generate the test instances here? If so, what would happen with the open-weight models? What would be the cost implication of it?

We understand this point, and, for the final version of the paper, we will produce a version of Figure 2 when using Qwen2.5-72B-Instruct as a data generator and judge—the conclusions are similar to those when using Claude. Thank you for raising this. As far as cost is concerned, it depends; if using an API like Together AI, the cost of using a model like Qwen2.5-72B is similar to Claude-3.5. If, on the other hand, one has compute to run such a model locally, the cost is virtually zero (apart from the cost of the compute itself).

After reading couple more lines, I realized that it's [1]. Can you add the citation here too?

Yes, thank you for pointing this out. We will add the citation in the final version of the paper.

Please let us know if any remaining clarification is needed.

I would like to thank the authors for the detailed reply.

In fact, one of the main advantages of our framework is that ZSB is agnostic to the model used, which allows for the quality of benchmarks to improve as existing models improve.

My main point is that 1) the claim needs to be backed up, and 2) claiming "model agnostic" needs to be careful and more granular. Lines 239-240 "choosing the right model is crucial for the quality of the benchmark—for example, Qwen is especially strong in Korean, while Llama lags behind." already has a sentence describing it so I would suggest rather than claiming to be "model agnostic", the pairwise accuracy is model dependent (which is obvious).

we ran Llama-4-Maverick as a judge I would not recommend using Llama-4 in the further experiments due to the issue highlighted in [1] and an example output in [2]. According to Section 3.2, "LLM General Capability" is defined as the preference on The Chatbot Arena (Zheng et al., 2023) dataset, and Llama-4 has the doubts to explicitly trained to be optimized for the Chatbot Arena outputs (e.g., more emojis). I believe this is worth noting as discussion or limitations somewhere in the submission.

[1] https://x.com/lmarena_ai/status/1909397817434816562

[2] https://huggingface.co/spaces/lmarena-ai/Llama-4-Maverick-03-26-Experimental_battles

Thank you for your additional feedback.

My main point is that 1) the claim needs to be backed up, and 2) claiming "model agnostic" needs to be careful and more granular. Lines 239-240 "choosing the right model is crucial for the quality of the benchmark—for example, Qwen is especially strong in Korean, while Llama lags behind." already has a sentence describing it so I would suggest rather than claiming to be "model agnostic", the pairwise accuracy is model dependent (which is obvious).

We agree that the quality of a given benchmark (as measured, for example, by pairwise accuracy) is model-dependent. Our point about model agnosticity is that ZSB, conceptually, does not depend on any particular kind of model (e.g., proprietary). This is true by the definition of the framework, but we also show that it is true in practice by experimenting with various models—strong results can be obtained with open models (e.g., ZSB with Qwen2.5-72B outperforms standard benchmark averaging). We will carefully revise how we introduce this claim in the paper, taking into account your feedback.

I would not recommend using Llama-4 in the further experiments due to the issue highlighted in [1] and an example output in [2]

Thank you for raising this issue. We are happy to run further experiments with other state-of-the art open LLMs, such as Qwen3 and Gemma-3. These additional experiments will also serve to reinforce what we mean by ZSB being model-agnostic. We are now too close to the end of the discussion period to obtain results in time, but there is time to obtain them for the camera-ready version. Regarding including some discussion about the shortcomings of the Arena and Llama 4 in the final version of the paper, we completely agree. This information had not come out at time of writing, but it is definitely relevant.

Please let us know if you have any further concerns.