From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline

We thank the reviewer for the constructive feedbacks. Below, we address your concerns and propose our revisions.

W1: We acknowledge that our pipeline heavily utilizes LLMs for prompt quality scoring and response evaluation. To address this concern and enhance credibility, we validated annotation quality through majority voting among state-of-the-art models, achieving an 85.6% agreement rate with GPT-4-Turbo on 200 prompts in the validation set.

After reading your review, we integrated human judgments by manually labeling a subset (50 out of 200) of the validation set, yielding an 84.3% agreement between GPT-4-Turbo and human labels, and a 94.9% agreement between majority votes from LLMs and human labels. While OpenReview rules currently prevent us from updating during the discussion period, we assure you this important validation step will be included in the final manuscript to enhance reproducibility and credibility (in Section 4.2).

W2: We agree that demonstrating broader generalization is important. Although our primary experiments focused on Chatbot Arena, we also evaluated the pipeline on the WildChat dataset (Wild-O-Matic), obtaining comparable improvements in benchmark quality. Both datasets are crowdsourced from real-world users, aligning with our goal of creating benchmarks reflective of genuine user interactions. By validating our method on two distinct real-world datasets, we demonstrate sufficient generalization of our approach.

We further highlight that benchmark validation is resource-intensive, as each evaluation of 20 models incurs significant costs (~$600 per validation cycle). Hence, we strategically focused our ablations on different LLM judges, quality annotation models, and datasets (Section 4).

W3:

• We used a fixed random seed (42) across all experiments, including bootstrapping, and will explicitly document this detail in the revised manuscript (Section 6.1).
• Confidence intervals were calculated using 100 bootstrap iterations and are 95% unless specified otherwise; this clarification will also appear in Section 6.1.
• We will document the minimum cluster size (8) and note that the number of resulting clusters varies based on datasets and min cluster size (Section 4.1).

Regarding ablations on dropping qualities, we clarify that our experiments (Section 4.3, Figure 3) effectively cover this aspect by demonstrating improved separation between strong and weak LLMs as the number of included qualities increases. As we drop qualities, we see less distinction between strong and weak LLMs (e.g. GPT-4 vs Llama-2-70B and Claude-Opus vs Claude-Sonnet). We will further emphasize this connection in the revision.

W4: Thank you for the suggestions. In our revised paper:

• Figure 1 will be referenced in related works.
• Elaboration on the selection of the quality criteria, and their specific contributions, will be detailed in Appendix C (p. 15).

W5: Spearman correlation only tells you whether two rankings share a similar order: if Model A is above Model B under one benchmark, does that same ordering hold under human judgments? However, it glosses over two crucial aspects:

• Confidence or “Separation” Among Models: Even if two benchmarks induce the same ordering of models, some may do so with very different levels of certainty. In practice, it matters whether “Model A beats Model B” is a large-margin result the benchmark can replicate consistently, or whether it’s a fragile result based on noise. Our “Separability” metric explicitly checks how often two models have non-overlapping confidence intervals in their benchmark performance. If a benchmark repeatedly yields overlapping intervals, you cannot reliably conclude that the top model is truly better.
• Magnitude of Differences (Beyond Rank): Our “Brier Score,” for example, rewards a benchmark not just for correctly ranking pairs of models, but also for assigning appropriate probabilities to those rankings. A pairwise victory by 90–10 is stronger evidence of a real performance gap than a 55–45, yet Spearman treats them identically so long as the order is preserved.

Even with 20 models, fine distinctions can be critical. Modern LLMs often cluster close to one another in quality, so you need to know if a newly finetuned is truly outperforming a similar rival or if the difference is effectively within the margin of error. If your benchmark and evaluation metrics don't capture this level of granularity and confidence, you may end up training or deploying a model you think is better but is actually equal (or even worse) when considering statistical noise.

We will expand on this distinction clearly in the revised manuscript.

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.

We sincerely thank the reviewer for the valuable and insightful feedback. We address your concerns as follows:

"The choice of chatbot arena as a reference for computing confidence agreement and separability is somewhat limiting. Does this method extend to more challenging tasks, such as Olympiad-level mathematics or SWEbench, where state-of-the-art models generally perform poorly?"

We would like to clarify that while confidence agreement relies on human preference rankings as a reference, computing separability does not require human preferences or chatbot arena as a reference. Separability effectively captures a benchmark's ability to distinguish among model performances, a critical aspect as state-of-the-art models increasingly demonstrate similar levels of performance. This metric provides meaningful differentiation crucial for model developers.

We believe our method indeed extends to more challenging tasks. Our approach successfully extracts difficult tasks from vast unfiltered datasets, such as "PyTorch Autoencoder Implementation" and "Computability and Automata Theory" (see page 18, Figure 6). Here, we show topic clusters specifically ranked for their complexity and desirable quality and the highest scored topic clusters are selected for benchmark curation, suggesting our method aligns well with the current demands for more challenging tasks. We will make sure to clarify this connection in our revision.

"Currently, this setup is limited to single-turn evaluation, which is restrictive, as many contemporary LLM applications involve agentic or multi-turn interactions."

We acknowledge that our current method is indeed limited to single-turn evaluations. Multi-turn benchmarks are inherently challenging to automatically curate from crowdsourced datasets because responses beyond the initial turn can depend significantly on prior model outputs. Extending Bench-O-Matic to robustly support multi-turn evaluations represents a valuable future direction we intend to pursue.

"When using the same model family for both judge and annotator, the judge model often exhibits bias towards generations from the same family. Have you investigated this issue, and could style control potentially mitigate this bias?"

The reviewer raises an important point. We have studied and quantified biases arising from using judge and annotator models from the same family (see Section 6.6: Mitigating Self-Biases in LLM-Based Evaluation). While we have not yet specifically explored whether controlling for style mitigates this bias, we definitely plan to investigate this.

"Given the growing emphasis on evaluating technical and challenging tasks (such as reasoning and coding), how well do you believe your method extends to these benchmarks?"

Our analyses indicate that the clusters selected by our approach indeed prioritize more technically demanding tasks, such as reasoning and coding, over trivial ones (as demonstrated in Figure 6 on page 18). For example, clusters involving sophisticated topics score higher, increasing their likelihood of inclusion in benchmarks compared to simpler ones like "Flirty Texting Strategies." Thus, we believe our method aligns strongly with the trend towards evaluating models on more technical and challenging tasks.

"Have you studied whether the increased separability results from filtering for difficult questions within benchmarks?"

We directly investigated the impact of filtering for difficult questions on separability in our experiments (see Section 4.3, Figure 3). The results confirm that selecting tasks with higher difficulty indeed enhances the benchmark’s ability to differentiate between stronger and weaker LLMs, reinforcing the benefit of our filtering strategy.

We deeply appreciate the reviewer’s feedback and hope our responses fully address your concerns.

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address each concern raised and propose revisions:

"However, one large concern is the fact that the experiments rely on the LLM-as-a-Judge evaluation framework, which, as noted in the paper, is known to exhibit certain biases. While the authors have attempted to mitigate some issues, it is still an inherent limitation which could influence the robustness of the results."

We agree that biases inherent to the LLM-as-a-Judge evaluation framework pose challenges. Nevertheless, it remains the most practical and cost-effective method to evaluate LLMs on diverse, open-ended real-user queries. Besides implementing multiple bias-mitigation strategies described in the paper, we highlight that our benchmark demonstrates greater robustness compared to similar LLM-as-a-Judge evaluations, as evidenced by the significantly higher agreement to human preference and stronger separability.

"Bench-O-Matic is currently rather limited to single-turn, English interactions. The authors acknowledge this limitation; however, it does raise questions about the generalization of this pipeline for various real-world applications."

We acknowledge that our current method is indeed limited to single-turn, English evaluations. Multi-turn benchmarks are inherently challenging to automatically curate from crowdsourced datasets because responses beyond the initial turn can depend significantly on prior model outputs. The authors are also primarily proficient in only English. However, extending Bench-O-Matic to robustly support multi-turn and multilingual evaluations represents a valuable future direction we intend to pursue.

"The reliance of LLMs as various parts of the pipeline may introduce bias which I believe has to be further investigated and regulated"

We recognize the reviewer’s concern regarding potential biases introduced by using LLMs throughout the evaluation pipeline. To address this, we validated the quality and reliability of our annotations using majority voting among several state-of-the-art models (GPT-4o, Claude-3-Opus, Gemini-1.5-Pro), achieving an 85.6% agreement with GPT-4-Turbo on 200 prompts in the validation set (as detailed in Section 4.2).

After reading your review, we additionally conducted human validations by manually annotating 50 prompts from our validation set. These showed an 84.3% agreement between GPT-4-Turbo and human annotations, and notably, a 94.9% agreement between human annotations and majority LLM votes. Due to OpenReview's policy restricting updates during the discussion phase, this important validation results will be fully detailed in the final manuscript revision to enhance Bench-O-Matic's reproducibility and credibility (will be detailed in Section 4.2).

"Prompts are filtered based on 'quality scores' produced from an LLM. How was this threshold determined, and do you know how adjusting this could potentially affect the quality?"

To clarify, we did not rely on LLM-generated numeric "quality scores." Instead, the LLM was instructed to provide binary judgments indicating whether each prompt met specific qualitative criteria (e.g., “Problem-Solving,” “Domain Knowledge”). The prompts and clusters with most of the criteria satisfied are selected for benchmark curation. We will clarify this distinction in Section 4.1 of our revision. Furthermore, our ablation analysis in Section 4.3 (Figure 3) demonstrates that including more qualitative criteria enhances the differentiation between stronger and weaker models (e.g., GPT-4 vs. Llama-2-70B, Claude-Opus vs. Claude-Sonnet), hence explains the enhanced separability of our final benchmarks (e.g. Eval-o-Matic, Wild-o-Matic) from other popular benchmarks. We will make sure to clarify this connection in our revisions.

We greatly appreciate the reviewer’s constructive feedback, which improves the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.

We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:

W1: We do not treat Chatbot Arena’s user-vote-based rankings as an unquestionable “gold standard.” Instead, we and other works (e.g., Dubois et al., Lin et al.) view it as a practical "silver standard" worth approximating. Although Chatbot Arena likely contains biases and noise, it currently represents the largest publicly available dataset of human evaluations, with over two million interactions. Hence, benchmarks demonstrating high agreement and correlation with Chatbot Arena rankings provide valuable guidance to model developers. We will include this in our revisions.

W2: Our critique of Spearman correlation does not depend on assuming unbiased LLM judges. Rather, confidence-based metrics provide clarity on two critical questions:

1. When benchmarks predict model performance, do these predictions align with actual user preferences?
2. Can benchmarks effectively distinguish between similar model checkpoints?

If LLM judges exhibit systematic biases, this would naturally result in lower confidence, agreement, and correlation metrics. Conversely, if our evaluation demonstrates high confidence, agreement, correlation, and separability, it affirms the benchmark's usefulness.

We acknowledge potential biases inherent to LLM-based evaluations and address them through multiple strategies:

• Style Control: As detailed in Section 6.5, our methods significantly reduce biases toward particular styles or longer responses, improving benchmark alignment with human preference rankings.
• Bias Diagnostics: We systematically compare judge rankings with human leaderboards across multiple scenarios, using metrics like confidence agreement to quantify biases and misalignments.

We agree that detailed exploration of systematic biases is important and plan to expand our analysis.

W3: Our primary focus was comparing Eval-O-Matic with benchmarks like MT-Bench and AlpacaEval, which similarly utilize LLM-based judges for open-ended tasks without ground-truth references (instead of LiveBench for example). While recognizing that LLM-as-a-Judge inherently introduces challenges, this method remains the most scalable and affordable approach to model developers for open-ended prompts.

W4: We agree with the reviewer regarding the presence of low-quality and unsafe prompts within crowdsourced datasets. In Section 4.1 (line 196), we explicitly describe our process for filtering prompts and retaining only high-quality clusters based on average quality scores. Figures 4 and 6 (Section 4.3) illustrate how clusters are differentiated qualitatively, with higher scores correlating with complexity and relevance, while lower scores correspond to trivial or ambiguous prompts. Further details and examples are provided in Appendix C.

Moreover, our ablation study (Section 4.3, Figure 3) demonstrates how qualitative criteria effectively differentiate strong and weak models, explaining Bench-O-Matic's produced benchmarks' enhanced separability compared to other benchmarks. We will clarify these points further in the revision.

W5: We agree with the importance of evaluating benchmarks in specialized domains (scientific, legal, programming). However, our core contribution specifically targets diverse, real-world, open-ended interactions. Benchmarks such as Chatbot Arena, MT-Bench, and AlpacaEval are widely used precisely because they reflect authentic user interactions. Hence, Eval-O-Matic remains valuable for assessing practical, downstream performance. Nevertheless, we recognize this limitation and will clearly outline it in our manuscript.

W6: We have validated Bench-O-Matic’s performance beyond Chatbot Arena by also evaluating on WildChat-1M. Ensured no Eval-O-Matic prompts overlap with publicly released Chatbot Arena data. If a model trained on similar queries generally improves at addressing real-world interactions, we anticipate performance gains across both Eval-O-Matic and actual deployment settings.

W7: While our current framework focuses exclusively on text-based evaluations, it could be adapted to multimodal benchmarks by substituting LLM annotators and evaluators with VLMs. Indeed, Chou et al. have successfully applied a similar approach to the Vision Arena dataset, demonstrating the feasibility of extending Bench-O-Matic’s principles to image-based tasks.

Chou et al. "VisionArena: 230K Real World User-VLM Conversations with Preference Labels."

Lin et al. "Wildbench: Benchmarking llms with challenging tasks from real users in the wild."

We greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.