We thank the reviewer for finding this work solid and effective! Below are our responses to the concerns:

Concern 1

In Section 3.2, the authors use dot product to match the query with the original benchmark, but do not explain how to use new queries in the test process and whether the queries are rewritten adaptively to match the ground truth answers. Why the answers from the original benchmark can be considered as the answers for the new queries?

We thank the reviewer for the insightful comments! To clarify, after matching the web query with the original benchmark, the original query will be dropped and we only use the questions and answers of the matched benchmark samples. The benchmark mixture aims to map the real-world task distributions to the ground-truth-based benchmarks.

We specified this in line 47-48, Figure 4 (graphic illustration), and Section 3.2 (notation illustration). We will carefully improve the related specifications in the later versions.

Concern 2

In Figure 5, the description of the experimental setup is missing. It is not clear 0-shot or 5-shot is used in Figure 5, and how the inputs of "Mixed" and "Original" are formatted for the models.

We thank the reviewer for pointing that out! The setting for Figure 5 is the same as other experiments: we use 0-shot and official, unified model input formatting; the correlation numbers of the "original" benchmarks are the same as Figure 1 and 10, whose settings are specified in Section E. We will make it clearer in the next version.

Concern 3

It is better to demonstrate the "User query" and "Ground truth-benchmark" processes are effective for constructing benchmarks through the ablation studies.

We wish to highlight that we did the ablation study in Section 4.2–"MixEval outperforms both benchmark-level and uniform mixtures." This ablation study aims to illustrate the effectiveness of the core step: benchmark mixture of MixEval. We compare the MixEval benchmark mixture with two other mixture schemes: Benchmark-level Mixture (ablating the sample-level mixture of MixEval) and Uniform Mixture (ablating both the sample-level and benchmark-level mixture of MixEval). Benchmark-level Mixture samples data points uniformly from each benchmark of the benchmark pool, with the benchmark size proportional to its split size in MixEval. In other words, its benchmark size distribution is the same as MixEval, while it's uniform inside each benchmark. The Uniform Mixture simply uniformly samples an equal number of data points from all benchmarks of the benchmark pool. Both methods yield significantly lower Arena Elo correlations than MixEval mixture, illustrating the effectiveness of our method.

Beyond that, we report the quantitative results of the whole query detection pipeline below (on our devised benchmarks) to illustrate the importance of the looped training. Before being trained, a language model achieving high recall was chosen (Vicuna-33B), with 99.12% recall and 46.21% precision; The looped training (as described in Section 3.1) significantly improves the precision while maintaining the recall, illustrating the accuracy of the devised web query detection pipeline and the importance of the looped training.

Table 2: The breakdown metrics of the web query detection pipeline.

Besides the above two ablations, the whole Section 4.2 has demonstrated the effectiveness of MixEval based on experimental results from multiple perspectives, which we believe has adequately illustrated the effectiveness.

Concern 4

Why can queries from different batches of the same distribution mitigate the contamination problem, instead of sampling queries from different distributions?

We thank the reviewer for this insightful comment!

The main reason is that there is a significant difference between every two batches, as illustrated in the Table 2 of the submitted paper. Note that beyond batch web query update, we will also perform source web query update, which updates all the web user queries with the latest Common Crawl splits and it can be interpreted as updating queries from different distributions.

We analyzed the dynamism and contamination of MixEval in detail in the general response to all reviewers and AC (on the top of this page). Hope that will help solve this concern!

Concern 5

Why the performance of the Benchmark-level Mix in Figure 5 is lower than the average performance of the Mixed benchmarks?

Because benchmark-level mixture (Benchmark-level Mix) is supposed to be worse than the sample level mixture (Mixed). As we mentioned earlier, the "Benchmark-level Mix" is part of the ablation study, which samples data points uniformly from each benchmark of the benchmark pool, with the benchmark size proportional to its split size in MixEval. In other words, its benchmark size distribution is the same as MixEval, while it's uniform inside each benchmark. So such mixture scheme is supposed to perform worse than the "Mixed" benchmarks in Figure 5, which, as specified in line 238-239, performs the same sample-level mixture as MixEval. This performance gap arises from the fact that MixEval reconstructs real-world use cases more effectively than "Benchmark-level Mix".

We thank the reviewer for appreciating the comprehensive experiments, analysis, and insights of this work. Meanwhile, we understand the reviewer's concerns, which are also very important to us. Below we clarify:

Concern 1

Given that alignment with Arena Elo is used to measure the degree of alignment with human preferences, why not use data directly from Arena Elo when constructing MixEval, or even create a subset sampled from Arena Elo (considering Arena Elo directly reflects human preferences)?

We thank the reviewer for raising this concern. We think there may exist some misunderstandings here.

As a warm reminder, we illustrated the reasons of using web queries in Section A.2 (FAQs). Below we reorganize Section A.2 as a response to this concern:

First of all, it's worth noting that we are just taking Chatbot Arena as a measure of the capable model ranking of MixEval, while our approach and data have nothing to do with Chatbot Arena–our real-world user queries are solely detected from the web and the benchmark pool consists of the existing ground-truth-based benchmarks.

More importantly, our goal isn't to fit Chatbot Arena's distribution; we are trying to fit real-world task distributions instead. As shown in Section 2.2, line 127, Chatbot Arena queries are slightly biased towards technical users. However, web queries are grounded in the largest human population (5.4 billion internet users) among the accessible query sources and thus being more representative for the real-world users (illustrated Section 2.2).

Moreover, the web queries are multi-modal, meaning that the queries on the web do not only comprise the text-to-text user queries, they also feature other modalities, such as image-to-text, text-to-audio, etc. Such queries are not available on Chatbot Arena or other chat platforms if they do not support models with the corresponding inputs/outputs. Our detected queries of other modalities provide a proxy of building an upgraded MixEval with any-to-any modalities, which facilitates a better evaluation suite for the whole AI community. However, without web queries, it's impractical to get well-distributed any-to-any real-world queries within the community. We will release the multi-modal version soon.

Concern 2

To avoid models overfitting on existing benchmarks, the paper updates the benchmarks by drawing new queries from the internet. However, I believe overfitting occurs due to models overfitting on a fixed benchmark. Without changing the benchmark, the overfitting issue will not be resolved. Therefore, a method to determine if a benchmark is overfitted should be designed to add LLM benchmarks to the pool. Additionally, Table 2 does not demonstrate that this method alleviates overfitting.

We thank the reviewer for this insightful comment!

We analyzed the dynamism and contamination of MixEval in detail in the general response to all reviewers and AC (on the top of this page). Hope that will solve this concern!

As mentioned, although the contamination ratio of MixEval is comparatively low, we will indeed do benchmark contamination detection before putting it to the benchmark pool in the future versions of MixEval to further reduce the contamination. In addition, we will perform benchmark pool update to incorporate newly released ground-truth-based benchmarks to the benchmark pool, e.g., replacing MMLU with MMLU-pro, to mitigate contamination.

Concern 3

Queries drawn from the internet might have already been seen by LLMs, so the retrieved samples from existing benchmarks might be simpler. The authors could exclude such factors during query extraction.

Thanks for pointing that out! We will carefully add a pre-processing step accordingly in the future MixEval releases. And at the same time, we illustrate below that the described situation has negligible impact to the model evaluation.

This concern can be converted to "We should exclude the benchmark queries leaked to the web from the extracted web queries to avoid introducing bias in the later benchmark mixture stage" (This conversion requires some reasoning. Due to the character limitation, we do not expand here and would be happy to discuss later.) We argue that it will have negligible impact to the model evaluation. Below we do a rough estimation about the ratio of the leaked benchmark queries in our extracted web queries.

The benchmark pool has a size of 10^5-10^6. Suppose the web sentences has a size of 10^12-10^13 entries [1] and it has a query ratio of 6%, i.e., in 100 web sentences there are 6 valid web queries, then the web query quantity would be around 10^11-10^12. Thus, the ratio of benchmark entries in the web queries will only be around 10^-6. Containing such a low proportion of leaked benchmark queries in our web queries will introduce negligible bias to the later benchmark mixture stage, because the amount of web queries we finally sample for benchmark mixture is only around 10^3-10^4, meaning it’s highly possible that there won’t be any leaked benchmark query in the final sampled web queries.

Besides, in the previous illustration of contamination and dynamism (Table 1 of this rebuttal, in the general response), it is shown that MixEval is relatively less contaminated (~10%) compared with other popular benchmarks, which can also be interpreted as "MixEval samples are less likely to have already been seen by LLMs".

[1] Xue, Fuzhao, et al. "To repeat or not to repeat: Insights from scaling llm under token-crisis." Advances in Neural Information Processing Systems 36 (2024).

Concern 4

Some symbols are not explained, such as "j" in line 187, "tau" in the formula between lines 187 and 188, and "lambda" in line 189.

Thanks for pointing that out! These symbols are indeed under-specified, and we will carefully fix it in the next version. Here the j denotes the model's index; τ denotes the rejection threshold; λ denotes the scaling hyperparameter.

We thank the reviewer for recognizing MixEval as a timely work to the community! We are also grateful for your acknowledgment of the novelty, efficiency, thoroughness, and clarity of our work. Below are our responses to the concerns:

Concern 1 & 3

Pipeline Brittleness: a. Web User Query Detection. The web user query detection phase has an overabundance of negative examples, which does little to help distinguish positive examples. b. Benchmark Mixture. Additionally, the sentence transformer used in the benchmark mixture phase has limited performance. Have you tried other sentence embedding models? Also it's unclear how many ground-truth LLM benchmarks can be accurately matched to mined queries. c. The error rates for each module and the accumulated error rate are not provided. Could you provide detailed qualitative and quantitive analysis of module Web User Query Detection and Benchmark Mixture?

Overabundance of negative examples We controlled a balanced positive and negative samples for training by down-sampling the negative samples. The original specification may be a bit ambiguous, we will carefully fix this in the revised version later.

Weak retriever It's worth noting that in our retrieval task, both the query and key are short sentences, as the queries are real-world user prompts and the keys are benchmark questions, therefore it does not require state-of-the-art embedding models. We tested with both all-mpnet-base-v2 and openai's text-embedding-ada-002, they showed negligible difference in our task (only 6 cases are significantly different over 500 retrievals). Besides, as shown in Figure 2, MixEval data points exhibit a perfect mapping from the original web queries (the lowest C-Dist), further illustrating its effectiveness.

Quantitative analysis of web query detection We report the quantitative results of the whole detection pipeline below (on our devised benchmarks). Before being trained, a language model achieving high recall was chosen (Vicuna-33B), with 99.12% recall and 46.21% precision; The looped training (as described in Section 3.1) significantly improves the precision while maintaining the recall, illustrating the low error accumulation rate of the devised web query detection pipeline.

Table 2: The breakdown metrics of the web query detection pipeline.

Concern 2

Single-Language Focus: The all-mpnet-base-v2 model used in the framework is designed for English only, raising concerns about its adaptability to different linguistic and cultural contexts.

We thank the reviewer for pointing this out! This is indeed a good point. Though MixEval and MixEval-Hard are English-dominant (>95% English), keeping the whole pipeline as multi-lingual is beneficial. Note that MixEval is dynamic and can be updated with time. We will replace the current retriever with a capable multi-lingual one in the future MixEval releases to further improve the distribution.

Concern 4

It seems that the query of MixEval-Hard is longer than MixEval, is there any distribution difference between them?

Yes, they have some distribution differences, as they correspond to different difficulty levels. However, we have successfully controlled the distribution shift of MixEval-Hard with the rejection sampling mechanism introduced in Section 3.3. As shown in Figure 2, MixEval-Hard achieves a low C-Dist with the original web queries. The average length difference might arise from the fact that harder tasks tend to be longer in its average length, which aligns well with our commonsense.

We thank the reviewer for finding this work novel, effective, comprehensive, and solid! Below are our responses to the concerns:

Major Issue:

Concern 1

MixEval dynamically updates by mixing popular benchmarks (e.g., MMLU, BoolQ, GSM8K), which may not mitigate contamination. Most of these benchmarks are saturated and suffer from contamination. Although the authors claim they will "dynamically expand our benchmark pool with newly released benchmarks to further enhance the mixed benchmark distribution," this does not address the root issue of contamination.

We thank the reviewer for this insightful comment!

We analyzed the dynamism and contamination of MixEval in detail in the general response to all reviewers and AC (on the top of this page). Hope that will solve this concern!

Minor Issues:

Concern 2

While the topic distributions in Figure 2 are impressive, most of these benchmarks were not designed to simulate human preferences, hence the skewed topic distributions may be intentional for evaluating specific tasks, domains, or capabilities. Therefore, the section title "LLM Benchmarks are Biased from Realistic User Queries and Preferences" seems biased, implying these benchmarks are meant to measure human preferences.

Thanks for pointing that out, this is an interesting topic to discuss!

First of all, as indicated in the section title, what we care about are real-world use cases and user preferences, instead of solely the user preferences. "Real-world use cases" depicts the distribution of tasks, while "user preferences" depicts the grading process.

We believe the core principle of doing model evaluation is to evaluate them as they will be used in the real world. As a result, the evaluations should be designed based on this principle. (We also illustrated this in Section A.1 "Why are real-world human queries and preferences important?")

However, most of the existing evaluations are developed in a way that is based on the interest of its creators instead of the real-world users, e.g., GSM8K was created to measure models' mathematical reasoning abilities. Is mathematical reasoning something that we should measure? Yes, but we are not sure how important it is. Hence, Section 2 aims to measure the significance of existing evaluations by (1) compare their distributions with real-world use case distribution and (2) show their correlations with large-scale human preferences. "LLM Benchmarks are Biased from Realistic User Queries and Preferences" is the conclusion we got after rigorous analysis.

It is important to note that Section 2 did not aim to completely criticize the existing evaluations, because they are still a good measure for specific tasks, domains, or capabilities. Section 2 is only serving as a meta-evaluation to the community: (1) it tells people that the results obtained from these evaluations may not generalize to real-world use cases, as they show a deviated task distribution and limited correlation with real-world use cases, (2) it visualizes the task distributions of different evaluations to help people select the correct benchmarks for the abilities they want to measure–e.g., it shows that you shouldn't take WinoGrande or DROP as a general-purpose benchmark, as their tasks only focus on a very small range of topics.

Concern 3

Another potential issue is the assumption that ChatBot Arena serves as the groundtruth for calculating correlations. As noted by the authors in lines 127-133, ChatBot Arena itself is biased. Therefore, even though MixEval has a larger user base, it might overfit to ChatBot Arena to some extent, making it more of an alternative rather than a more accurate evaluation of user preferences.

We thank the reviewer for this insightful comment! As illustrated by the footnote of the first page, the Chatbot Arena leaderboard is indeed not the sole indicator of human preference, but it currently serves as the only gold standard large-scale human preference benchmark within the community. Therefore, we could not find a better proxy of real-world user preferences than Chatbot Arena to compared with. Because of this, we didn't claim that MixEval provides a better approximation of real-world human preference than Chatbot Arena (it might be better or not); instead, MixEval is providing an efficient and low-biased evaluation that can reflect real-world use cases.

Concern 4

I am a bit confused about Table 2. The authors briefly mention "periodically update" in the main text, but I did not fully understand the details of this update. Could you explain this further?

Sure! As illustrated in lines 51-53 and Section 3.4 of the submitted paper, we update the data points of MixEval via (1) batch web query update (sampling different web queries batches from the crawled web queries), (2) source web query update (updating all the web queries with the latest Common Crawl) or (3) benchmark pool update (incorporating new ground-truth-based benchmarks to the benchmark pool). Since the mechanism of MixEval is to match web queries with benchmark pool samples, the above three updating methods refreshes both the web queries (the first and the second method) and benchmark pool samples (the third method). We will specify it more clearly in the next version.

Dear Reviewers,

Thank you for your dedication and time spent reviewing our paper.

As the discussion period draws to a close, we wish to remind you of the approaching deadline.

Please feel free to raise any points or questions about the paper that may need additional clarification.

We deeply appreciate your valuable feedback.

Thank you once again.