Dear reviewer k5n7.

We appreciate your insightful comments. We are encouraged that you found our experiments extensive and robust, our quantitative analysis thorough, and the manuscript to provide a rigorous, timely reality check. Below, we address your specific concerns and suggestions.

Q1: Ranking preservation. We appreciate your valuable suggestion. We follow your recommendation and calculate Kendall's tau between the real ranking and the predicted ranking of target models. The results are averaged across 100 trials for each benchmark. We further average the results across all benchmarks, and the results are shown below. Our conclusions mostly remain unchanged, with all benchmark prediction methods outperforming Random Sampling under interpolation, while none can surpass Random Sampling under extrapolation. We will include the detailed results in the camera-ready version.

Q2: Analysis when the number of source models is small / when the coreset is small. We appreciate your suggestion. We conduct an experiment on the HELM-GSM8K benchmark under the interpolation setting. Following your recommendation, we use only 25% of all models as source models, resulting in 21 source models and 62 target models in each random trial. The results are presented below. Interestingly, most benchmark methods perform worse than Random-Sampling. The exceptions include AIPW and Double-Optimize.

We also conduct another experiment in which the only change to Figure 2 is reducing the coreset size from 50 to 10. The results are presented below. We demonstrate that when the coreset is small, Random-Search-Learn remains the best-performing method, while the IRT method degrades to perform worse than Random-Sampling.

These results further highlight the potential limitations of existing benchmark prediction methods. We appreciate your insightful suggestion and will include it in the camera-ready version.

Q3: The variance of each method. We report the standard errors in Figures 6-11 in the Appendix. We thank you for the question and will be more explicit about this in the revised version.

Thank you again for your valuable feedback. We hope these clarifications address your concerns, and we look forward to further discussion.

Dear Reviewer 39bg.

We appreciate your insightful comments. We appreciate your recognition of the thoroughness of our experimental results, and we address your concerns and suggestions below.

Q1: Comparison to Bandit Algorithms. We thank you for pointing out the related literature [1] [2] on bandit algorithms. We agree that a comparison is important for clarity. However, we would like to clarify a key distinction between the research objectives.

Benchmark prediction aims to select a small subset of evaluation points to predict overall benchmark performance for any new model. This is useful when one wishes to obtain an accurate estimate for all models, not just the best one. In contrast, the bandit literature assumes a pool of target models and focuses on identifying the best model, rather than accurately estimating all models’ performance. By design, bandit algorithms will focus almost entirely on the best few models, which leads to bad performance estimation for all other models.

This paper focuses on the benchmark prediction task and explores when and how benchmark prediction is effective. All three other reviewers acknowledge the importance of addressing the problem. In the benchmark prediction literature, the state-of-the-art method, as far as we know, is the GP-IRT method, which we examined in our experiment.

To further address your suggestion, we experiment with the code from [2] out of the box within the benchmark prediction context. Specifically, we apply both the UCBE method and the UCBE-LRF method to the HELM-GSM8K benchmark under the interpolation model split. The results are shown below. Both bandit methods yield significantly larger estimation gaps compared to methods tailored for benchmark prediction. This supports our claim that while bandit algorithms excel at identifying the best model, they are less effective for accurate performance estimation across all models.

We appreciate your valuable suggestion and will include the citations and discuss the differences in the related work.

Q2: List of Models Used. We appreciate your suggestion and will include the model list and characteristics in the revised version.

Thank you again for your valuable feedback. We hope these clarifications address your concerns, and we look forward to further discussion.

[1] Shi, Chengshuai, et al. "Best arm identification for prompt learning under a limited budget." arXiv preprint arXiv:2402.09723 (2024).

[2] Zhou, Jin Peng, et al. "On Speeding Up Language Model Evaluation." arXiv preprint arXiv:2407.06172 (2024).

Dear Reviewer V6AP,

Thank you for your detailed and thoughtful review. We are encouraged that you found the problem we addressed important, the empirical analysis thorough, and the paper overall strong. Below, we will address your specific concerns and suggestions.

Q1: Citing Prior Relevant Work. Thank you for highlighting the missing citations. Among them, MetaBench closely resembles the P-IRT method we discussed in our paper. There are two main differences: First, MetaBench employs lower-dimensional IRT models and selects the coreset based on information filtering. Second, metabench jointly fits its models on multiple benchmarks, making a fair comparison to other methods difficult. Prabhu et al. propose a method called Sort & Search for efficient lifelong model evaluation. Rather than predicting benchmark performance, their method focuses on predicting the performance on individual data points. We further address our similarities and differences to this work in Q2 and Q3.

We will include the appropriate citations and discussions for both papers in the revised manuscript, and we appreciate your suggestion.

Q2: Similarity of Key Takeaways to Prior Work. We appreciate your valuable question. Prabhu et al. select the coreset by first sorting all data points by difficulty and then uniformly sampling from various difficulty levels. They assert that this method is more effective than random sampling for sample-wise performance estimation, as suggested in their Figure 4(d).

However, in Figure 5, they demonstrate that the difference between sampling methods is smaller than the so-called “aleatoric sampling error,” which arises from their single rank order design, as defined in their equation 1. Therefore, they conclude that generalizing beyond a single rank ordering, rather than using better sampling strategies, is more effective in reducing overall error. However, this conclusion only applies to the framework defined by their equation 1 and does not generalize to other benchmark prediction methods (which do not use a “single rank ordering”).

In contrast, we have conducted an extensive empirical study on a set of benchmark prediction methods and found that coreset selection plays a minor role compared to prediction across methods. We appreciate your suggestion and will acknowledge Prabhu et al. for this relatedness in our revised version.

Q3: Validity of Aggregate Accuracy Metric. We appreciate your insightful question. The goal of benchmarking is to obtain an aggregated score, such as accuracy, from the testing data. According to Hoeffding's inequality, this score estimates the model's average performance on the data distribution and generalizes to unseen IID data. Therefore, the aggregated score serves as an effective indicator for comparing models and estimating performance. Benchmark prediction aims to estimate benchmark performance using a few data points to reduce computational burden, so it’s valid to use the aggregated metric.

Theorem 2.1 in Prabhu et al. essentially states that it is possible to predict aggregate performance without correctly predicting performance for individual samples. This is indeed true (most of the methods tested in our work do not even attempt to make performance predictions for individual samples). In our view, Section 2.1 seems to argue that using sample-wise prediction metrics is preferable, if you care about sample-wise prediction quality rather than directly motivating the focus on sample-wise quality. While sample-wise quality metrics can be useful, machine learning for better or worse mostly relies on aggregate benchmark scores, further motivating our focus on aggregate metrics.

We appreciate the comment and will add this discussion to the camera-ready version.

Q4: Performance on "Near-Extrapolation" Settings. We appreciate your insightful suggestion. In the rolling bin setting, without modifying the methods, we must evaluate all models in the previous bin using the full dataset for the next stage to proceed. However, the main purpose of benchmark prediction is to begin with the already evaluated models, rather than assessing new models on the entire dataset.

Therefore, we conduct an experiment on the HELM-GSM8K benchmark under the single-bin near-extrapolation setting. Different from the paper setup (lowest 50% as source models and top 30% as target models), we now use the lowest-performing 75% of models as source models and the remaining 25% as target models. The results are shown below. Consistent with the findings in the paper, most methods fail to outperform Random Sampling, except for AIPW. We appreciate the valuable question and will include these results in the revised version.

Thank you again for your valuable feedback and for highlighting these important points. We hope these clarifications and additions address your concerns, and we look forward to further discussion.

Dear reviewer iudt,

Thank you for your thoughtful and constructive review. We are encouraged that you found our core findings compelling, experiments convincing, and insights useful. We are glad that you recognized our potential impact on changing how people think about benchmark prediction. Below, we address your specific questions and suggestions.

Q1: Value of IRT methods. Thank you for highlighting the broader utility of IRT-based methods beyond efficient benchmarking. We agree that these methods can provide valuable insights, and we will acknowledge these additional motivations in the camera-ready version.

Q2: Presentation of Figures 2 and 3. Thank you for your suggestions regarding figure clarity. Our primary focus is to compare different benchmark prediction methods in both interpolation and extrapolation settings. We will clarify this in the camera-ready version and add explicit takeaways to the captions when we have more space.

We did not intend to compare the estimation difficulty of different benchmarks in these two figures, as this difficulty is largely determined by the score distribution. Consider the Random-Sampling method; the estimation difficulty for any target model is bounded by its standard deviation and the coreset size, as stated in line 271 of the paper. Consequently, the estimation difficulty for any benchmark largely depends on its model score distribution, since the standard deviation of Bernoulli is $\sqrt{p(1-p)}$. If most models have very high or very low accuracy, the benchmark is easier to estimate; conversely, if accuracies are clustered around 50 %, the benchmark is harder to estimate. We appreciate the suggestion and will incorporate this discussion into the revised version of the paper.

Q3: Writing inaccuracy in paragraph 1 of the related work. We appreciate the comment and will revise this in the camera-ready.

Q4: Source model set not the same for different benchmarks. We mainly collect model scores from existing benchmarks, so it’s hard to use the same model list for all benchmarks. Furthermore, our work extends beyond LLM benchmarks; we also include ImageNet. We appreciate your suggestion and will include the model list and characteristics in the revised version. Please refer to the discussion about the comparison of benchmark estimation difficulties in response to Q2.

Q5: How is the information from 100 trials used? We report the average estimation gap over all target models in 100 trials, and the standard error is reported in Figures 6-11 in the Appendix. We thank you for the question and will be more explicit about this in the revised version.

Q6: Coresets selection similarity. We agree this is an interesting direction. However, we find it challenging to quantify the coreset similarity using a specific metric. This difficulty arises because we conducted 100 different random trials for each method, and each trial often involved a different selected coreset. For instance, on ImageNet, we examined the coreset selected by AnchorPoint under random seeds 0 and 1, and we found that only three data points overlapped out of 50.

Q7: Intuition of Notation. We appreciate your feedback on notation clarity, and we will provide more intuition in the revised version.

Q8: Placement of ablation results. We will move the results to the main body in the camera-ready version. Thanks for the suggestion.

Q9: Coreset size being a static number instead of a ratio. We appreciate your insightful question. In our experiments, the dataset sizes of the benchmarks vary significantly, ranging from 277 for RTE to 50,000 for ImageNet. Consequently, it is challenging to find a ratio that applies universally to all benchmarks. Additionally, as mentioned in the response to Q2, the difficulty of estimating the benchmark is partly influenced by the coreset size rather than the ratio. We believe it is more effective to compare benchmark estimation difficulties by keeping the coreset size consistent across all benchmarks, while this is beyond our primary focus.

Thank you again for your valuable feedback and suggestions. We hope these clarifications and planned revisions address your concerns, and we look forward to further discussion.