On Speeding Up Language Model Evaluation

We thank the reviewer for the constructive feedback and pointing out the effectiveness of the proposed method. We address individual points below.

(1) The paper seems to be recycled and revised from a longer sumbisison, and many parts (especially figures or tables) are with tiny fonts, which are difficult to read.

Thank you for the feedback and we apologize for the small font sizes. We have updated the PDF of our paper with larger fonts for Table 1, Table 2 and all figures. Additionally, we have added vertical grid lines for Figure 3 and 4 to make the figures more interpretable. Please let us know if there is anything we can do to further improve the presentation of our paper.

(2) Some experimental results are difficult to understand, e.g., Table 3.

Thank you for the great suggestion and we have updated Table 3 in the PDF (also presented below). Both Table 3 and Figure 6 are designed to empirically verify that the scoring matrices for the datasets exhibit a low-rank structure. We use the following theorem as the foundation for Table 3:

Suppose $A_r$ is any rank-$r$ matrix, then $\frac{\sigma_{r+1}}{\sigma_1} = \min_{A_r} \frac{||A - A_r||_2}{||A||_2}$ where $\sigma_i$ denotes the $i$-th largest singular value of $A$. The LHS represents the ratio of singular values, while the RHS quantifies the reconstruction error for the best rank-$r$ approximation of $A$. Intuitively, this ratio measures how much of the total information in $A$ is captured by its top $r$ singular values.

In the table below, we observe that (1) the absolute values in the first row are much smaller than 1, and (2) as $r$ increases from 2 to 3, the ratio $\frac{\sigma_{r+1}}{\sigma_1}$ decreases only slightly across all datasets. This suggests that the scoring matrices exhibit low-rank behavior (typically rank 1 or 2), as the majority of their information is captured by the top three singular values.

(3) Overall, I believe evaluation is quite important, and it often involves a number of influencing factors. What if there exists biases in the test datasets? How the comparison results are consistent with human evaluation, since automatic evaluation may not reflect the real capacities of LLMs?

Thank you for your insightful questions and for highlighting this important aspect of evaluation! We fully agree that biases may exist in automatic evaluation metrics, which apply to two of the six datasets we used. However, addressing these biases is orthogonal to the goal of this paper. Our primary focus is to identify the best-performing LLM or method as efficiently as possible, given any evaluation metric—whether automatic or human. Notably, our proposed algorithms are flexible and can accommodate human evaluation metrics, enabling bias reduction by simply switching from automatic to human evaluation without modifying the algorithms themselves.

Furthermore, we have demonstrated the effectiveness of our algorithms on four datasets from GSM8K and PIQA, where the automatic evaluation metrics rely on regular expressions to verify whether the predicted answers match the ground truth. On these datasets, the automatic evaluation aligns closely with human evaluation. Detailed descriptions of the datasets are provided in Appendix C.

(4) The related work part is weak, which needs more discussions on evaluation of LLMs.

Thank you for the great note. Due to space constraints, we have included an expanded and more detailed discussion on the evaluation of LLMs in Appendix A of our submission PDF. This section provides an overview of evaluation metrics and benchmarks of LLMs, which are relevant but orthogonal to our approach. We have also updated the main text in the related work section to reference Appendix A for a more in-depth discussion.

Thank you so much for your detailed review and suggestions for us to improve the paper. We respond to the individual questions below.

In section 3.2, low-rank factorization: “Intuitively, if the method-examples are vert correlated, there should exist….” while I do agree with the intuition, it would be nice to have a citation here. At least the citation from the appendix: “Chen et al., 2020; Cai et al., 2019”.

Thank you for the great suggestion and we have updated our paper PDF with added citations.

Even though the information is in the paper, it requires going back and forth to find it. For example, the figure captions are lacing information that is present elsewhere in the text, or not present at all. Some redundancy in the text for the sake of clarity is always welcome. I added suggestions to improve this in the Question section below.

Thank you for the valuable paper writing suggestions! We apologize for missing captions and legends for some figures (addressed individually below). Following your and Reviewer 1wFE’s suggestions, we have also incorporated a table of notations (Appendix E Table 4) to enhance clarity and easier referencing.

In the whole paper, only one baseline is described: uniformly sample and evaluate T examples, but three baselines are mentioned later on, and shown in the figures. What are the two other baselines? Can they be given some attention in the paper?

Thank you for raising this great question. In Section 4.3, we describe the three baselines: Row Mean Imputation, Filled Subset, and LRF with algorithmic descriptions provided in Appendix D. Row Mean Imputation and Filled Subset uniformly sample queries in two different ways, while LRF only uses low-rank factorization without any active selection.

We recognize that the original paragraph formatting and the use of italic text for baseline names have made these descriptions less noticeable. To improve readability, we have updated the text in our PDF to use bold formatting for baseline names.

In Table 2, the H1 value for each dataset is stated. But there is no explanation of what a higher or lower value means in the caption, or anywhere near where the table is cited. I had to refer to the Corollary 1 where it is mentioned, re-read to figure out what a higher or lower value means, to later find an explanation in section 4.4.

Figure 3 has the datasets ordered from highest H to lowest, and it is mentioned in 4.4 (2 pages forward) that they are ordered by hardness. There is no mention that they are ordered from hardest to easiest, and that higher H means harder and lower H means easier. It can be deducted from the whole text, but it is not immediately obvious.

Thank you for the suggestion. We have updated our paper PDF with a more detailed description of $H_1$ in Corollary 1, Table 2 and Figure 3. Specifically, we reference the original definition of $H_1$ and explicitly state a higher $H_1$ means a harder setting.

In Figure 3, there is no legend for the curve colors. In the caption of Figure 4 it is stated that the UCB-E and UBC-E-LRF are blue and red (at least for Figure 4), but there is no mention of the other curves anywhere.

Sorry for the confusion, we sincerely apologize for missing the legend for Figure 3. We have added the legend in our revised PDF. The reviewer is also correct that the curve colors for UCB-E and UCB-E-LRF are consistent between Figure 3 and 4.

In Table 2, the columns are ordered: “Dataset Name”, “Size m x n”, “Method Set”. The size is m x n, m stands for methods, n for data samples. I would either swap the “Dataset Name” and “Method Set” columns, or transform the “Size m x n” column to “Size n x m” to have a natural ordering of the columns and the order of the sizes.

Thank you for the suggestion and we have updated our Table 2 with a more natural ordering of the columns.

Thank you for your encouraging review and strong support! We also appreciate the reviewer for pointing out the proposed algorithms can be used for other use cases, and are not limited to LLMs. We respond to individual questions below.

I suggest some table/mapping to keep track of the different notions used and their meaning.

Thank you for the great suggestion! We have incorporated a table of notations (Appendix E Table 4) in our revised PDF to enhance clarity and easier referencing. Please let us know if there is anything we can do to further improve the presentation of our paper.

Line 186: by adaptive selecting --> by adaptively selecting

Proof reading is needed to fix typos.

Thank you for reading our paper carefully. We have updated the PDF with fixes to typos and grammatical errors.

What does a stand for in equation in line 190?

Sorry for the confusion. $a$ is a hyperparameter that controls the tradeoff between exploration and exploitation for UCB-E, where a larger $a$ value encourages more exploration. We had added a definition for $a$ where it is first referenced in the paper.

We thank the reviewer for pointing out omitted details and potential points of confusion. We address individual questions below.

Some descriptions in the paper are unclear. For instance, Figure 3, which presents key experimental results, lacks a legend, making it difficult to interpret.

Apologies for the oversight and the confusion caused by the missing legend for Figure 3. We have added the legend in our revised PDF.

Additionally, the paper does not clearly define the baseline methods used in the experiments.

Thank you for bringing this to our attention. In Section 4.3, we describe the three baselines: Row Mean Imputation, Filled Subset, and LRF. Their algorithmic details are provided in Appendix D. Row Mean Imputation and Filled Subset uniformly sample queries in two different ways, while LRF only uses low-rank factorization without any active selection.

We recognize that the original paragraph formatting and the use of italic text for baseline names have made these descriptions less noticeable. To improve readability, we have updated the text in our PDF to use bold formatting for baseline names.

Some results also lack in-depth discussion. For example, Figure 3 shows that UCB-E and UCB-E-LRF perform inconsistently across different datasets. The authors attribute this to varying dataset difficulty; however, when comparing dataset pairs like (GSM8K Models, PIQA Models) and (GSM8K Prompts, PIQA Prompts), the conclusions are contradictory. More detailed explanation and discussion from the authors are needed here.

Thank you for this excellent question, we have conducted additional analysis exploring this observation! While dataset difficulty, as indicated by $H_1$, is a significant factor in the performance differences between UCB-E and UCB-E-LRF, it is not the only one. We have identified that the singular value ratios of the datasets also play a critical role. Our analysis is grounded in the following theorem:

Suppose $A_r$ is any rank-$r$ matrix, then $\frac{\sigma_{r+1}}{\sigma_1} = \min_{A_r} \frac{||A - A_r||_2}{||A||_2}$ where $\sigma_i$ denotes the $i$-th largest singular value of $A$. The LHS represents the ratio of singular values, while the RHS quantifies the reconstruction error for the best rank-$r$ approximation of $A$. Below we present the singular value ratios for all datasets.

Intuitively, smaller singular value ratios indicate that UCB-E-LRF is better able to approximate the underlying scoring matrix, enhancing its effectiveness. In contrast, a higher $H_1$ (more difficult dataset) reflects more methods with smaller performance gaps relative to the best method, increasing the queries required for UCB-E, which does not leverage low-rank factorization.

For the dataset pairs mentioned, the apparent contradictions (e.g., GSM8K Models vs. PIQA Models, and GSM8K Prompts vs. PIQA Prompts) can be reconciled by considering both the relatively high $H_1$ and the low reconstruction error (as indicated by the singular value ratios) in the table above. Datasets with smaller singular value ratios allow UCB-E-LRF to perform better due to improved low-rank approximations. Conversely, datasets with higher $H_1$ demand more evaluations for UCB-E. We will incorporate this expanded analysis into the final version of the paper to provide a more comprehensive discussion.