AutoEval Done Right: Using Synthetic Data for Model Evaluation

We thank you for your thoughtful comments and positive feedback on the relevance of our work, the strength of our benchmark, and of our theoretical analyses. Below are detailed answers to your comments.

Understanding when AutoEval > classic.

Our approach adjusts to the quality of synthetic labels. In particular, AutoEval can rely on power tuning to identify the optimal λ used in Equation 4 or 6. This mechanism ensures that our estimates (i) have small asymptotic variance (ii) ignore synthetic labels when these are unreliable. Thus, our method is at least as good as the classical approach and typically better with informative synthetic labels.

We also cover several scenarios in our work where multiple synthetic annotation schemes could be competing. Figures 3 and 5 show that the correlation between synthetic and true labels is a good indicator of AutoEval performance and recommend using this metric to prioritize annotation schemes.

In the context of LLMs, we indeed note overall small correlations between human and LLM judge preferences, likely due to noise in human preferences and the ambiguity of many real-world prompts (multiple valid responses may exist). For instance, agreement rates of 85-90% among experts and 75-80% between experts and crowds have been reported in the past (see Chiang et al. 2024). These small correlations also highlight the limitations of LLM judges, biased by answer length, position, and stylistic similarities (Zheng et al. 2023). Mitigating bias via careful prompting, randomization of answer order (Li et al. 2024), and stronger judges (e.g., via fine-tuning on human preferences) could improve effective sample sizes of AutoEval.

You also highlight the relevance of understanding which LLM judges produce high-quality synthetic labels and on what inputs. Future work could apply AutoEval to specialized areas for LLMs where validation data is too scarce to provide insights into underrepresented tasks, providing better insight into what constitutes good LLM judges across domains.

Covariate shifts

Covariate shifts are central to key applications of AutoEval. A common scenario arises when unlabeled inputs are drawn from a distribution different from the target distribution. Labeled inputs could themselves not be sampled from the distribution of interest. These effects are relevant to address fairness concerns. Applied to LLM evaluation, for instance, covariate shift toward specific prompts can lead to biased evaluations that are not representative of target use cases.

We will add a supplementary note on AutoEval estimators under covariate shifts. In the canonical case, when unlabeled and labeled inputs respectively drawn from distributions P and Q, we can apply AutoEval using traditional covariate shift adjustment methods. Key assumptions here are that P is absolutely continuous with respect to (w.r.t.) Q and that the Radon-Nikodym derivative of P w.r.t. Q is known.

Novelty

Our work is more than a repackaging of PPI for model evaluation. We outline specific theoretical and methodological novelties in our response to Reviewer 3tuY and refer you to that discussion for details.

Runtime

We compared runtimes between AutoEval and classical in the ImageNet experiment:

Table: Runtime comparison between AutoEval and the classical approach for the evaluation of Resnet101 in the ImageNet experiment ($n=1,000, N=50,000$). This experiment was run on a workstation with an Nvidia RTX 3090 GPU, 128GB RAM, and an i9-12900KF CPU.

The main bottleneck here is synthetic label generating, typically scaling linearly with the number of samples. This cost is generally acceptable given the gains in evaluation efficiency and statistical power. We will include this analysis in the revised Supplement.

Other

Coverage guarantees are not always obtained [...]

Undercoverage in Figure S1b is due to insufficient Monte Carlo (MC) trials. Rerunning the experiment with $K=500$ MC trials yields:

Table: Coverage for $\alpha=0.1$ in the protein fitness experiment

The coverage is now much closer to nominal levels and validates our coverage claims.

It would be useful to augment the related work [...] on LLM judges [...] synthetic data for model evaluations.

We thank you for the suggestion. Since LLM judges are core to our work, we will update the related work in the revision with the suggested references.

I recommend moving figure S1 to the main paper [...]

We thank you for this comment and will move S1 to the main paper in the revised paper.

Thank you for your kind and insightful comments. We are grateful for your appreciation of our work, particularly our methodologically sound approach and practical demonstration of efficient model evaluation across multiple domains.

Regarding your comment on the relevance of AutoEval to address benchmark saturation, we would like to highlight a central application that motivates our work. One major inspiration for this work is the chatbot arena project (Chiang et al. 2024). This project serves as a dynamic alternative to traditional static benchmarks, allowing human participants to compare and vote on large language model (LLM) responses..

Despite the availability of large amounts of human validation data, resolving ties between models remains challenging in this project. First, many users prompt models with questions but decide not to provide preferences between models. Consequently, a large number of conversations without human preferences is available, which cannot be leveraged by traditional model evaluation approaches.

There are also many cases where there is not enough validation data to resolve ties between LLMs. This can happen when a new LLM releases, requiring large amounts of votes (which may correspond to several days of traffic) to quantify its relative performance. Unresolved ties also occur when evaluating model performance on specific applications where relevant conversations are limited, such as coding assistance or creative writing, for instance. Helping resolve ties timely in dynamic benchmarks such as Chatbot Arena with AutoEval is one way we believe our work can help address benchmark saturation.

We agree with your observation regarding the i.i.d. assumptions in our work. In the chatbot arena, for instance, these assumptions are challenged when users ask multiple questions or repeat the same question, leading to data dependencies. To address this, future work should extend AutoEval to more general settings with data dependencies to produce reliable inferences.

Finally, AutoEval does not solve the problem of designing hard inputs. It builds on the assumption that inputs are easy to sample but harder to label. While this assumption is limiting, the applications of our work—including in NLP—show that this paradigm can be useful. Beyond NLP, labeling at scale remains a significant challenge in many domains, such as biomedical applications, where we believe AutoEval can make an impact in facilitating model evaluation.

We thank you for your detailed and thorough feedback. We are grateful for your acknowledgement of the importance of the problem we are tackling, on the clarity of our claims and the methodological soundness of our approach. Your review raises a number of points relating to the theoretical contributions and to the experimental design of our work. Please find below our detailed response to these points, that we hope will address your concerns.

Lack of theoretical contributions.

We would like to address your concerns about the theoretical contributions of our work. While our approach indeed builds upon the prediction-powered inference (PPI) framework, it contains several novel theoretical and modeling aspects, relevant both in the context of auto-evaluation and PPI.

The first contribution is to build a general framework for model evaluation using synthetic labels. For metric-based evaluation, our framework accommodates various and nontrivial synthetic annotation schemes (e.g., external model annotation, or self-annotation, see Section 2.1), and allows both for hard or soft synthetic annotations. We also extend this framework to comparison-based evaluation, enabling rigorous model performance inference from pairwise comparisons. Our work also introduces chi-squared confidence sets for PPI, an important extension of the original PPI machinery that is not covered in the original papers. Indeed, PPI and its extensions typically focus on simple, low-dimensional inference tasks. The realization that chi-squared confidence sets can be used may open up new research directions to extend the PPI framework to higher-dimensional problems. These developments, along with the demonstration of the practical applicability of our framework across a variety of tasks and domains, highlight that our work is more than a direct result of the PPI framework.

More importantly, we believe our work is particularly timely and of significant scientific relevance given the growing need for efficient evaluation of increasingly complex machine learning models across scientific disciplines. This need is particularly obvious in the context of LLMs, where even crowdsourced validation like Chatbot Arena (Chiang et al. 2024), studied in our work, often suffers from insufficient data to resolve ties between models.

Marginal improvements relative to the classical approach

The other concern we would like to address relates to the perceived magnitude of improvement over the classical approach. In our experiments, AutoEval reaches efficiency ratios (effective sample sizes over sample sizes) between 1.25-1.40. Another way to appreciate these numbers is to think about the width of the obtained confidence intervals. An efficiency ratio of 1.25 translates into a 11% reduction in the width of confidence intervals, which, as shown in our experiments, significantly facilitates model ranking. See for instance Fig. 1c, where tighter confidence intervals translate for most sample sizes into at least a two-fold improvement in predicted rank correlation with ground-truth.

We believe these numbers are meaningful and highlight the practical relevance of our work in a variety of ML settings. That being said, the practical relevance of this improvement depends on the financial and time costs to produce human and synthetic validation data. Our method is particularly relevant in instances where human validation cannot be obtained in a timely manner. Examples include the Chatbot arena, where human annotation depends on web traffic, which cannot be controlled, or in biological assays where obtaining validation data may take weeks or months, but where producing large amounts of synthetic data can be done in a few hours.

Benchmark

But this method could also provide confidence intervals based on asymptotic normality, and this paper did not include that in most of experiments [...].

We would like to make an important clarification, as building valid confidence intervals of model performance is central to our work. In particular, Figure S1 of Appendix C shows confidence intervals not for one, but all of the main experiments (in Sections 2.3, 2.4, and 3.3) considered in this work.

These confidence intervals, which we demonstrate are well-calibrated and tighter than classical approaches in all experiments, are fundamental to our methodology as they enable statistically rigorous model comparison and selection even with limited labeled data.

I would like to see other metrics discussed.

We thank you for suggesting the exploration of additional metrics. In response, we have extended our analysis in the following way. We studied confidence interval coverage for multiple alpha values (0.8, 0.85, 0.9, and 0.95) beyond the original α=0.1, confirming consistent calibration across confidence levels. Regarding cross-entropy metrics, we would appreciate your clarification on how you envision these metrics being applied in our evaluation framework.

We thank you for your comments. We appreciate your recognition of our work's practical relevance in reducing ML evaluation costs and the strength of our benchmark.

Please find below point-by-point answers to your comments.

Positioning of our approach relative to semi-supervised training strategies.

This problem has been tackled in the literature with different names[...]. It is important to compare [...] against previously published work.

Thank you for the comment. Although these previously published topics handle related topics, our work is very different and does not fit in these existing areas. The fundamental reason is simple: our work is about evaluation, and your suggested topics are about model training.

To elaborate, pseudo-labeling uses predicted labels for unlabeled data as ground-truth during training (Lee 2013; Arazo et al. 2020). Curriculum learning is another model training paradigm that progressively increases training difficulty, e.g., via multi-stage training, to improve generalization (Bengio et al. 2009). Consistency regularization aims to enforce semantically similar inputs to have similar label predictions, relevant in semi-supervised training settings (Bachman, Alsharif, and Precup 2014; Laine and Aila 2016).

The key difference is that our focus is on the reliable evaluation of already trained models, rather than the training of these models on limited data. Our framework provides strong guarantees on the behavior of model performance estimates which is not the case for the mentioned literature. While these guarantees may not be relevant in a model training setting, they become crucial when evaluating models before deployment.

The proposed problem statement is relevant for training samples as well.

While PPI can be employed for training purposes, this direction is beyond our scope, and we make no claims for training applications. We plan to revise our related work to cover synthetic label for model training to emphasize distinctions with our work.

Evaluation of LLM responses [...] has also been studied in measuring uncertainty using structural similarity and other pair-wise metrics.

Thank you for pointing towards this related research area. A growing body of literature has indeed focused on evaluating LLMs from pairwise comparisons, often using LLMs as judges. We will revise the manuscript to include a more comprehensive and detailed discussion of this literature by including the references mentioned by Reviewer f29Q.

Benchmark

The effective sample sizes considered are very small, and important to evaluate the approach with larger datasets.

The labeled sample sizes in our work are small to illustrate practical settings with limited human validation; however, the results certainly hold for larger sample sizes. To address this point head-on, we studied the behavior of AutoEval for larger sample sizes in the imagenet experiment, as shown below:

Table: ImageNet experiment for $n=10,000$.

Above, AutoEval compares favorably to the classical approach for pointwise performance evaluation. It also provides tighter confidence intervals, large effective sample sizes, and proper coverage.

The claim that proposed methodology provides calibrated and tight confidence intervals is clearly apparent [...]. Need to compare with other principled uncertainty metrics.

To answer your comment, we conducted additional analyses to evaluate confidence interval calibration across significance levels (0.8, 0.85, 0.9, and 0.95), and compared directly with the classical approach. The results, to be included in the revised Supplement, demonstrate our confidence intervals maintain proper calibration across these significance levels. Empirical coverage closely matches the nominal coverage, indicating well-calibrated uncertainty estimates for AutoEval. We hope this addresses your concern, and would be happy to implement other metrics if you have other suggestions.

The approach is evaluated on Imagenet, protein fitness experiment and pairwise preferences for LLMs. The evaluation is limited.

The current manuscript describes several canonical applications of our framework, for a variety of tasks, in model evaluation and ranking across different domains. We believe that the current benchmark offers a comprehensive evaluation of the proposed framework in a variety of real-world settings.

Other comments

“There is no clear theoretical justification for PPI++ metric.

We would be happy to take your suggestions into account to clarify the distinctions between PPI and PPI++ if our description lacked clarity. Could you expand on which specific section of the paper you are referring to?