Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data

Thank you for the feedback. For the updated pdf, we have rewritten the experiment section to improve clarity and better highlight how the experimental results relate to our theoretical results. We also have added additional details to the figure captions.

Regarding the experiments, we would like to highlight that we also did experiments on MT-Bench, an arena-style benchmark (See Figure 3). As suggested by reviewer eL89, we also added results on TruthfulQA (Appendix B2).

Regarding the questions:

“Is it straightforward to extend the theory from the binary case to the continuous case (as in the experiments)? Or are there techniques specific to binary outcomes.”

We do not think that an analogon to Theorem 5 (the strict optimality of PPI) is possible for continuous scores, as the likelihoods become non-parametric in that case. As such, we can only hope to obtain results on the sample efficiency gains from PPI rather than for arbitrary debiasing methods. We have added new results on this for the case with a non-binary proxy score to Section 5, making use of a generalized formulation of the agreement score.

“What is the intuition behind the constant factor equalling two exactly?”

It is difficult to provide a strong intuition for the specific factor of two. The proof of our new Theorem 10 provides some intuition for why the sample efficiency gains are finite: They are equal to the ratio of the Variance of s and the MSE between $\tilde{s}$ and s. Letting both the judge accuracy and the evaluated model’s accuracy go to one has both the MSE and the variance go to zero at a similar rate, yielding a finite factor.

“What is the relation between PPI and doubly robust estimation from causal inference?”

PPI and doubly robust estimation are closely related. They both aim to debias estimates based on machine learning predictions and often reach identical estimators. Their crucial difference lies in the assumptions made about the machine learning model. Doubly robust estimation, as part of semiparametric inference, typically assumes that the machine learning model estimates the true nuisance function consistently, e.g., at a rate of $o(n^{1/4})$. PPI does not impose these restrictions. The machine learning model is truly treated as an unknown black box. Further discussions can be found in Angelopoulos, et al, 2023.

References: Angelopoulos, A. N., Duchi, J. C., & Zrnic, T. (2023). PPI++: Efficient prediction-powered inference. arXiv preprint arXiv:2311.01453.

Thank you for the positive review!

We have added additional results on stratified PPI to Appendix B3 in the updated pdf. To disentangle the gains from stratification (which have nothing to do with using an LLM judge) from the gains of (stratified) PPI, we focus on per-stratum sample efficiency gains, using MMLU subtasks as strata. When utilizing a weaker model to evaluate a stronger one, we rarely observe sample efficiency gains of more than a factor two for any stratum. This is perhaps to be expected, as our main theorem still applies for each stratum. So a judge model can only yield sample efficiency gains of more than a factor of 2 on a stratum if it is stronger than the evaluated model on that stratum.

We now provide experiments on TruthfulQA in Appendix B2. We use the top three models from the HELM leaderboard as judges, and find no sample efficiency factor exceeding 1.3. This is substantially below our upper bound of 2, validating our theory.

Thank you for your thorough feedback.

We took your feedback to heart and made a thorough pass for clarity and readability for the updated version of the pdf. In particular, we have added more detail to the transition between sections to improve the narrative flow and clarify the structure. We are happy to address any further suggestions.

We have added more details on the last experiment on incorporating the uncertainty of judge models and supplemented it with an additional theoretical result. In short, there are two key takeaways: First, making use of a judge model’s uncertainty improves sample efficiency gains. Second, the sample efficiency factor remains clearly below two for evaluating SOTA models. The latter is in line with our new Theorem 10 on sample efficiency gains for non-binary proxies $\tilde{s}$.

Based on your review, we moved the section on balanced agreement to the appendix. We hope this improves the paper’s focus on the main message. Balanced agreement was intended as an intuitive alternative to the commonly reported agreement metric that provides more meaningful bounds on the sample efficiency factor $\tau$. We added a plot for the bounds on $\tau$ afforded by balanced agreement on MMLU to Appendix A, showing that they are valid and meaningful. However, we now believe that directly computing $\tau$ is more useful than computing either agreement or balanced agreement.

Answers to specific questions:

“Proposition 1: The explanation is unclear. What does "point-wise" mean in the context of line 166?”

By “better in a point-wise sense”, we refer to the condition in Proposition 1 that for all $x \in X$, $\tilde{m}(x) = y(x)$ implies $m_i(x) = y(x)$”. In other words, a model $\tilde{m}$ is better than another model $m_i$ point-wise if $\tilde{m}$ only makes mistakes on data points $x$, on which $m_i$ is also wrong.

“Line 133: Why is it stated that precisely modeling the specific relationship is infeasible? Could you provide an example to clarify this?”

Sure. Let us consider MMLU as an example. A model of the specific relationships would be a (parametric) mapping from a question text $x$ and LLM model $m$ to both the correct score $s(x,m)$ (i.e. whether LLM model $m$ correctly answered question $x$) as well as the proxy score $\tilde{s}(x,m)$ (i.e. whether LLM model $m$ correctly answered question $x$ according to the LLM judge). Such a model would be highly complicated. For instance, given the question “Paper will burn at approximately what temperature in Fahrenheit?” from MMLU, it would need to correctly predict both whether GPT-4 answers the question correctly, and whether GPT-4 gives the same answer as Claude.

“In the second setting in the experiment, arena-sylye benchmark, we see sample efficiency lower than 2 when using GPT 4v. but when evaluating on llaam2 -7b it's over 2, what's the examplantion here? is it due to the hugh performance gap between llama2-7b and the judge model?”

We assume that you are referring to the performance shown in Figure 2. In this case, you are correct. Our theorem states that we can at most get a factor 2 if the evaluated model is better than the judge model. But GPT-4 is a lot better than LLama2-7b, such that the larger sample efficiency factor is not ruled out by our theory. We have added additional details to better explain this in the experiment section.

We hope that this addressed all your concerns. If so, we would kindly ask you to raise your score to reflect the high relevance and strong contribution of our work indicated in your review.

Thank you for the positive review.

We made extensive edits to improve the writing in the updated pdf, and would be happy to address any further suggestions.

Regarding the binary evaluations, we would first like to point out that most popular benchmarks use binary evaluations. For example, all tasks in the huggingface open LLM leaderboard (https://huggingface.co/docs/leaderboards/open_llm_leaderboard/about) use binary evaluations. That said, neither the need for debiasing methods nor the specific methods we employ require binary evaluation.

On the theoretical side, moving away from binary scores s and/or proxies $\tilde{s}$ yields non-parametric likelihoods, making an analogon to Theorem 5 incredibly hard. Correspondingly, PPI might not be strictly optimal any more, even though the results from Theorem 5 imply that it is optimal in a certain worst-case formulation.

However, we were able to extend Theorem 6 to the case of non-binary proxy scores. The results are presented in Section 5 and are based on a novel generalization of the agreement score to non-binary proxies. Our new theorem 10 suggests that using PPI, a weaker judge evaluating a stronger model is not going to improve sample efficiency by more than a factor of two, even for non-binary proxy scores $\tilde{s}$.