Scaling Up Active Testing to Large Language Models

Thank you for your hard work and helpful feedback! We’re delighted that you appreciate the practical significance and rigorous experimental validation of our work. We hope our response below addresses your remaining questions.

Did you consider running ablations for the individual components of your technique? I’m specifically curious about the ablation of adding previously acquired test labels into the in-context training of the surrogate models, rather than fixing the in-context training. I acknowledge that the ablations may have been prohibitively costly to run. [...] The relevant importance of each of the individual components of the proposed technique is unclear, because there were no ablations for the individual components.

Excellent suggestion! Following your feedback, we have conducted ablation studies comparing fixed versus dynamic surrogates, where the dynamic approach incorporates newly acquired labels into the surrogate’s context and recomputes predictions, and thus acquisition probabilities, at each sampling step. Our experiments evaluate the few-shot Llama-2 7B and 70B models with dynamic and fixed Llama-2 7B surrogates, starting with 10 in-context examples over 40 steps (50 runs each for dynamic surrogates, 3000 runs for fixed surrogates and uniform sampling). You can find a comparison of the mean relative error to random uniform sampling for acquisition with and without the dynamic surrogate below along with the 5th and 95th percentiles.

In summary, we find that dynamic surrogates show no clear consistent improvements in risk estimation over fixed surrogates, while imposing prohibitive computational costs. The experiments with 50 repetitions each were over 1500 times more expensive for 40 steps than the frozen ones, whose cost remains fixed as the number of steps increases. Additionally, dynamic surrogates require unique predictions at each step, eliminating the ability to precompute and share predictions across multiple evaluations. While there may be interesting applications of dynamic surrogates, for our selection of tasks and models, the results support our design choice of fixed surrogates, which achieves the same accuracy benefits while maintaining the computational efficiency that makes this active testing method practical. We are happy to add these results to the revised version of the paper.

Scaled mean squared error (x$10^{-4}$, 5th-95th percentile) for uniform random sampling, dynamic surrogate active testing, and fixed surrogate active testing on 7B-few and 70B-few across datasets

The authors conclude that the surrogate and target models do not have to be from the same family. However, the experiments only used two model families, so this result may not be very general. Additionally, the advantage of using a same-family surrogate model versus a different-family surrogate model was not quantified. This may be an area for future work.

That’s a good point, thank you for giving us the opportunity to clarify! Our experiments actually span three model families, namely Llama-2, Gemma-3 and Phi-2 but some of these are only in the supplement (see Figures C.1. and C.2.). Evaluations demonstrate the robustness of active testing across model families. We agree that further investigation of the effects of model diversity between the surrogate and target would be interesting future work but are happy to offer some thoughts here: Interestingly, while the original active testing paper suggests that surrogate-target diversity is generally beneficial, we find that using smaller surrogates from the same model family can be highly effective. Although, there may still be scenarios where diversity between the target and surrogate model is more advantageous, it is possible this is less pronounced for LLMs, which are likely much more correlated in their predictions (e.g. due to shared architecture and parts of the training data) than the models considered by Kossen et al. (2021).

How were the datasets for the experiments selected?

The datasets we chose are common LLM tasks, especially in the in-context learning literature (see In-context learning learns label relationships but is not conventional learning, Kossen et al, 2022 from which we took direct inspiration). We added MMLU as a more challenging, widely-used benchmark to demonstrate the robustness of active testing.

The authors ran experiments using datasets that are relatively low-difficulty for recent LLMs in 2025, and did not include datasets requiring open-ended generation. Open-ended generation datasets would have been a more realistic reflection of the problem setting, because these datasets are expensive to acquire human labels for.

Evaluation on tasks requiring open-ended generation represents an interesting direction for future work, but we note that this faces fundamental technical challenges that extend beyond the scope of this submission. The acquisition functions for active testing that we study require the computation of the models predictive uncertainty. For open-ended generations with LLM, uncertainty quantification is still an area of active research (see for example Detecting hallucinations in large language models using semantic entropy, Farquhar et al, 2024), in particular when considering cost-effective uncertainty estimation. As an alternative, we would like to point out that our classification tasks and especially MMLU remain challenging for the selection of models we studied, as shown in tables C7 and C8 of the supplementary material.

The graphs are too dense and hard to read.

Thank you for pointing this out! We will revise the graphs accordingly to your suggestions and make sure they appear more clearly.

Again, thank you for your review. We appreciate your questions and suggestions, which contribute to a more thorough analysis of our approach. Please let us know if there is anything else that we can clarify.

Thank you for your review. Unfortunately, we believe there may be a fundamental misunderstanding about our work’s scope and contributions. We hope to clarify these in our response below.

Most importantly, we do not propose a method to improve model predictions or to reduce their cost for standard inference applications. Our work only targets LLM evaluation. The core challenge is that evaluating frontier LLMs on large datasets is prohibitively expensive–both computationally and in terms of labeling costs. This is a critical bottleneck as models become larger and more numerous and as evaluation datasets expand.

We do not claim novelty for in-context learning or entropy-based acquisition (W1), but rather demonstrate how to combine these techniques to make active testing practical for LLMs at scale. Our technical innovations aim to reduce the computational cost of active testing through (1) our use of fixed, in-context-learning–based, surrogate models, (2) using surrogates that are computationally cheaper than the target model, and, ultimately, (3) our proposal to replace even target model predictions with surrogate predictions for the largest computational cost savings.

W1) Limited novelty: The central method primarily relies on approximating a target prediction with a cheaper prediction using ICL, an idea that by itself lacks novelty [1,2,3]. Additionally, the acquisition method presented in Section 3.3 resembles entropy-based acquisition methods commonly used in active learning. Philosophically, I do not know if we want to replace the induced test distribution by a cheap surrogate model over the specialized target model.

We believe there may have been a fundamental misunderstanding here that we hope is easy to clear up. Our use of the surrogate model does not “induce” a different test distribution, nor are we trying to improve on the predictions of the original model in any way: it is being used to construct a proposal over which test points to acquire labels for, as is required by our R_LURE importance sampling estimator (Eq. 1). This estimator will be unbiased for any acquisition proposal distribution q that has full support (which ours do) and thus for any surrogate. For well-chosen q, the estimate with R_LURE can be significantly lower variance than a naive random sampling baseline. As the surrogate only affects the proposal, it does not introduce any difference in test distribution. Importantly, the true labels and target model predictions always enter the estimator via the loss term, L(f(x), y), in the estimator, regardless of the choice for q.

To reiterate, we do not suggest replacing target model predictions with surrogate predictions for general prediction scenarios. We only consider this for the purposes of computing the acquisition proposal q, as using a non-uniform proposal allows us to reduce the number of test labels we need to acquire, while using a small surrogate means this proposal is cheaper than using a large LLM. The estimator remains unbiased and estimates the test performance on the true distribution underlying the test data. It further retains the same convergence guarantees as a standard uniform Monte Carlo estimator.

The works [1, 2, 3] that you refer to study how to actively construct the subset of samples to include in ICL, aiming to improve predictions. This is entirely orthogonal to our work as we do not actively construct the subset of samples used in the ICL of the surrogate. Instead, we use a fixed and randomly selected set of samples. Our acquisition function acquires samples for use in the active testing estimator and not for inclusion in ICL. We further note that Kossen et al 2021 have previously shown that active learning acquisition strategies are not directly optimal for the active testing estimator itself as the active learning and active testing problems are quite different in which points we wish to target. As such, the fact that strategies related to entropy have been used in active learning is tangential to our actual contributions.

We hope this clears up this misunderstanding and will clearly highlight differences and similarities to related work in a revised version of the draft.

W2) The proposed idea could be interesting if the ICL surrogate model effectively replaces a trained, task-specialized LLM. However, the evaluation predominantly tests only within the scope of in-context learning, and in the context of classification benchmarks. Thus, if the surrogate and target models exhibit similar capabilities–even with different model sizes– particularly in certain domains, the findings become less compelling.

Again, this point suggests there may have been a fundamental misunderstanding of our method: our work has nothing to do with replacing the original “task-specialized LLM” with an ICL surrogate. As explained above, the surrogate is only used as a mechanism to more efficiently evaluate the original LLM by constructing an importance sampler. Note also that it is a surrogate of the ground truth label distribution, not of the target’s distribution, with the aim to identify incorrect predictions of the target. Even a much smaller model (eg Gemma-3 4B or Llama-2 7B) can often identify errors made by a larger model (eg Llama-2 70B), because discrepancies in predictions can detect errors independently from task performance. Our experiments demonstrate this clearly, in Figures 2, 3, and Figures C1, C2 of the supplementary material. Whether the target model and final surrogate have similar capabilities when used in general prediction scenarios is irrelevant to our contribution, as we never suggest replacing the predictions of the former with the latter.

W3) Consequently, if we assume small LLM to have similar capabilities to large LLMs, then the question is when this does not hold. I am a bit sceptic that the proposed approach “always” work. Since the paper is mostly empirical, I think we need to understand when does not work and when it work, which is not extensively explored.

We want to clarify that we never claim our approach always works independently from the surrogate’s choice, but this does not mean that we have to assume a small LLM has similar capabilities as a large LLM. We provide extensive experiments across a wide range of surrogates size, using Llama-2 7B and 70B, Gemma-3 4B and 1B as well as Phi-2 (note there are additional results in the appendix to investigate surrogate choice to those in the main paper), to analyze failure cases and boundary conditions. Our results demonstrate that active testing is generally beneficial when the surrogate is appropriately selected, even when the surrogate is a lot smaller than the target model, and fails gracefully in worst-case scenarios, with performance comparable to uniform sampling. Even a small surrogate can provide meaningful signal in the acquisition function that allows us to improve over uniform random acquisition.

Could the author expand more real world applications where labelling cost is not as important as inference compute time?

Computational costs dominate in several settings, such as model selection based on test datasets, or benchmark leaderboards when evaluating numerous models for comparison on large, already labeled datasets.

Again, thank you for your review. We hope our response helped to clarify the motivation of our work and why its contributions are highly distinct from other uses of surrogates in an LLM context. Please let us know if there is anything else that we can clarify.

Thank you for your hard work and helpful feedback! We are delighted that you appreciate the novelty and practical impact of our approach for cost-effective evaluation. We hope our response below alleviates your remaining concerns.

Theoretical underpinnings: While empirically effective, the reliance on in-context learning for surrogate models may need more detailed theoretical analyses for generalization error bounds.

We respectfully disagree with the claim that the use of in-context learning for surrogate models significantly complicates obtaining theoretical guarantees: the surrogate is only being used as a proposal within a static importance sampling framework, so we inherit all the well established strong guarantees of unbiased i.i.d. Monte Carlo estimators (noting that our proposal is constructed in such a way that it is guaranteed to form a valid proposal so the estimator is unbiased and consistent). The guarantees will all be effectively the same as when using uniform sampling of new datapoints, except with a single sample variance. If our proposal is better than a uniform, this variance will be lower (and this is typically the case as shown in our experiments), if it is worse the variance will be higher.

Instability in Few-Shot and Complex Tasks: Figure 2 for the 70B-few model on Subj and HS datasets, risk-estimation error is close to or worse than uniform random sampling. This may stem from surrogate models (trained via few-shot in-context learning) failing to capture the target model’s behavior when both are in few-shot modes.

While it is indeed an important point that the method can do similar or worse to uniform sampling if the surrogate performs poorly (and we have tried to ensure this is clearly shown and discussed in the paper), we emphasise that these are more graceful degradations rather than sudden instabilities.

In particular, even in these worst-case scenarios, our method gracefully degrades to uniform sampling performance, maintaining a consistent –and close to 1– relative error. This highlights the robustness of our method, and is crucial for practical deployment as practitioners can safely apply our method without risk of obtaining estimations that are significantly worse than random sampling. We also note that such failures correspond to poorly capturing the true label distribution, rather than the target model’s behaviour itself. We would be happy to add further clarifications, discussions, and/or ablations on this if you think it would improve the paper.

In few-shot scenarios, why does the method fail more often than in zero-shot setups? Is it due to mismatches between surrogate and target model capabilities?

Excellent question! The key insight is that while the surrogate model uses few-shot learning in both scenarios, the target model receives different contexts. In zero-shot settings, the target model is not given any context and therefore performs worse, making more errors that can be easily identified by the surrogate model. In few-shot settings, the target model’s improved performance reduces the number of errors and the errors that do occur can be harder to predict, making it harder for the surrogate to identify them.

Can dynamic surrogate adaptation (e.g., updating surrogates with small label batches) mitigate errors in few-shot or complex tasks?

Thank you for raising this important question! To test it, we explored surrogate adaptation and ran experiments with dynamic surrogate updating (detailed in our response to reviewer cFrJ). We didn’t observe clear consistent benefits in performance, but doing it drastically increased the computational costs: updating surrogates requires recomputing predictions over the entire sampling set with increasingly large contexts, which can be prohibitively costly in practice. We observed that experiments with surrogate updating took over 1500 times longer to run for 40 steps than when freezing surrogates.

Given the huge expenses associated with this uncertain performance gain, we don’t believe surrogate adaptation is beneficial to active testing in most cases. However, you are right that there might be specific applications where adaptation could be valuable, particularly when labeling costs massively exceed computational costs. We view this as an interesting direction for future work and will add discussions on it along with our new results.

Again, thank you for your review! We appreciate your questions and suggestions, which contribute to a more thorough analysis of our approach. Please let us know if there is anything else that we can clarify.

Thank you for your feedback! We appreciate that you recognize the substantial computational savings and flexibility of our method. We hope our response below alleviates your concerns.

The acquisition rule–uncertainty sampling via Shannon entropy–and the use of surrogate models are already commonplace in active-learning research, limiting methodological novelty.

There is a significant difference between active learning and active testing and our core novelty is being the first paper that shows how active testing can be successfully used in LLM evaluation. We do not claim that the use of Shannon entropy is a novelty, nor the use of surrogate which was already introduced in the original active testing paper. Our technical contribution is the overall adaptation to active testing to make it practical for LLMs at scale. Technical innovations that have not previously been done in the active testing context include: (1) using in-context learning to create surrogates that are fixed during sampling, without expensive retraining; (2) using a significantly smaller surrogate compared to the target, thereby reducing inference costs for surrogate predictions; (3) replacing the target model predictions with surrogate predictions to reduce computational costs even further when budgets are tight.

Uniform random sampling is a weak baseline; comparisons against stronger alternatives such as margin sampling, BALD etc. are needed to gauge true gains.

We disagree with the claim that uniform sampling is not an appropriate baseline, as it is the standard approach used in practice for evaluation. The idea of using an active approach for LLM evaluation is itself novel, while approaches like margin sampling and BALD are active learning acquisition strategies, not active testing acquisition strategies. As established in the original active testing paper (Active testing: sample-efficient model evaluation, Kossen et al, 2021), most active learning acquisition functions are inappropriate for testing scenarios because active testing aims for accurate risk estimation rather than model improvement, which is conceptually a different problem. Our contribution lies in making active testing practical for LLMs despite prohibitive training and inference costs, demonstrating that existing acquisition functions can be useful for computational and labeling efficiency.

There are no theoretical guarantees presented for the recommended estimators (using surrogate model), leaving their performance behaviors under different settings unclear.

We strongly disagree that our method lacks theoretical guarantees. Our approach is grounded in established importance sampling theory: we employ a static importance sampler that is provably unbiased (On statistical bias in importance sampling: How and when to fix it, Farquhar et al, 2021) and samples uniformly at random (i.i.d) since the surrogate distribution remains fixed. It therefore maintains the same theoretical guarantees as uniform sampling. In particular, following guidelines from Active testing: sample-efficient model evaluation, Kossen et al, 2021, our proposal is known to be valid, and thus its convergence and unbiasedness are guaranteed. Additionally, this estimator can drastically reduce the variance of estimates compared to uniform sampling when the acquisition proposal is properly chosen.

We appreciate though that these theoretical properties could have been made clearer in the submission and will update it accordingly.

Although the authors have discussed some limitations, the paper would benefit from a deeper analysis of the conditions under which the proposed estimators are reliable—and where they may fail—as well as a clearer discussion of how far the empirical findings can be generalized beyond the specific datasets and models tested.

The variety of experiments we conducted does in fact allow for extensive analysis of when our estimators are reliable versus when they fail. Our estimators prove reliable when the surrogate can effectively identify target model errors. When this correlation breaks down, performance gracefully degrades to uniform sampling. This can happen when the surrogates are too small for the target’s complexity (see examples with Phi-2 and Gemma-3 1B in Figures C1 and C2 of the supplementary material), for challenging tasks where surrogates struggle to capture incorrect answers (see MMLU dataset in Section 5.7), or when labels are noisy (as discussed in Section 5.8). Even in failure modes, our method degrades to uniform sampling’s performance, providing estimates that a practitioner can still rely on.

Our experiments span three model families, sizes from 1B to 70B parameters and diverse tasks. The consistent 32% average improvement across these settings demonstrates broader applicability. The key insight is that effectiveness depends on the acquisition proposal and therefore the surrogate’s ability to identify the target’s error rather than absolute capabilities, making the approach transferable to new models and tasks.

Freezing the surrogate model will prevent the active sampler from incorporating information gained during data collection, which can degrade efficiency.

Our decision to freeze surrogate models is empirically justified by our experimental results, leading to significant gains in error estimation, and computationally motivated, saving additional costs. Updating surrogates requires recomputing predictions over the entire sampling set, with increasingly large contexts at each step, which defeats our motivation for computational efficiency.

That said, we have now added a new ablation with dynamic updating of the surrogate to assess how big the benefits of doing this might be, see our response to reviewer cFrJ for the new results. These new results show that dynamically updating the surrogate does not provide clear consistent benefits, while prohibitively increasing computational costs: in our new experiments using dynamic surrogates for 40 steps only was over 1500 times more expensive than the frozen ones we recommend.

The paper does not justify its choice of datasets; outlining the selection criteria and explaining how these datasets reflect the broader evaluation landscape would strengthen the empirical study / Can the proposed approach be extended to regression tasks with LLMs, and what modifications would that require?

The SST-2, FPB, Hatespeech and Subjectivity datasets are very common LLM tasks, especially in the in-context learning literature (see for example In-context learning learns label relationships but is not conventional learning, Kossen et al, 2022 from which we took direct inspiration for our experiment selection). We added the more challenging MMLU dataset to confirm our results on a commonly used and more difficult classification task. While our method extends naturally to any task type, as the acquisition function can be anything, as explained in Section 2, LLM evaluation predominantly focuses on classification due to the discrete token space. With classical neural networks, Kossen et al. 2021 (Active Testing: Sample-Efficient Model Evaluation) have already explored application of active testing to regression tasks, where the acquisition function now approximates MSE.

How small can the surrogate model be before performance deteriorates, and how rapidly does evaluation accuracy decline as the surrogate's quality decreases?

Good question! In general, active testing provides bigger benefits the better the surrogate is. Our analysis of small surrogates (Phi-2 in Figure C1 and Gemma-3 1B in Figure C2 of the supplementary material) identifies the practical size limits for surrogate models for our selection of tasks. For some tasks, we observe performance deteriorating when these small models evaluate much larger targets like Llama-2 70B-few. Importantly, active testing does not fail catastrophically for decreasing surrogate quality. Instead, it gracefully degrades towards the label-efficiency of standard uniform sampling. Note however, that small models do outperform uniform sampling in many settings, including when evaluating Llama-2 7B.

While these results provide empirical evidence demonstrating the robustness of active testing under deteriorating surrogate quality, we agree it would be interesting work to further quantify this. We note, however, that any useful measurement of ‘surrogate quality’ would need to take into account the target model and evaluation task – and is thus unlikely something a practitioner could compute upfront. This is where we see our bootstrap estimates providing valuable insight for practitioners applying active testing!

How will this approach perform under distribution shifts?

Given that the models we used haven’t directly been trained on these tasks, we are not sure one can really think of things in terms of a “distribution shift” so much as there being tasks the LLM finds relatively harder to predict than others. We test on tasks of varying difficulty and find our approach to be helpful on both easier and harder tasks, indicating that the approach remains robust to this kind of problem.

Again, thank you very much for your review. We hope that our response has addressed your concerns. Please let us know if there’s anything else we can clarify.