Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees

Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.

1. Not sure if significance is well justified. For example, in Figure 9a), it seems like R-AutoEval and R-Eval are very similar. It is hard to know whether this will be a widely used evaluation method in the future.

The fact that in some cases using auto evaluator (e.g., LLMs-as-judges) may bring no gain as compared to a vanilla approach is a fundamental limitation of the autoevaluation framework. When the autoevaluator is of insufficient quality and/or the problem at hand is easy enough to be sufficiently analyzed by the human-labeled data, existing approaches such as [17] suffer from performance degradation by utilizing autoevaluators. This is exactly the behavior observed in Figure 9a, where R-AutoEval [17] (blue line) shows worse performance than R-Eval (green).

However, the benefits are evident when the autoevaluator is sufficiently accurate. For instance, the results for TriviaQA data set (Fig. 4(a), Fig. 7(a)) show that unless the autoevaluator is heavily quantized (i.e., BF16 and MX6), autoevaluator can help finding smaller quantized models. Furthermore, the results on Instruction-Induction task results in Fig. 4(b), Fig. 7(b) show that unless autoevaluator is operated on low quality prompts (few number of in-context samples), autoevaluator can help finding shorter prompts for the base models.

To further validate the significance of the proposed method, we have carried out additional experiments on more challenging LLM tasks that solves math problems. Specifically, in light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set $n=1000, N=4000, \delta=0.1$, and report the averaged output from 100 independent runs. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework.

Table 1: LLM test-time scaling experiment for math problem (GSM8K)

2. Seems like the authors renamed the method in [17] to R-AutoEval without mentioning the original name.

Thank you for pointing this out. The original method in [17] was dubbed “SS-RCPS”, and we have introduced a new name for the consistency. We will revise the manuscript by mentioning the original name when we first introduce [17].

3. In practice how do you deal with the probably correct result i.e. giving the correct answer with probability greater than $1-\delta$? What happens if the prediction is wrong?

Wrong predictions are inevitable due to the randomness associated with the realization of the calibration data (see standard PAC-type guarantees). That said, even in such worst-case settings, R-AutoEval+ can still yield close-to-safe decisions with a minimal violation rate. This can be seen from Fig. 1, where we have reported the worst case within 1.5 interquartile range (IQR) range. Furthermore we have provided the full box plots in Fig. 6 (Appendix D). There, we can observe the attained worst-case performance. The tables below (from high-quality autoevaluator to low-quality autoevaluator) confirm that R-AutoEval and R-AutoEval achieves worst risk lower than 0.125 which corresponds to $1.25\alpha unlike R-Eval that suffers from worst risk near 0.14 (\1.4\alpha$). Furthermore, the actual violation ratios of R-AutoEval and R-AutoEval+ are lower than R-Eval up to 1$\delta=10$%), demonstrating that autoevaluation can provide further benefit in terms of managing such extreme cases.

Table 2: Worst-case analysis for LLM quantization, autoeval=Llama3.3-BF16 (Fig. 6(a))

Table 3: Worst-case analysis for LLM quantization, autoeval=Llama3.3-MX6 (Fig. 6(a))

Table 4: Worst-case analysis for LLM quantization, autoeval=Llama3.3-MX4 (Fig. 6(a))

4. Have authors compared the introduced method with some other evaluation methods (apart from R-Eval, R-AutoEval and others discussed in the paper)?

In the manuscript, we have focused on comparing our methods with Eval and AutoEval (mean-based approach, no reliability guarantees) and R-Eval and R-AutoEval (non-asymptotic reliability guarantees), since they are the state-of-the-art evaluation methods offering statistical validity. For reference, we have now also considered a method that only provides asymptotic reliability guarantees [Boyeau et al., 2024, Fisch et al., 2024] based on the central limit theorem. For the GSM8K data set, we have observed that its violation ratio is 0.12, which is higher than the target level 0.1. If the work gets accepted, we will further compare our scheme with other asymptotic benchmarks.

5. Line 168, should it be R-Eval instead of AutoEval?

You are correct! We will fix the typo, thank you.

References

[Boyeau et al., 2024] Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Autoeval done right: Using synthetic data for model evaluation. arXiv preprint arXiv:2403.07008, 2024.

[Fisch et al., 2024] Adam Fisch, Joshua Maynez, R Hofer, Bhuwan Dhingra, Amir Globerson, and William W Cohen. Stratified prediction-powered inference for effective hybrid evaluation of language models. Advances in Neural Information Processing Systems, 37:111489–111514, 2024.

Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.

1. Empirically validate their approach on toyish LLM tasks: Quantization and Prompt Shortening tasks. Could be useful to perform a broader evaluation on evaluation of an LLM on a broad range of topics (e.g math, coding, general QA)

During the rebuttal, we have carried out additional experiments on more challenging LLM tasks. In light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set $n=1000, N=4000, \delta=0.1$, and report the averaged output from 100 independent runs. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework.

Table 1: LLM test-time scaling experiment for math problem (GSM8K)

2. Evaluation limited to a single family of models and judge/autoevaluator (Llama family of models). Could benefit from a broader evaluation over different model families and sizes to see the effect of performance of the strength of the autoevaluator.

To address the reviewer’s point, we have considered autoevaluators with different sizes, accuracies on GSM8K data set, whose results and the corresponding test-time computation that can be saved by R-AutoEval+ as compared to R-Eval and R-AutoEval are shown in the table below.

Table 2: Broad choice of the autoevaluators

It is generally observed that choosing the autoevaluator from the same family of the model substantially reduces the gain of autoeval-based approaches. For instance, Llama-3.2-3B-Instruct autoevaluator achieves much lower accuracy – 0.16 – than Qwen3-32B autoevaluator but it can save much more reasoning budget -- 48.34 tokens. This demonstrates the importance of the model choice as the reviewer correctly anticipated. We believe that such behavior is related to the positive feedback of LLM judges within the same family [Zheng et al., 2023] also known as preference leakage [Li et al., 2025], which makes the bias correction (8) a more challenging task. We will summarize this behavior in the updated draft if this work gets accepted.

3. Have you done a study of your approach over model size as the reliability of LLM as judge may change with respect to model parameters?

In Fig. 1, Fig. 4(a), Fig. 6, and Fig. 9, we have considered three different LLM as judges by adjusting their size from 70.6 B to 26.7 B and 17.9 B. Furthermore, as described in the reply above, in the new experiment on test-time scaling for GSM8K data set, we have investigated the crucial impact of the choice of LLM as judges.

4. Have you looked at scaling properties over risk with respect to the amount of human and synthetic data that is used?

In Fig. 4(a) and Fig. 7(a), we have varied the amount of human data from 100 to 300 ($n: 100 \rightarrow 120 \rightarrow \cdots \rightarrow 300$) and observed its scaling trend. To further understand the scaling trend with the synthetic data size, we have carried out additional experiments that vary the size of synthetic data from 1000 to 6000 ($N: 1000 \rightarrow 2000 \rightarrow \cdots \rightarrow 6000$) for a fixed amount of human data for the new math experiment. We adopt GPT-4.1 as the autoevaluator and the results are summarized in the following table. It is observed that all the schemes ensure the target risk (accuracy gain no less than 3%), and that autoeval-based approaches can increasingly save the reasoning budget with increased number of synthetic data. Interestingly, R-AutoEval+ shows a more robust behavior than R-AutoEval, demonstrating again the power of the proposed adaptive design.

Table 3: Impact of the number of synthetic data $N: 1000 \rightarrow 2000 \rightarrow3000 \rightarrow4000\rightarrow5000\rightarrow6000$

5. Would it be possible to evaluate this framework in a larger scale setting as the ones described above?

In the new experiments on the GSM8K data set, we have considered a wide range of autoevaluators from model size from 0.85 B to 70.6 B, as well as for the GPT4.1 models which are generally assumed to be in a much larger scale setting. The results in Table 2 confirm the general applicability of the proposed scheme.

References

[Zheng et al., 2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023.

[Li et al., 2025] Dawei Li, Renliang Sun, Yue Huang, Ming Zhong, Bohan Jiang, Jiawei Han, Xiangliang Zhang, Wei Wang, and Huan Liu. Preference leakage: A contamination problem in llm-as-a-judge. arXiv preprint arXiv:2502.01534, 2025.

Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.

1. The details of real data experiments could have been presented more thoroughly.

If this work is accepted, we will make sure to add all the relevant details for the real-data experiments, including the method used for constructing the unlabeled data set and the measure of similarity between the human labeled data and synthetic data.

2. Since the weights are learned dynamically by the observations up to , does the order of the data to the algorithm affect the results? How robust is the algorithm against the data ordering?

Yes, the order of the data can affect the final result. To clarify this point, we have carried out additional experiments. Specifically, in light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set $n=1000, N=4000, \delta=0.1$, and report the averaged output from 100 independent runs with different calibration-test data splits. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework.

Table 1: LLM test-time scaling experiment for math problem (GSM8K)

To focus on the robustness of the algorithm to the ordering of the data, we fix the calibration-test data split and only change the ordering of data to compute the normalized deviation (std / mean) of the test-time computation required by each algorithm. We will consider GPT-4.1 and DeepSeek-R1-0528-Qwen3-8B as the autoevaluators for the experimental results henceforth. As can be seen from the table below, R-AutoEval and R-AutoEval+ are more robust to the data ordering than R-Eval, possibly attributed to its reduced variability of averaged autoevaluated data (at each round we average $r=N/n$ autoevaluated data) while R-AutoEval+ tends to be slightly less robust than R-AutoEval for its adaptive nature that incorporates the data ordering.

Table 2: Sensitivity of algorithms with respect to data ordering

3. Regarding the real-data experiments, how was the unlabeled data constructed for TriviaQA? How similar does the synthetic data need to be with the human labeled data? How were the hyperparameters, such as $n, N, S$, initial weight, synthetic-to-real, etc., determined?

As TriviaQA does not explicitly contain unlabeled data, we have split the data set and ignored the presence of label information in one split to construct the unlabeled data set. The synthetic data indeed need to be similar enough with the human labeled data to ensure improved efficiency of R-AutoEval+. To clarify this similarity, we have measured the accuracy of the autoevaluator in the first table of this reply, which essentially captures the similarity of autoevaluated data with human labeled data. The hyperparameter $n$was searched within the interval 100 to 300 as shown in Fig. 4(a), while we have chosen the hyperparameter$N$based on the amount available unlabeled data (from the above construction). In our experiments, we have chosen the hyperparameters associated with the candidate factors by setting$S=10$ with equal grid and equal weighting, since such setting gave us a good result in toy experiment. Nonetheless, we have further carried out careful experimental analysis on the impact of hyperparameter choice during the rebuttal.

We first investigate the impact on the number of candidates, S, with candidate factors equally spaced within the interval [0,1]; and with equal initial weights. In what follows, we will adopt GPT-4.1 and DeepSeek-R1-0528-Qwen3-8B as the autoevaluators.

Table 2: Impact of the number of candidate factors ($S$)

As can be observed in the table above, as long as the candidate factors includes $\rho=0 (R-Eval) and \\rho=1 (R-AutoEval), i.e., as long as \S \geq 2$, R-AutoEval+ consistently outperforms both R-Eval and R-AutoEval, with its gain being maximized at$S=5$, and saturates after$S=10$.

Next, we investigate the impact of the values of each candidate factors by going beyond the equally spaced grid. To this end, we consider Dirichlet distribution with concentration parameter 0.1, 1, 10 to allocate the $S-2$ values within the [0,1] interval (we keep the first and the last factors to be 0 and 1 to ensure the efficiency guarantee as per our Theorem 3). The cumulative probabilities of the Dirichlet distribution are used as the candidate factors. An increased concentration parameter makes samples from the Dirichlet distribution closer to the uniform distribution, and hence the grid becomes closer to the equally spaced grid. The table below shows that equal grid performs the best.

Table 3: Impact of the value of candidate factors ($\{\rho_s\}_{s=1}^S$)

The last remaining degree of freedom is given by the choice of the weights associated with each candidate factor. To this end, we go beyond the uniform distribution and consider again the Dirichlet distribution to allocate the weights for the $S$ candidate factors. The table below confirms again the preference for equal weighting.

Table 4: Impact of the weights associated with the candidate factors ($\{w_{s,0}\}_{s=1}^S$)

4. Similarly, for the instruction-induction task, how was the unlabeled data constructed? How many human evaluation and synthetic samples were used?

We followed the same approach as TriviaQA, and we have considered 200 human evaluation and 1800 synthetic samples.

References

[Cobbe et al., 2021] Karl Cobbe, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

[Muennighoff et al., 2025] Niklas Muennighoff,et al. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.

[Snell et al., 2024] Charlie Snell, et al. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.

Thank you for the constructive comments. Below, we provide a point-by-point response to each of the reviewer's comments.

1. Limited experimental validation (only tested on LLM quantization and prompt design)

During the rebuttal, we have carried out additional experiments on more challenging LLM tasks. In light of the growing attention on test-time scaling of LLMs [Snell et al., 2024, Muennighoff et al., 2025], we applied our framework to the problem of identifying the minimum amount of additional computation that ensures desired performance gain. We adopted Qwen3-1.7 B as the base model as it provides both non-reasoning and reasoning modes [Yang et al., 2025]; and we considered the task of solving math problems (GSM8K) [Cobbe et al., 2021]. We allow the computation budget for the reasoning mode to vary between 128 to 1280 number of tokens, and our goal is to find the minimum reasoning budget that achieves at least extra 3% accuracy gain as compared to the non-reasoning mode. The table below shows the amount of test-time computation in the unit of tokens (i.e., the number of generated tokens to answer the math question) that can be saved by R-AutoEval+ as compared to R-Eval (4th column) and R-AutoEval (5th column) for autoevaluators with different sizes, accuracies on GSM8K. We set $n=1000, N=4000, \delta=0.1$, and report the averaged output from 100 independent runs. We omit the accuracy gain results as all the schemes always achieve at least the target accuracy gain 3%. The table confirms again the efficiency and validity of the proposed framework.

Table 1: LLM test-time scaling experiment for math problem (GSM8K)

In the following, we will use the new setting to address the reviewer’s comments.

2. Uses a fixed, discrete set [...], potentially limiting flexibility

This is indeed a limitation of the scheme as we briefly described in Sec. 5 (Conclusion and Further Discussions). To further explore the impact of the selection of candidate factors, we have carried out an experiment in which we vary the number of candidates, S, which are equally spaced within the interval [0,1]. In what follows, we will adopt GPT-4.1 and DeepSeek-R1-0528-Qwen3-8B as the autoevaluators.

Table 2: Impact of the number of candidate factors ($S$)

As can be observed in the table above, as long as the candidate factors includes $\rho=0 (R-Eval) and \\rho=1 (R-AutoEval), i.e., as long as \S \geq 2$, R-AutoEval+ consistently outperforms both R-Eval and R-AutoEval, with its gain being maximized at$S=5$, and saturates after$S=10$.

Next, we investigate the impact of the values of each candidate factors by going beyond the equally spaced grid. To this end, we consider Dirichlet distribution with concentration parameter 0.1, 1, 10 to allocate the $S-2$ values within the [0,1] interval (we keep the first and the last factors to be 0 and 1 to ensure the efficiency guarantee as per our Theorem 3). The cumulative probabilities of the Dirichlet distribution are used as the candidate factors. An increased concentration parameter makes samples from the Dirichlet distribution closer to the uniform distribution, and hence the grid becomes closer to the equally spaced grid. The table below shows that equal grid performs the best.

Table 3: Impact of the value of candidate factors ($\{\rho_s\}_{s=1}^S$)

The last remaining degree of freedom is given by the choice of the initial weights associated with each candidate factor. To this end, we go beyond the uniform distribution and consider again the Dirichlet distribution to allocate the weights for the $S$ candidate factors. The table below confirms again the preference for equal weighting.

Table 4: Impact of the initial weights associated with the candidate factors ($\{w_{s,0}\}_{s=1}^S$)

3. Sample efficiency [...] high target reliability levels

This is indeed a limitation of the study, as described in Sec. 5 (Conclusion and Further Discussions). That said, in practice, autoeval methods may be particularly relevant for safety-critical risk-averse scenarios [Giambona et al., 2018], in which high reliability levels are a requirement. Furthermore, we have observed empirically that, even under moderately low reliability levels $1-\delta (with larger \\delta$ indicating a less strict reliability requirement), R-AutoEval+ consistently outperforms R-Eval and R-AutoEval. For the ongoing example, this is shown in the following table.

Table 5: Efficiency gain for a range of target reliability levels $\delta$

4. What's the computational overhead of the algorithm compared to other methods?

Given both the human-labeled (size $n) and autoevaluated data (size \N), mean-based approaches (Eval and AutoEval, which do not come with reliability guarantees) require \O(n) (for Eval) and \O(N) (for AutoEval) operations for computing the respective average. The reliable approaches (R-Eval and R-AutoEval) need additional computation steps for updating the e-values as well as the betting strategies (Sec. 2). While updating the e-values at each round requires a single multiplication, the betting strategy may be more complex. For example, especially the universal portfolio (UP) strategy with sublinear regret (Assumption A1) has a complexity of order \O(nG)$where$G$is the size of the grid used to approximate the integral operation of UP (Sec. C.1), although this can be decreased by adopting sampling-based approaches [Kalai and Vempala, 2002]. Lastly, R-AutoEval+ requires$S$times more computation than R-AutoEval due to its consideration of$S$ candidate factors. It is worth emphasizing that, the costs described above are generally negligible with respect to running the autoevaluator (e.g., an LLM judge). The summary of the computation complexities can be found in the below table.

Table 6: Computational complexities of the algorithms considered in this work

5. How sensitive [...] hyperparameters?

The hyperparameters in the proposed R-AutoEval+ encompass the number of candidate factors, the values of the candidate factors, and the weights associated with the candidate factors, whose sensitivity has been analyzed in the reply above.

6. The paper goes into mathematical details quite [...] depth.

We have tried to provide a high-level discussion through the introduction section and Fig. 1, but the page limitations at the time of submission made it difficult to provide further general descriptions. We will certainly add further discussions on the high-level problem statement using the additional content page in case the work is accepted.

References

[Cobbe et al., 2021] Karl Cobbe, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.

[Muennighoff et al., 2025] Niklas Muennighoff,et al. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.

[Snell et al., 2024] Charlie Snell, et al. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.

[Yang et al., 2025] An Yang, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

[Giambona et al., 2018] Erasmo Giambona, et al. The theory and practice of corporate risk management: Evidence from the field. Financial Management, 47(4):783–832, 2018.

[Kalai and Vempala, 2002] Adam Kalai and Santosh Vempala. Efficient algorithms for universal portfolios. Journal of Machine Learning Research, 3(Nov):423–440, 2002.

Thank you for providing additional details and clarifications in the rebuttal.