Predicting the Performance of Black-box Language Models with Follow-up Queries

We thank the reviewer for their detailed comments. We appreciate your recognition of the clarity of our paper and the practical relevance of the black-box performance prediction task. We address your comments below:

"The technical contribution of the proposed QueRE is limited, as it relies heavily on LLM-generated prompts (i.e., follow-up questions) and an overly strong hypothesis that the predictive distributions for these questions meaningfully vary with correctness."

While the mechanism of QueRE is intentionally simple, our core contribution lies in the empirical discovery and validation that this straightforward, non-obvious hypothesis holds true to a surprisingly effective degree. The key finding is not the hypothesis itself, but that such a simple black-box technique can yield representations powerful enough to consistently match, and even sometimes outperform, complex white-box methods that require privileged access to model internals. This result challenges the assumption that performant monitoring requires internal access and opens a new, practical avenue for black-box model auditing.

"The significance of this task is closely tied to the achievable performance upper bound. I think the authors should further investigate whether the task of black-box performance prediction is in fact predictable—i.e., whether LLMs are sufficiently aware of their own factually incorrect outputs, and whether this internal perception can be effectively extracted via follow-up questions."

While measuring a theoretical performance upper bound is a challenging open question, our work provides the strongest evidence to date that a significant degree of a model's "internal perception" is indeed extractable via black-box queries. Standard uncertainty quantification methods perform poorly on these tasks, suggesting the problem is non-trivial. The fact that QueRE significantly outperforms these methods demonstrates that there is substantial headroom for improvement and that our approach is a major step toward that upper bound.

From a usability perspective, this task is practically relevant. If we are able to accurately predict when a model will make a mistake, we can know when to use a model and when to defer to human intervention, a key goal for deploying LLMs in high-stakes settings.

"In the model performance prediction task, the authors should additionally report precision and F1-score on negative samples (i.e., cases where LLMs produce incorrect answers), as higher precision indicates more reliable detection of incorrect behavior."

Thank you for this suggestion. We have added precision and F1 scores on negative (incorrect) examples for LLaMA3-8B and LLaMA3-70B. QueRE remains the strongest black-box method across nearly all tasks. Below are results in the form of precision / F1:

We will include these results in our revision.

"A more realistic setup for black-box model identification should be determining whether a given response is or is not from a specific model. This implies that the negative samples in the test set should come from multiple different LLMs. The current setting—distinguishing between two specific models—is much easier and less representative of real-world applications, such as detecting fraudulent API substitutions."

Thank you for this suggestion. We completely agree that this is an important and realistic setting. We have now extended our evaluation to include multi-negative identification: determining whether a given response is from GPT-4o-mini, where negatives come from a pool of 3 other models (LLaMA3-8B, LLaMA3-70B, GPT-3.5). QueRE remains highly effective, achieving near-perfect performance and demonstrating its value for real-world applications like detecting fraudulent API substitutions. We will include these results in our revision.

Thank you again for your detailed review, and we hope our new experiments and our clarifications address your concerns.

We thank the reviewer for their detailed and positive feedback and for their excellent suggestions. We are encouraged that they found our approach to be simple, strong, and practical. We address the questions and suggestions below.

"As mentioned in weaknesses point 1, the datasets in section 4 are mostly released before the models evaluated have been released. Do the authors think this would generalize to newer datasets that are definitely not in the training set of these models?

Yes, we believe our method generalizes well to evaluation datasets that are not part of the model's pretraining distribution. To support this, in addition to GPT-3.5 on HaluEval (as you note), we include results for LLaMA2-7B and LLaMA2-70B on HaluEval, a dataset released after the training of LLaMA2. We observe that QueRE continues to perform strongly:

This suggests that performance remains strong on evaluations that are guaranteed to not be contained in the training data.

"The choice of using pre- and post- confidence scores in addition to the original vector z in the case of QA tasks should be ablated."

Thank you for the suggestion. We conducted an ablation in which we removed the pre- and post-confidence scores and found that the performance of QueRE remains strong. Notably, even without these features, QueRE continues to outperform the next-best black-box method (Semantic Entropy). Below are results for LLaMA3-70B:

We will include these results in our revision.

"In L117, how is the distribution over the set of possible answers calculated in the case of LLMs that don’t return probabilities? Also, I think the sentence is grammatically incorrect. Either the distribution over possible answers or what?"

In our experiments, we use the top-5 logits returned by the model to compute the distribution over possible answers. If a candidate answer appears in the top-5, we normalize its logit over the total mass of the top-5 to estimate its probability; otherwise, it is assigned probability 0.

Thank you for catching the grammatical issue. The intended sentence is: "we append the distribution over possible answers." We will correct this in our revision.

"Would it be possible to measure the ability of this approach to differentiate between a full/half precision model and a more severely quantized (<=4bit) version of the model? I think this is another important usecase for evaluating serving APIs aside from checking if they are serving the right model."

This is an excellent suggestion for another practical use case. We have run this experiment, extending our work on distinguishing between models. We find that QueRE can accurately distinguish between full-precision, half-precision, and 4-bit quantized versions of LLaMA3 models on the SQuAD dataset. These promising results suggest QueRE can be a valuable tool for auditing APIs to detect if providers are serving lower-precision models.

"The code has a typo with the name of the Boolean Questions (BoolQ) dataset, using i instead of an L for the dataset name."

Thank you for catching this! We have fixed this typo in our code and will ensure the correction appears in the final version.

We thank the reviewer for their positive feedback! We appreciate that you find our approach "simple yet highly effective" and that we tackle "multiple critical tasks". We address your comments below:

"Computational cost overhead compared to model internals approach"

Thank you for pointing this out. While QueRE does require multiple forward passes---one per follow-up question---we note that this cost is a trade-off inherent to the black-box setting, where model internals approaches are not applicable. We will update the paper to explicitly discuss this trade-off and its implications for deployment. In many high-stakes applications (e.g., safety-critical monitoring, auditing), we believe the improved predictive performance of QueRE justifies this cost.

"The authors note in line 116 that these follow up questions can be asked in parallel adding minimal computational overhead. However, parallelizing the computation would reduce latency but the cost would still remain high..., other baselines do not have such high computational costs."

We would like to highlight that QueRE is quite comparable in terms of both computational cost and latency to the other competitive black-box baselines of Semantic Entropy and Self-consistency. A key advantage of QueRE is that it only requires $k$ forward passes for the probabilities of a single token (e.g., "Yes") in response to $k$ follow-up questions.

First, this can even be faster than generating $k$ full sequences of many tokens (each of which requires $N$ forward passes to generate $N$ tokens), which is required by both Semantic Entropy and Self-consistency. In addition to these generations, Semantic Entropy then requires an additional step after sampling of using an embedding model and then clustering the embeddings of the responses. Secondly, in terms of computational cost, APIs often charge for the number of output tokens, which is much higher for these baselines.

To make this comparison more concrete, we have provided the results on SQuAD with LLaMA3-8B in terms of average runtime per example below. We find that our approach dominates other baselines at the same runtime. For example, QueRE with 10 follow-up questions outperforms Self-Consistency, which takes a similar amount of time, and strongly outperforms Semantic Entropy, which is much slower.

"Results only shown on simpler tasks: The datasets considered in the paper are comparatively simpler for the LLMs shown in the paper... I would also encourage authors to add model’s accuracy for each of the tasks"

This is an excellent suggestion. To clarify that the benchmarks remain challenging for the models we evaluated, we have compiled the models' accuracies below. As the data shows, performance is far from saturated on these tasks, making performance prediction a non-trivial and valuable problem. We will add a similar table to the Appendix in our revision.

"Comparison to stronger baselines like MICE: The authors have included white box and black box baselines for comparison"

Thank you for pointing us to this interesting work. We wish to re-emphasize that our paper is focused on the black-box setting, where such white-box approaches are not applicable. We include white-box baselines like RepE and Full Logits not as direct competitors, but rather to demonstrate that QueRE can surprisingly match—or even exceed—their performance without access to model internals. Furthermore, the MICE method appears to be a strong white-box technique for confidence estimation. However, as of writing, the code for [1] has not been publicly released—their GitHub repository is currently empty—which prevents a direct comparison.

"The approach shows surprisingly strong performance in predicting correctness on reasoning-intensive tasks like GSM8K and CodeContests, where many other methods fail. Given that the follow-up questions seem to probe for general confidence rather than specific reasoning steps, do the authors have a hypothesis for why these features are so predictive for complex reasoning tasks?"

This is a great question. While our follow-up questions are general, we hypothesize they are particularly effective on reasoning tasks (e.g., GSM8K, CodeContests) precisely because those tasks require long, structured outputs (i.e., chain-of-thought). After generating a complex reasoning chain, the model's internal state is likely much richer and more nuanced than after producing a single-token answer.

When we probe this state with follow-up questions, the model’s response probabilities (e.g., P(“Yes”)) serve as rich signals—capturing subtle cues in its reasoning trajectory. In essence, the model has more context from its own reasoning to "reflect" upon, allowing our probes to better distinguish between confident (and likely correct) reasoning paths versus flawed ones. In contrast, simple QA tasks often involve short outputs (e.g., one-word answers), which provide less of a signal.

"Have the authors investigated methods for optimizing this set of questions to identify a minimal, yet highly informative, subset? This could help reduce the computational cost associated with the method."

This is an excellent suggestion. As noted in lines 134–138, we have considered question optimization strategies (e.g., learning soft prompts), but opted for a fixed-question approach for several reasons: (1) it provides strong transferability across models and tasks, and (2) it enables tighter generalization bounds, since the questions are not optimized on data.. That said, we agree that identifying a minimal, highly informative question set is a promising direction, and we will highlight this as a valuable area for future work.

We thank the reviewer for their positive feedback. We are glad to hear that you find our problem setting as "interesting" and that our "experiment results are strong". We address your individual comments below:

"While QueRE gets strong performance on open-sourced models, it sometimes gets limited performance on closed-sourced models (e.g. Figure 2)"

We thank the reviewer for this careful observation. While QueRE's performance on closed-source models in Figure 2 (NQ, SQUAD) is competitive, we agree it is not as dominant as with open-source models on those specific tasks. However, we wish to highlight that across a broader range of evaluations, QueRE demonstrates consistently state-of-the-art performance for closed-source models. For instance:

• On reasoning benchmarks (Table 1), QueRE is the top-performing method for both GPT-3.5 and GPT-4o-mini on GSM8K and CodeContests, significantly outperforming all baselines.
• On closed-ended QA tasks (Figure 3), QueRE is the best black-box method for GPT models on HaluEval, BoolQ, and DHate.
• In our other application settings, QueRE achieves near-perfect accuracy in detecting adversarially influenced GPT models (Table 2) and distinguishing between different GPT models (Figure 4).

This comprehensive evidence suggests that our method is highly effective for closed-source models across a wide variety of important tasks beyond standard QA tasks.

"The studied benchmarks (e.g. SQUAD, Natural Questions) are outdated. It'd be great to see experiments on more realistic benchmarks (e.g. coding, human preference)."

We appreciate this suggestion. While we included SQUAD and NQ as standard, widely-recognized QA benchmarks, we made a concerted effort to evaluate on more modern and diverse tasks to ensure the relevance of our findings. Our experiments include results on:

• Coding: The CodeContests benchmark (Table 1).
• Mathematical Reasoning: The GSM8K benchmark (Table 1).
• Recent Datasets: The HaluEval benchmark, a recent and challenging hallucination detection dataset from 2023 (Figure 3).

Our evaluation spans commonsense reasoning, factual recall, toxicity detection, and hallucination detection, in addition to the crucial applications of detecting adversarial tampering and auditing model APIs. We believe this diverse suite of benchmarks provides a robust validation of QueRE's capabilities.

"It's unclear how different follow-up questions might influence the performance of the proposed method"

This is an excellent question. We investigated this in our ablation study in Appendix A.4 and Figure 7. In this study, we prompted GPT-4o to generate sets of questions with varying levels of human-interpretable diversity ("similar" vs. "diverse" questions). Our findings indicate that while performance steadily increases with the number of questions, the specific generation strategy (e.g., prompting for diversity) does not have a major impact when a sufficient number of questions are used. This suggests our method is robust to the exact composition of the follow-up query set. We will emphasize this analysis in our revision.

"While the authors claim that the latency issue can be mitigated through batching, I didn't find any relevant reports about the computation overhead... Could you present the results of latency v.s. AuROC?"

We have now included latency-vs-performance results for LLaMA3-8B on SQuAD, varying the number of follow-up questions up to the full 50 questions we use in our experiments. The following table compares QueRE to the key black-box baselines:

We find that QueRE consistently outperforms other approaches at similar or lower runtimes. The results clearly show that QueRE offers a superior trade-off. For a comparable latency to Self-Consistency (0.17s vs. 0.19s), QueRE with just 5-10 follow-ups achieves a dramatically higher AUROC (~0.90 vs. 0.53). Furthermore, QueRE significantly outperforms the much slower Semantic Entropy baseline. We will include this latency analysis in our revision.

"Do you think there are any types of tasks that the proposed method might fail?"

This is a great question! Our approach is designed to create a low-dimensional representation that is highly predictive of an LLM's behavioral properties (e.g., correctness, truthfulness, model origin). We believe this representation would be less suitable for fine-grained tasks, such as generation. For example, it would likely not be effective for predicting the exact sequence of tokens in a long-form text generation task.

Its strength lies in classification and regression tasks related to the model's state and the quality of its output. We believe this is a reasonable and important scope, as these representations are shown to be highly effective for crucial applications like monitoring model performance and detecting adversarial influence. We will add this clarification to our revision.