Thank you for your thoughtful and detailed feedback on our paper. We are glad to hear that you found the topic interesting and appreciated the clarity of our writing and structure. Your comments have helped us improve the clarity of our work and we have updated the manuscript accordingly.

The insight that there are connections between in-context learning and Bayesian methods is not novel (https://arxiv.org/pdf/2111.02080).

Thank you for the opportunity to clarify. We note that the paper of Xie et al. is cited in the original manuscript (line 201). We hope our revised manuscript better conveys that our work builds off the Bayesian interpretation of ICL and addresses the underexplored question of whether the latent model implied by a CGM is appropriate for a given task. This is related to testing Assumption 4 (Well-specification) of Xie et al.. We have added Figure 2 to illustrate the consequences of using a misaligned model to make inferences about data generated according to a given ICL task. Figure 2d illustrates a particularly concerning case where the model is confident and wrong about its predictions. We design our methods to predict these cases and cases when there are not enough in-context examples for reliable task completion. For the latter cases, the utility of the NLL discrepancy function becomes apparent. We have added synthetic experiments in Section 7.2 to illustrate that while the NLL and NLML discrepancy functions lead to accurate predictors of model capability, only the NLL discrepancy provides information about whether there are enough in-context examples. We further demonstrate in Figure 10 that this information can reduce risk, a necessary consideration in safety-critical applications. Our contribution enables the estimation of discrepancies like the NLL using CGMs like large language models.

The experimental analysis is limited to a synthetic regression task and two real-world datasets.

We have added additional experiments to address this concern. For the language task, we add AG News as another in-capability task and RTE as another out-of-capability task. Figure 8 illustrates that the generative predictive $p$-value is still a robust predictor of out-of-distribution tasks. In applications of generative AI such as image or document completion, accuracy is either poorly defined or does not capture the complexity of the task. We devise an in-context generative fill experiment using the SVHN (in-distribution), MNIST (near OOD), and CIFAR-10 (far OOD) datasets. The model is prompted with several in-context image examples and asked to complete the missing half of the last example. The goal is to produce sensible completions, but not necessarily reconstruct the target image exactly. Figure 5 of the updated manuscript illustrates this task. Figure 5a demonstrates that when the model is trained on sequences of SVHN images, it provides plausible completions given the first half of missing images from the held-out test set. Figure 5b illustrates that when the model is asked to do the same task on OOD MNIST images, it produces plausible completions often, but also frequently hallucinates odd completions or artifacts as in rows 4 and 5. Finally, when the model is prompted to complete OOD CIFAR-10 images, it produces consistent completions but is also prone to clear hallucinations from the SVHN domain as in row 4. Figure 9 plots the OOD detection metrics and shows that the generative predictive $p$-value again yields an accurate predictor for a task where accuracy is not a suitable metric.

Why are there no references in the first two paragraphs of the paper?

We appreciate your concern and have added citations for in-context learning, posterior predictive checks, and Medprompt to the introduction.

Figure 1: Did you sample? If yes, at what temperature?

We sampled responses at a temperature of 0.95.

Why not consider a bigger model (e.g. Llama-3 405B) and see if it can perform better than random on MQP?

We are very keen to assess our methods using larger models, but are currently limited by compute resources to do so. More importantly, the tasks we evaluate become in-capability as model size increases, so they would not be sensible for out-of-capability detection.

"ancestral sampling":

Thank you for your question! At the end of the day you are right that ancestral sampling just ends up reducing to sampling next tokens. We chose it, however, because we were refering to the higher level "abstractions" of $x_i$ which can be made up multiple tokens and we sample "chunks" of $x_i$ at a time and we thought that it made a better allusion to this fact. Moreover, we thought this was ok as it is a commonly used term for autoregressive sampling.

I don’t think the posterior predictive p-values always share the theoretical properties of frequentist p-values

Thank you for this question. In our case, similar properties hold. For the sake of the argument, assume that $x^{\text{test}} \sim p_{\text{test}}(x)$.

We can then use as our null hypothesis the statement that $p(f \mid x^n)$ concentrates around values of $f$ such that $p_{\text{test}}(x) = p(x \mid f)$. That is, the null is that our model is concentrated around the distribution from which $x^{test}$ was sampled. Under this null, the $p$-values used in the paper are uniformly distributed with respect to the distribution of $p_{\text{test}}$.

It is easily shown why this is the case. First, note that if $f, f'$ are two values around which $p(f \mid x^n)$ concentrates, we have that $p(x \mid f) = p_{\text{test}}(x) = p(x \mid f')$. In turn, this also implies that $g(x, f) = g(x, f')$. Therefore, picking any such $f'$, we can rewrite

as

Because $p(x \mid f) = p_{\text{test}}(x)$, this can be further rewritten as

Assuming that the CDF of $g(x, f')$ is continuous under $p_{\text{test}}(x)$, this can be rewritten as

where $F_g()$ is the CDF of $g(x^{\text{test}}, f')$ under $p_{\text{test}}(x)$. Finally, by the properties of the CDF, we know that $F_g(g(x^{\text{test}}, f'))$ is uniformly distributed, and so $p_{\text{ppc}}$ must also be uniformly distributed.

Practically, this has consequences similar to the frequentist case. In particular, we could use it in the traditional hypothesis-testing style to reject models with guarantees on Type 1 errors.

The notation |(z_i, y_i)| was a bit confusing to me. I think this refers to the number of tokens comprising the pair of query and response?

We have added a clarification on line 383 of the updated manuscript.

In the abstract, “is equivalent sampling” -> “is equivalent to sampling"

Thank you, we have corrected this in the updated manuscript.

[T]o what extent do we need discrepancy functions that rely on having the model likelihood and model posterior explicitly?

The differences between $g$ become apparent when the model is uncertain because there are too few in-context examples to specify the task.

To illustrate this difference, imagine two models. The first model has posterior $p_1(f \mid x^n) = N(0, 1)$ and likelihood $p_1(x \mid f) = N(f, 0.0001)$. The second model has posterior $p_2(f \mid x^n) = N(0, 0.0001)$ and likelihood $p_2(x \mid f) = N(f, 1)$. In both models, the posterior predictive is essentially the same standard normal, $p(x \mid x^n) = N(0, 1)$. However, in the first model, the posterior predictive variance is due to epistemic uncertainty (we are unsure about the correct $f$), while in the second model, it is due to aleatoric uncertainty (we are confident about $f$, but the task is inherently stochastic). We illustrate this example in Figure 16 of appendix F in the updated manuscript.

Now assume that we have a test point $x^{\text{test}} = 0.5$. Ideally, we want to say that the second model does a good job of predicting this data point because the task is well specified, while the first one does not because we are still uncertain about the task. That is, the first model assigns high probability to many values of $f$ where $x^{\text{test}}$ is unlikely.

Depending on the discrepancy we choose, we may or may not be able to distinguish between the two scenarios and reject the correct model. For example, if we use

$\log p(z_i, y_i \mid x^n)$ is the same for both models and we will have identical $p$-values, which will indicate that both models are suitable—an undesirable outcome.

On the other hand, if we use

the $p$-values will be quite different. For the first model, many values of $f$ will fall far away from $x^{\text{test}}$ when computing

As a result, $g(x^{\text{test}}, f)$ will be much lower than $g(x, f)$, and the PPC will be quite low. However, in the second model, this will not happen, as all values of $f$ will be sampled around $0$, and $x^{\text{test}}$ is a reasonable observation for a normal distribution centered at $f \approx 0$ with standard deviation $1$. Therefore, using the $f$-dependent PPC provides the desired behavior because it informs us when the model is confused about the task.

We have added Section 7.2 to highlight that the NLL discrepancy function is indicative of whether there are enough in-context examples. This added information is useful in risk-sensitive applications where the cost of responding incorrectly is high. For example, a medical recommendation system that autonomously responds if the $p$-value is greater than the significance level $\alpha$. We use prediction RMSE over task responses to measure reliability. Figures 10a and 10b plot the RMSE against the $p$-values computed under the NLL and NLML discrepancies for in-distribution polynomial tabular tasks. We see that lower $p$-values correlate with higher RMSE for the NLL discrepancy, but not for the NLML discrepancy. At $\alpha=0.1$, the NLL discrepancy reduces the generation of responses with higher error because it accounts for the number of examples provided. To quantify this intuition, we define risk as the sum of task RMSEs for in-capability predicted tasks. Figures 10c and 10d show that for equally accurate predictors the NLL discrepancy results in substantially reduced risk.

Since the $p$-value computed under either discrepancy yields accurate predictors of model capability, the choice between discrepancy functions ultimately comes down to a decision on whether the added computational cost of generating dataset completions is justified. If you need to know whether there are enough in-context examples to generate an accurate response---a necessity in risk-sensitive applications---then we recommend using the NLL discrepancy function. If computational efficiency or the cost of response deferral are primary concerns---practical user experience considerations---we suggest using the NLML discrepancy.

Thank you for your thoughtful and constructive feedback on our paper. We are delighted that you found our Bayesian framework for model criticism in in-context learning (ICL) interesting and that it helped deepen your understanding of ICL. Your positive remarks on our theoretical formulation and the intuitive estimator we introduced are greatly appreciated. We address each of your concerns below and have updated our paper accordingly. Your feedback is sincerely appreciated and has significantly improved the quality and clarity of our work.

The method is presented as a way to estimate when a CGM can solve an ICL task. Can’t you often just directly evaluate this, though?

Thank you for raising these points. We agree that when ground truth labels are available and a clear metric is defined, directly evaluating the CGM’s performance on an ICL task is straightforward. However, as the reviewer noted, the strength of our proposed method lies in its applicability to situations where evaluation metrics are hard to specify or labels are hard to source. For instance, if the ICL task involves open-ended queries or responses, metrics like accuracy may be insufficient, but our proposed checks are still effective. This flexibility is especially valuable when the CGM operates as part of a larger system where in-context examples are highly heterogeneous, making it impractical to define a single evaluation metric (e.g., an API servicing diverse responses using ICL). Moreover, since our approach gives an interpretable measure of model capability, it could be an invaluable tool for the scalable oversight of models capable of discovering solutions that even domain experts have difficulty judging. This flexibility is essential for enabling the automated use of CGMs in diverse scenarios.

We address this concern directly by adding an experiment to the empirical evaluation. In applications of generative AI such as image or document completion, accuracy is either poorly defined or does not capture the complexity of the task. We devise an in-context generative fill experiment using the SVHN (in-distribution), MNIST (near OOD), and CIFAR-10 (far OOD) datasets. The model is prompted with several in-context image examples and asked to complete the missing half of the last example. The goal is to produce sensible completions, but not necessarily reconstruct the target image exactly. Figure 5 of the updated manuscript illustrates this task. Figure 5a demonstrates that when the model is trained on sequences of SVHN images, it provides plausible completions given the first half of missing images from the held-out test set. Figure 5b illustrates that when the model is asked to do the same task on OOD MNIST images, it produces plausible completions often, but also frequently hallucinates odd completions or artifacts as in rows 4 and 5. Finally, when the model is prompted to complete OOD CIFAR-10 images, it produces consistent completions but is also prone to clear hallucinations from the SVHN domain as in row 4. Figure 9 plots the OOD detection metrics and shows that the generative predictive $p$-value again yields an accurate predictor for a task where accuracy is not a suitable metric.

We clarify that the MQP example is used solely to validate the scalability of our method to natural language tasks using pre-trained LLMs. We acknowledge that Llama-2 7B is not well-suited for this task, which is precisely why we chose to test our method against it.

Thank you for your thoughtful and constructive feedback on our paper. We are delighted that you found our pedagogical writing and the gradual development of the Bayesian framework for in-context learning (ICL) engaging and enjoyable. Your appreciation of our effort to make the material accessible is truly encouraging. We address your concerns below and have updated our paper accordingly. We appreciate your feedback, which has greatly enhanced the quality and clarity of our work.

[I]f the goal is to judge whether my model can solve a task in-context, can I not just evaluate on said task via a benchmark and get a performance estimate (e.g., accuracy)?

Thank you for this question. We agree that if a good metric exists, then PPCs should follow this metric closely. However, the advantage of using PPCs is that they also work in situations where no such metric exists or where metric specification is hard. For example, if the in-context learning problem consists of open-ended questions, then simple metrics like accuracy or correctness would not suffice. This flexibility is especially valuable when the CGM operates as part of a larger system where in-context examples are highly heterogeneous, making it impractical to define a single evaluation metric (e.g., an API servicing diverse responses using ICL). Moreover, since our approach gives an interpretable measure of model capability, it could be an invaluable tool for the scalable oversight of models capable of discovering solutions that even domain experts have difficulty judging. This flexibility is essential for enabling the automated use of CGMs in diverse scenarios.

We address this concern directly by adding an experiment to the empirical evaluation. In applications of generative AI such as image or document completion, accuracy is either poorly defined or does not capture the complexity of the task. We devise an in-context generative fill experiment using the SVHN (in-distribution), MNIST (near OOD), and CIFAR-10 (far OOD) datasets. The model is prompted with several in-context image examples and asked to complete the missing half of the last example. The goal is to produce sensible completions, but not necessarily reconstruct the target image exactly. Figure 5 of the updated manuscript illustrates this task. Figure 5a demonstrates that when the model is trained on sequences of SVHN images, it provides plausible completions given the first half of missing images from the held-out test set. Figure 5b illustrates that when the model is asked to do the same task on OOD MNIST images, it produces plausible completions often, but also frequently hallucinates odd completions or artifacts as in rows 4 and 5. Finally, when the model is prompted to complete OOD CIFAR-10 images, it produces consistent completions but is also prone to clear hallucinations from the SVHN domain as in row 4. Figure 9 plots the OOD detection metrics and shows that the generative predictive $p$-value again yields an accurate predictor for a task where accuracy is not a suitable metric.

[I]f the model generates the data, wouldn't it have a relatively high likelihood it, especially compared to holdout data that it did not generate?

The concept that the generated data can have lower likelihood than the test data can seem counterintuitive. However, when the model accurately estimates the distribution of the task, it can generate data with higher or lower likelihood than the holdout data, which is sampled from the task distribution. If the discrepancy for generated data from is often greater than or equal to that of the holdout data, it suggests that the model does not overfit and generalizes well to new data.

[E]xperimental section [is] rather sparse on discussion

We appreciate your feedback regarding the sparsity of discussion in the experimental section. We have streamlined the background sections to allocate more space for detailed analysis and discussion of our experimental results. Specifically, we have added a discussion section to compare the OOD detection metric results in Section 7.1. Further, we have added additional experiments and discussion in Sections 7.2 and 7.3 to give more insights into the choice of discrepancy function.