Large Language Bayes

Thanks for your review!

When sampling PPL models, are the output generations individualized, meaning they each have their own “message thread”?

This is a good question. Yes, each sampled model had its own "thread", as mathematically required for model to be a valid IID sample from p(m|t).

If yes, how did the authors deal with the LLM generating/suggesting the same model multiple times? Any statistics on this? Wouldn’t this behavior possibly be a bias built into the LLM? Do you have any thoughts on how to handle this when writing the model with the PPL?

If the LLM wanted to suggest the same model multiple times, it was free to do so. We simply treated those as separate samples and processed them independently. This is correct, mathematically, but is potentially computationally wasteful. (It's silly to run MCMC multiple times on the same model.)

Note that it's extremely rare for the LLM to generate exactly the same model. However—unfortunately—it's possible to create equivalent models, and these are somewhat challenging to detect. For example, equivalent models might use different variable names, or declare those variables in a different order. For these reasons we unfortunately can't easily offer any hard statistics. However, our qualitative impression is that this happens fairly often for the very simple models (e.g. coins) with the more complex models, a large fraction of the generations are unique. (This can be seen, to some degree, by looking at the distribution of estimated marginal likelihoods in the detailed experimental results in section B.)

How would this method compare with traditional Bayesian learning, where the optimization problem is to find the model parameters from a distribution that could, potentially, have a prior?

We're very sorry, but we can't quite understand this question. We're confused by the description, which doesn't correspond to traditional Bayesian learning as we understand it. We'd like to reply, so we'd be grateful for a bit more detail on what's being asked here.

Any thoughts on using the approximate ~p(z|x,m) as a score for dynamically prompting the LLM? In an iterative approach, one could use the posterior approximation as a guide for the LLM prompt template, aiming for more accurate models.

Another good question! We have considered several variants of this idea, but haven't validated one yet.

Here's one option. Imagine you generate one model at a time and do inference on it before generating the next model. One could conceivably take some of the top scoring models and provide them as additional examples as part of the prompt. Done naively, this would, of course, bias the results. However, one could potentially correct this with importance sampling—after the chain of thought text, generate the model with the extra examples, compute the probability of that generation, and then also compute the probability using the LLM without the extra example models. In this way, one could hope that the model would converge to the true results more efficiently. We caution that we haven't actually done this yet. But we believe this general kind of strategy (along with the ideas we allude to on lines 303-307) may greatly improve the method.

Thanks for your review!

model averaging as a weighting strategy can suffer from high variance and unstable weighting in the case of model misspecification

We believe these issues are captured by our analysis in section 5.1.In particular, see the expression for the asymptotic variance of the estimate in Eq. 9. (Or a more careful analysis in C.2) This variance is determined rather the divergence between the LLM prior p(m|t) and the posterior p(m|x,t). So the main issue, as we see it, is if the LLM prior p(m|t) is reasonably broad. We allude to these issues on line 228-231, though it may be appropriate to give this discussion more emphasis. (Please also see our response to eRLK for more discussion of this point.)

Can the authors systematically characterize which varieties of approximate inference they tried out in order to get the scheme we see in this manuscript? How do the experimental results vary if hyperparameters or the choice of Monte Carlo algorithms are varied?

Sure. For the MCMC step, we immediately settled on using Stan's default algorithm with no modifications. We never experimented with any variants of this. The only question was how to estimate the marginal likelihood. To do this, we experimented a little bit with doing importance-weighted VI using normalizing flows. However, we found that this was a mess, as it's very difficult to make stochastic optimization over 10,000 models fully automatic. Thus we discarded it and switched to the current method.

The final method, while pretty expensive, is absolutely bulletproof and has very few hyperparameters to set. All that happens for each model is:

1. Do MCMC using Stan's default algorithm.
2. Compute the mean and covariance of those MCMC samples.
3. Create a Gaussian with that mean and covariance.
4. Repeat N times: Draw M samples from the Gaussian, and estimate the importance-weighted ELBO.

(See Algorithm 9 in section F)

There are literally no hyperparameters in this algorithm other than N or M. We used M=25 and N=10,000, simply on the basis that they seemed reasonable. We never experimented with tuning either of them.

Provided that Stan's MCMC method succeeds, this method is extraordinarily robust. After creating it on small test problems, we only ran it once to generate all the results. There was no secret fiddling. There may be some bias to the marginal likelihoods if the posterior for some model is poorly approximated by a Gaussian and importance-weighted VI cannot correct this. However, notice that if this happens, the failure is "gentle"—the resulting ELBO will be lower than the marginal likelihood, meaning that model gets lower posterior weight than it deserves. In this way, the method is very "safe".

Thanks for your review!

Is my understanding of "only need to match target" correct?

It is true that the user only needs to specify the target variables of interest. While running, the LLM can and does invent other random variables. This is extremely convenient since most applied statistics problems involve, e.g., hierarchical modeling and partial pooling. But here the LLM takes care of all that for you. (Please let us know if we misunderstand the question.)

Can the authors provide some discussion on how can one apply their method in practice?

Sure. In practice, all the user needs to do is provide a model text (as a .txt file) and a data file (in our implementation, as a .json file). Then, our implementation uses a custom system prompt and sequence of example generations to generate a set of candidate Stan models (See sections F.1, F.2, and G). Then, each of these models is compiled into a Stan executable and (if it compiles) MCMC inference is run. Finally, those MCMC samples are used to construct a Gaussian distribution. That Gaussian distribution is then used for importance-weighted VI to estimate a marginal likelihood. Finally, all the MCMC samples are weighted by the marginal likelihoods and returned to the user.

Practically speaking, running this algorithm is a challenge. We run the LLM on a cluster with an A100 GPU using continuous batching. The generations are saved as text files. Then, separate (CPU) jobs are submitted for each to run MCMC and estimate the marginal likelihoods. The final combination is easy and can be done on any local computer.

How does the computational cost behave when applying the proposed algorithm?

Please see section F for specifics on this point. Note also that we also give some ideas for future directions to improve efficiency on lines 303-307.

Thanks very much for your review!

Regarding this quote:

"However, existing algorithms for Bayesian model averaging (e.g. reversible jump MCMC [8, chapter 11]) do not appear to be applicable here, since they involve explicitly iterating over all models (here very larger or possibly infinite) and/or using evaluations of p(m|t)."

We believe this remark is correct as written, although we concede that we weren't as clear as we could have been.

We certainly agree that the final posterior can be seen as an instance of Bayesian model averaging, and we tried to be explicit about that (See lines 50-51).

However, there is a crucial distinction. In the setup of this paper, we assume that it is not possible to evaluate p(m|t). We briefly allude to this on lines 75-79, but we should be more explicit as it is a subtle point.

There are two issues here:

First, most commercial LLM providers (e.g. OpenAI) simply do not provide probabilities for their generations—probably to make it harder for other companies to distill their models.

Second, for open-source LLMs (like llama, which we use) generation probabilities are available. However, in order to encourage the LLM to generate better models, we ask it to generate a "chain of thought" where it informally describes the modeling strategy. (See the examples in Section G.) We have found this to be crucial for generating high quality models. But the impact is that the generation probabilities for the model (alone) are not available. Rather, the generation probabilities give you something like p(informal text, formal model). Finding the probabilities of the model (alone) is completely intractable as it would require marginalizing out all possible informal texts.

Regarding scalability, we must be straightforward that we simply do not intend to claim great scalability. The current inference method is, frankly, extremely expensive, as it requires doing MCMC on ~1000 models. (See section F for some discussion of timing.) However, we believe that there is great opportunity for faster inference methods (see lines 303-307) that would greatly increase scalability. At the same time, we respectfully suggest that given that this is the first method to directly define this kind of posterior directly from user text, it is useful to establish the basic sanity of the framework, and that even a method that can only (currently) address modestly-sized problems constitutes a valid contribution.

Why has the algorithmic variants in section 5.3 been provided when there are no experiments to back up the claims here?

We have given a series of increasingly complex algorithmic variants so that it is possible to discuss and analyze the different algorithmic components separately. Our final algorithm (given explicitly as Alg 7, section D) uses all the enhancements discussed in this section, and these are crucial for making inference possible.

What kinds of models would experts and non-experts have generated for these scenarios and how do the highest weight models compare to those?

We were advised that doing this would be considered human subjects research and thus subject to extremely cumbersome legal and IRB requirements. Our hope was that the reader of the paper could largely play this role. For example, in Figures 6-8 (appendix B.1) it is shown that for the rain example, essentially all posterior weight is given to a single model, which we provided in the text as m^(742). Realistically, we believe it is reasonable to expect the reader of the paper to look at the given informal text and this model and judge for themselves if the final model model should be considered reasonable. The same is true for the other models.

How well calibrated are the posteriors?

We ask for a bit more detail on what is requested here. (We may be able to provide it.) Typical notions of calibration (as, e.g., in a brier score decomposition) require doing inference a number of times and comparing the predictions to observed outcomes. This would not be practical given the expense of this method. For the city temperature problem, we actually did provide some small amount of evidence in Figure 30.

We do not claim that the work is unrelated to the BMA literature. We must protest that this is not a fair summary of the paper or our response. On the contrary, the posterior is not just related but mathematically equivalent to BMA, the only difference being that the prior comes from an LLM. This is stated prominently in the paper right after the main derivation on lines 50-51:

Note that Eq. (4) can be seen as an instance of Bayesian model averaging [16 , 32 ], just with a prior over models that’s defined using an LLM.

This is also stated before the quote under dispute:

As mentioned in Sec. 2, LLB can be seen as an instance of Bayesian model averaging [16, 32 ]. However, existing algorithms for Bayesian model averaging (e.g. reversible jump MCMC [8, chapter 11]) do not appear to be applicable here, since they involve explicitly iterating over all models (here very larger or possibly infinite) and/or using evaluations of p(m|t).

We accept that this may not be as clear as possible. But we stress that second sentence is only intended to refer to algorithms for BMA. It is true that many BMA algorithms explicitly iterate over the space of models. It is also true the many BMA algorithms (including reversible jump MCMC) require evaluations of the prior density. (We should have been clearer that by "evaluations of p(m|t)" we meant evaluation of the density, rather than sampling m~p(m|t).)

All that this text is trying to say is that standard BMA algorithms all appear to have at least one of these two properties, and thus cannot easily be applied to this problem since the space of models is infinite and the prior cannot be evaluated.

We don't see any factual dispute here. The only issue appears to be the intended meaning of the text. Rest assured, we do not want to create the misleading impression that the posterior in this work is unrelated to BMA, and we accept that this passage could create that impression, even if that is not what we intend. We will revise the text in question to make it absolutely clear that it is only computational issues that make it difficult to apply standard BMA algorithms.

Thanks very much for your review! We particularly appreciate the summary, which is, as far as we can see, accurate in every detail.

Generated models failing to compile is indeed a significant issue. Though, in practice it's not as severe as you might think. We take advantage of continuous batching (parallel inference) when generating models from the LLM which speeds things up enormously. The good news about invalid models is that they are quickly rejected during the compilation step, meaning no computational effort needs to be expended on them during inference. (See section F for some more discussion of this point.)

We have indeed considered using constrained generation. In fact, we initially were generating models using llama.cpp, which supports constrained generation (https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md), and we noticed that Stan has a published BNF language syntax (https://mc-stan.org/docs/reference-manual/syntax.html). The truth is that we ultimately did not to pursue this simply because llama.cpp did not suppose continuous batching and we could not find any LLM software that supported both continuous batching and constrained generation. As continuous batching made generation 1-2 orders of magnitude faster, we could not forego it.

We apologize for being somewhat unclear with the statement in section 5.1 that "These results underline the importance that the LLM-defined prior be relatively broad.”

What we were trying to say here is that the variance in Eq. 9 is determined by the chi-squared divergence between p(m|x,t) and p(m|t). To clarify notation, here p(m|t) is the conditional distribution over models from the LLM given only the model text, while p(m|x,t) would be the conditional distribution when you also condition on the observed data x.

Consider, for example, a case where the LLM is extremely "overconfident" in that it assigns almost all probability mass to "bad" models m where the marginal likelihood p(x|m) is very low. In this case, it is conceivable that the posterior p(m|x,t) ∝ p(m|t) p(x|m) would be more influenced by the marginal likelihood p(x|m) then the LLM prior p(m|t) and so could concentrate in regions where p(m|t) is very low. If that were to happen, the chi-squared divergence between p(m|x,t) and p(m|t) would likely be very large, meaning that estimates from this method would have very high variance.

For this reason, we believe it is important that the LLM be instructed to be quite "open minded" about what a good model might be. This, of course, depends on the base LLM. But one can also try to encourage this by providing a good system prompt and a good set of examples, as we sought to do.

Imposing syntactic validity constraints would not have much impact on this issue. Since we reject all invalid models, the only impact of imposing those constraints would be to improve the fraction of valid models. Essentially, if you find 1000 syntactically valid models by generating and rejecting or imposing validity constraints, then the resulting accuracy would be the same.

Ultimately, we suspect that general progress in LLMs will ultimately improve the fraction of valid models, as it has done in the generation of other kinds of code (e.g. python code), enough that this would become a fairly minor issue. We also believe there is great opportunity for better inference methods that would make it practical to address more complex problems (see lines 304-307). We hope that this paper establishes the basic idea of defining posteriors directly from informal user text, and that with future progress in inference methods it will be practical to address large-scale problems.