Effortless, Simulation-Efficient Bayesian Inference using Tabular Foundation Models

We thank the reviewer for their overall positive evaluation of our manuscript – finding that our “paper reads very well” and acknowledging that the use of foundation models in SBI “tackles an idea that has been gaining more and more traction”.

Q1. Clear explanation of final posterior sampling.

We appreciate the suggestion and will provide a more thorough explanation in the updated version. In particular, we will add detailed pseudocode to the appendix to provide a clear and rigorous overview of our TSNPE-PF approach.

Q2. Optimal autoregressive factorization.

We discuss the question of whether an optimal order of the parameter dimension for TabPFN exists in the Appendix Section. D.6, Fig. D-7. Indeed, TabPFN is not equivariant to the ordering of the inputs. In general, given a different ordering, each one-dimensional conditional distribution, which TabPFN has to predict, changes. We conduct experiments for two synthetic tasks and five benchmark tasks, which are not permutation equivariant. Specifically, we randomly permute the parameter dimensions and evaluate the performance. Our results indicate that NPE-PF is relatively insensitive to the order in which parameter dimensions are permuted.

We additionally tested this on our 31‑dimensional pyloric network example: Here the total number of possible permutations is greater than $10^{33}$. Finding the optimal permutation in this space is highly challenging. To gain at least some insight, we performed an ablation study on 50 distinct random permutations using the context set identified by the final‑round posterior. We compute the presented metrics with 1000 simulations. We found that the energy score varies between 0.081 - 0.094 and the valid rates between 96% - 98% (5% and 95% quantile each). This is in line with what we found for the default order reported in the paper, Appendix D.6, Fig. D-7. However, these results do not prove the absence of an optimal permutation that would significantly improve performance. Finding such a permutation is likely to be very challenging in general and would present an interesting opportunity for future research.

We thank the reviewer for their overall positive evaluation ("very cool idea," "largely convincing") and relevant suggestions that led us to add new tasks, a full cost analysis, and an NPE hyper-parameter sweep, which significantly strengthened our work.

We first address the main issues and discuss new results (A1–A3), then give detailed replies to questions (Q1–Q9).

• Benchmark limitations: We use an established benchmark for SBI, but now add new tasks and metrics with more test points for broader coverage (A1).
• Computational cost: We acknowledge that our discussion was imprecise. We will update the manuscript to clearly write that the inference time of NPE-PF is orders of magnitude slower than that of NPE, and will add a careful discussion of our cost analysis (A2).
• Embedding nets: We do not claim NPE-PF outperforms NPE when embedding networks are needed; we merely explore this practical/frequent use-case (A3).

A1. Additional tasks and datasets.

We agree with the reviewer's assessment that the "used standard (low-dimensional) benchmarks" with "only 10 test data sets" have limitations. However, as the reviewer notes, this SBI benchmark is still widely used and tasks are not "easy to fit" for NPE methods (e.g., SLCP or Lotka-Volterra). Nonetheless, we extended our current evaluation suite with several new tasks from other SBI papers [1,2], also including particularly computationally demanding physics simulations ("streams", 30 min. per simulation, 2 dim parameters, 199 dim data). For the new tasks, we compute the average negative log likelihood (NLL) on 1000 test points. If the tasks provide a ground truth posterior, we use 100 test points in the C2ST evaluation.

On these new tasks, NPE-PF again shows strong performance, especially for the smaller simulation budgets, where it is superior to NPE for all tasks. For the detailed results, we refer to the response to reviewer Vbbm, Table V1/V2.

A2. Analysis of computational cost.

We ran additional timing experiments. As expected, inference time mostly scales with an increasing parameter dimension and a larger number of simulations. See Table w2 in the response to reviewer wFAY for details.

Providing a general answer to the question of when "would NPE become overall faster again" is hard because it is task and application dependent. Disregarding simulation time, NPE-PF quickly becomes less efficient than NPE: For example, assuming 100000 available simulations (8 dim parameters, 8 dim data), NPE takes around 60–90 minutes to train. After training, inference will be effectively instantaneous. In contrast, generating 10000 posterior samples for a single observation using NPE-PF takes ca. 50 seconds (Table w2, response wFAY). Therefore, the training time of NPE amortises after evaluating around 90 observations. However, for expensive simulators (e.g. stellar streams), simulation becomes the bottleneck and the simulation efficiency of NPE-PF compensates for the slower inference time. For sequential inference, the situation favours TSNPE-PF, because everything needs to be rerun per single observation.

We will add a thorough discussion of the computational times to the manuscript. Our goal with NPE-PF is not to replace NPE, but to provide an easy-to-use and simulation-efficient alternative, highlighting strengths and limitations such that users can make an informed choice.

A3. NPE-PF with summary (embedding) networks.

We thank the reviewer for raising this point and agree that it requires clarification:

Our main goal with NPE-PF is to introduce an easy-to-use method that allows users to obtain good results with few simulations, given a sufficiently low-dimensional inference problem (<500dim). In high-dimensional cases that need an embedding net, the number of required simulations typically also grows large, and NPE is expected to be superior. Thus, NPE (with embedding net) and NPE-PF serve different use cases.

To further clarify, we do not claim that NPE-PF will be superior to NPE when embedding nets are needed, nor that our approach for pretraining them is the best. Our goal was simply to investigate how NPE-PF performs together with an embedding network or generally a neural representation of data. For more common datatypes, networks to compress data can be already available. The question of how much "additional training and simulation budget" is needed to obtain "good" neural representations is independent of NPE-PF and beyond the scope of this work. For related questions including "how good the 'sufficient statistics' approximation" is, see [3,4].

Finally, even when simulators are high-dimensional, a set of well-engineered, low-dimensional summary statistics is still routinely used in many SBI applications. Such cases include expensive simulators, where training an embedding net is infeasible, or misspecified simulators, where summary statistics provide increased robustness. In these cases, NPE-PF provides a powerful alternative to traditional approaches, even in high-dimensional settings.

Q1. Inference time of real-world tasks.

For our experiment on the recordings from the Allen cell types database (Sec. 3.3), the inference times are as follows:

• TSNPE-PF, 1000 sim.: ca. 10 min, 10000 sim.: ca. 60 min.
• TSNPE, 1000 sim.: ca. 4 min, 10000 sim.: ca. 40 min.

Compared to TSNPE with 10000 simulations, our approach achieves comparable inference quality using only 10% of the budget.

For the Pyloric simulator (Sec. 3.4), inference takes 2-7 minutes per round (including simulation). See Appendix C.4 for an extended discussion and inference time comparison to baseline methods. Note that SNVI baseline needs 30 hours runtime on CPU, whereas the SUNLE baseline requires 3 hours on GPU to acquire a limited set of samples (excluding the time to simulate the much larger initial dataset).

Q2. Inference time of density evaluation.

Autoregressive density evaluation is as fast as sampling. For the exact timings, please refer to Table w2, wFAY. The ratio-based density evaluation is much faster than the autoregressive one. See a detailed analysis in Appendix D.4, Fig. D-5c. Both ways of evaluating densities with TabPFN are slower than a flow and we will make sure to state this more clearly in the manuscript.

Q4. Sweep for hyperparameters.

We agree that the performance of NPE on individual tasks can likely be increased, provided users have enough expertise and computational resources. Therefore, we have conducted an extensive computational experiment, where flow and optimization hyperparameters are swept for each task and simulation budget. Sweeping results are in Table B1 and details in caption. Even for the smaller budgets, this procedure required a substantial amount of compute, taking several hours per setting.

Sweeps give only modest gains—mainly for small budgets where exhaustive search is feasible—and demand far more time or better search for larger budgets. Tuned NPE still trails NPE‑PF on most low‑budget tasks. Importantly, NPE‑PF’s edge is its strong default performance: a better “perfect NPE network" certainly exists, but finding it with few simulations is challenging or basically impossible, and not "effortless".

Finally, this strong default performance becomes crucial for sequential inference, where sweeping is considerably more challenging and costly (long inference times, changing training conditions over rounds, difficulty of selecting appropriate metrics to optimize).

Table B1 Sweep results. C2ST averages across 5 seeds (SDs negligible, space constraints). Search space: flow type, number of transforms, hidden layer size, learning rate, and batch size. We use random search to explore over 1000 configurations with respect to the best validation loss. We limited the maximum time per setting to 10 hours.

Q3, 5-9. Clarifications.

• Number of features. Indeed, the dimensionality supported by NPE-PF is the combined dimensionality of observation and parameter space.

• Calibration results. We agree that using difference plots and including the confidence envelope will increase readability, and will update the plot accordingly.

• Misspecification. For the misspecification experiment, C2ST are computed with respect to non-misspecified ground truth posterior, as in [5].

• Other phrasing.

  • With "continuous parameters", we meant infinite-dimensional parameter spaces such as spatially varying parameters.
  • Indeed, we meant "parameters" instead of "data". Thanks for catching this.
  • With M we refer to the size of the datasets used for pretraining TabPFN, and with N we denote the number of simulations to be used with NPE-PF.

We will correct/phrase the above points more clearly.

[1] Hermans et al. "A trust crisis in simulation-based inference? Your posterior approximations can be unfaithful." 2021.
[2] Gloeckler et al. All-in-one simulation-based inference. 2024.
[3] Yanzhi et al. "Neural approximate sufficient statistics for implicit models.", 2020.
[4] Yanzhi et al. "Is learning summary statistics necessary for likelihood-free inference?.", 2023.
[5] Schmitt et al. "Detecting model misspecification in amortized Bayesian inference with neural networks: An extended investigation.", 2024.

We thank the reviewer for acknowledging that “the paper is well-written”, and that “the empirical results are promising and significant”. We also appreciate the recognition that ours is the “first paper to explore ICL approaches in the context of SBI”.

In response to the reviewer’s suggestions, we have:

• Conducted experiments across a broader range of filter sizes (Table w1).
• Added a rigorous computational time comparison (Table w2).
• Introduced several new, more challenging inference tasks to cover a larger range of tasks.

Below, we first address the reviewer’s main concerns and then provide detailed answers to their specific questions (Q1–Q4).

• Technical novelty of contribution: Our paper tackles two central challenges in SBI: the high simulation cost of traditional methods and the expertise required to train conditional density estimators. Our method, NPE-PF, provides a novel and elegant way to alleviate both of these issues. Furthermore, TSNPE-PF uses new ideas for fast in-context density estimation with classifier-based density ratios, and shows strong performance in real-world inference tasks (Fig. 4/5). To our knowledge, our paper hence introduces novel methods to the SBI literature that address problems relevant to the community, regardless of any perceived lack of technical complexity. Instead, we view this simplicity as a strength, lowering the barrier for adoption by researchers and practitioners.
• No in-depth insights into when and why ICL works: The reviewer states that our work “does not provide in-depth insights into when and why the ICL approach works” and asks “Is TabPFN expected to beat NPE in all problems?” While we did not test on all problems, we did test on a considerable number of problems and now further increased this number (results in Table V1/V2 in response to Reviewer Vbbm). All these results support our main claim that NPE-PF outperforms NPE on small simulation budgets, and is competitive for large ones. Yet, we agree that the manuscript could be improved by providing a more elaborate interpretation and conclusion of these results to better communicate the insights into when and where ICL likely works well (see Q1-Q4 for details).

Q1. Ablation on number of neighbours for filter.

We thank the reviewer for their suggestion. We have conducted an experiment on all SBI benchmark tasks using 100k simulations and varying filter sizes (16, 64, 256, 1024, 2048, 4096, 16384) in comparison to our default choice of 10000, which is based on the TabPFN recommended context limit. Results are in Table w1.

Table w1: Ablation on filter size. Average C2ST for each task (across five seeds) when varying the filter size. Standard deviations across seeds are all below 0.01, and are not reported individually. Each on a dataset of 100 000 simulations.

Across all experiments, performance decreases noticeably with small filter sizes. Large filter sizes (4096 and 16384) perform similarly to our default choice of 10,000. Importantly, our default choice is always optimal or near-optimal. These results suggest that the available context (as recommended by TabPFN) should be used fully and that little performance benefit can be gained by moving away from the default choice (see also Fig 2. “NPE-PF (unfiltered)”). Nonetheless, these results also suggest one can obtain good performance with smaller filter sizes to reduce the computational load (> 2048).

Q2. Analysis of computational time.

Beyond the timing plots in Fig 2b, we now perform additional runtime experiments (using an Nvidia A100 GPU) varying the number of simulations, and the dimensionality of parameters and simulations/observations (Table w2). As expected, the inference time mainly depends on the number of simulations in context and the parameter dimension.

In general, an answer to the question “How many hours would a practitioner save?” depends on many different factors:

1. First, we discuss amortized inference in an example setting. With 100000 simulations (8 dim parameters, 8 dim data), NPE typically takes around 60–90 minutes to train. After training, inference will be effectively instantaneous (Fig. 2b). By contrast, generating 10000 posterior samples for a given observation using NPE-PF takes ~50 seconds (Table w2). Therefore, after sampling posteriors for around 90 observations, the training time of NPE amortises. However, this holds only assuming that one ignores the time necessary to obtain the simulations. For very expensive simulators, obtaining simulations becomes the bottleneck and the simulation efficiency of NPE-PF will more than compensate for the slower inference time. Therefore the appropriate choice of method crucially depends on the application and simulator.
2. Second, with sequential inference the situation changes. When using flow-based TSNPE, many networks need to be trained per observation, so fast sampling becomes less relevant. For instance, in the experiment in Section 3.4, both TSNPE and TSNPE-PF require approximately one hour with a budget of 10000 simulations (here, the simulator is fast and the simulation time is negligible). However, TSNPE-PF is much more efficient in terms of simulations, achieving similar performance with just 1000 simulations and an inference time of 10 minutes. This effect would be amplified for a more expensive simulator.

We will add the timing results and above discussion to our manuscript to allow users to make an informed choice between NPE-PF and NPE.

Table w2: NPE-PF runtime (seconds) for different simulation counts and parameter/data dimensionality. Each row represents varying parameter dimensions ($\theta$-dim), each column varying data dimensions (x-dim).

Q3. TabPFN details and insights for SBI.

While NPE-PF beats NPE on all our tasks with a simulation budget of 100, it is certainly not expected that NPE-PF will be superior for all simulation budgets. Indeed, there are several benchmark tasks in which NPE performs better than NPE-PF, particularly for larger simulation budgets. In general, as the number of simulations grows large, NPE is expected to eventually outperform NPE-PF. Our main claim is that NPE-PF is simulation efficient, and performs particularly well when the number of simulations is small – a crucial and underexplored area within SBI.

We agree with the reviewer that identifying the types of simulators on which NPE-PF performs well is interesting. TabPFN is trained using purely synthetic datasets generated from random structural causal models (SCMs), see [1]. In practice, we have found that NPE-PF performs particularly well on 'structured' simulators, which are similar to SCMs in that they have conditional independences. Many real-world simulators exhibit some degree of this structure (e.g. SIR, Hodgkin Huxley, etc.). Explicitly incorporating such independencies into the inference network was already shown to improve simulation efficiency [2]. We hypothesize that the pretraining on SCMs enables TabPFN to automatically detect and leverage these independencies directly from the data. A detailed investigation into these mechanisms is interesting future work, but beyond the scope of this study.

In addition, we agree with the reviewer that certain formulations are misleading (i.e., “complex parameter spaces—such as spatially varying parameters” can simply be replaced with “high-dimensional”). We will make sure to increase the clarity of these statements and include the previous discussion.

Q4. TabICL and other foundation models.

We agree with the reviewer that using other foundation models within our approach would be highly valuable. Unfortunately, the TabICL model currently only provides in-context classification and therefore cannot be used for autoregressive sampling from the posterior, for which we use the TabPFN regression model. However, as the reviewer suggests, our proposed method is, in principle, independent of the foundation model used.

[1] Hollmann et al. Accurate predictions on small data with a tabular foundation model. Nature, 2025.
[2] Gloeckler et al. All-in-one simulation-based inference. ICML, 2024.

We thank the reviewer for finding our work “well‑organized,” for noting that our “methodology […] attends to many important practical considerations,” and for deeming our “performance evaluation […] convincing.”

Below, we first address the reviewer’s main concerns and provide a more detailed analysis of the new results developed to address these issues (A1, A2, A3). We then offer detailed responses to the reviewer’s specific questions (Q1, Q2).

• Inference tasks on a wider range of scientific disciplines: We agree that extending our evaluation suite to include simulators from additional fields (e.g., physics) is valuable. We have therefore added several new tasks. In A1, we analyze these new tasks in depth and will integrate the expanded results into the revised manuscript.
• Principled analysis of NPE‑PF’s diminishing advantages: Formulating a general proposition in this regard is challenging, because it is very specific to the task and/or the application at hand. But, we believe that our (now even more) comprehensive empirical investigation of NPE-PF’s performance in various situations (tasks, dimensionalities, simulation budgets, etc.) constitutes a principled empirical examination of NPE-PF’s advantages (A2 for details). We will improve the discussion on this topic within the manuscript (e.g. more concrete statements, when we expect it to work well).
• Computational costs: We agree that a more precise discussion of runtime trade-offs is needed. In A3, we present an ablation study examining the impact of context size and data and parameter dimensionality.

A1. Additional inference tasks

The new tasks include simulators from physics (Weinberg, Stellar Streams), computer science (M/G/1 queue) [1], as well as “structured” synthetic tasks (Tree, HMM) [2]. Out of these, the stellar streams simulator stands out as particularly computationally demanding (>30 min. per simulation per CPU core; 2dim parameters, 199dim data). The tasks in [1] do not come with a ground truth posterior. We therefore evaluate them only via average log posterior density (see Table V1 for NNL metric) as well as calibration error (not shown here due to space constraints). We also extended the HMM task to 50 parameter and data dimensions as an additional test for high-dimensional, but “structured” parameter spaces (see Table V2 for C2ST metrics). Overall, our results for the new tasks are consistent with our previous findings. Across all tasks, NPE-PF significantly outperforms NPE at lower simulation budgets (Table V1/V2). Furthermore, NPE-PFperforms well on larger simulation budgets e.g. Tree, HMM or Streams (Table V1/V2). We will include these results in an updated version of the manuscript.

Table V1: Extended task benchmark. Average negative log posterior density of the true parameters on 1000 test simulations. Each run was repeated five times and we report the average ± SD over runs. The better result is marked in bold.

Table V2: Extended C2ST evaluation. Average C2ST (± SD over 5 runs) on new synthetic tasks using 100 random observations.

A2. Analysis of NPE-PF's diminishing advantages.

First, the question at which “upfront sample size is TabPFN only minimally comparatively informative (or even a burden)” is, to a certain degree, task-specific. NPE can outperform NPE-PF after $10^3$ simulations (e.g. Gauss. Mixture) or require more than $10^5$ simulations (e.g. Lotka Volterra). We evaluate NPE-PFs performance on a variety of tasks with different dimensionalities to give insights into its behaviour under various conditions.

Second, whether NPE-PF becomes a burden also depends on the use case or type of inference task. If inference needs to be performed on thousands of observations, more time should be allocated to simulation and NPE should be chosen. If inference on only a few observations (or many different prior/simulator configurations) is required and/or the simulator is expensive, then NPE-PF is a great alternative. The stellar streams simulator is a good example of this. Attaining 100 simulations is feasible on a consumer-grade CPU (10 cores) in 5 hours, and NPE-PF achieves a comparable performance to NPE with 10,000 simulations, which would require 20 days (Table V1). In other words, NPE and NPE-PF serve different purposes.

Third, we agree that NPE performance can be improved with a “well-selected embedding network”. However, training such a network requires a large amount of simulations. We discuss this situation in Appendix D.3. Specifically, in Fig D-4c, we compare NPE-PF with NPE using a tailored 1D CNN embedding net (Lotka Voltera [long] task with a 300 dim. time series), where NPE-PF still outperforms NPE.

In summary, we believe that our experiments, both new and previous, conducted in various settings, constitute a "principled analysis of NPE-PF's diminishing advantages over standard NPE methods". Furthermore, we have already addressed several of the aforementioned “practical considerations” in the Appendix. However, we acknowledge that we did not communicate some of these aspects effectively throughout our manuscript and will address this in an updated version.

A3. Analysis of computational cost.

We agree with the reviewer that providing a comprehensive evaluation of the overall runtime is important. For this reason, we have included an overview in the main manuscript (Fig. 2b). However, we also agree that this is insufficient for assessing the "boundary beyond which one might consider a computationally faster method". To address this, we have now added a comprehensive runtime ablation section on data dimensionality and simulation size for NPE-PF.

For results on runtime across different scenarios we refer to the response to Reviewer wFAY Table w2. To summarize the runtime analysis for NPE-PF:

• The compute time grows with an increasing number of simulations and parameter dimensions, as expected.
• In lower dimensional cases the compute time to acquire a 10k posterior samples ranges between 4-23 seconds across all context sizes.
• In higher dimensional cases the compute time can grow to 1-2 minutes for context-sizes of 100/1000 and up to 8 minutes for 10000.

While these inference times are substantially slower than standard NPE, the choice between both methods requires careful consideration of various factors, as discussed in A1.

Q1. Inheriting TabPFN properties.

We appreciate the reviewer’s suggestion to "characterise when, or if, NPE-PF struggles systematically". Indeed, one of our original objectives was to identify configurations that might present a challenge to NPE-PF. To this end, Section D.5 examines performance under extreme, heavy-tailed noise models. Across all noise scenarios that were tested, NPE-PF proved to be remarkably robust.

Although we did not identify any systematic failure modes, we found that NPE-PF performs particularly well on simulators with an underlying graphical structure and conditional dependencies. We attribute this to the fact that TabPFN was pre-trained using random structural causal models (SCMs). See the response to Reviewer wFAY (Q3) for a more detailed discussion.

Q2. Parameter permutations in 31D example.

Testing the reliability of NPE-PF in high-dimensional spaces is inherently challenging. For the 31dim pyloric network, the total number of possible permutations of parameter dimensions is greater than $10^{33}$. Moreover, evaluating each candidate permutation requires expensive simulation, making exhaustive exploration impractical. To gain insight despite these constraints, we performed an ablation study on 50 distinct random permutations using the context set identified by the final‑round posterior. We compute the presented metrics with 1000 simulations. We found that the energy score varies between 0.081 - 0.094 and the valid rates between 96% - 98% (each the 5% and 95% quantile). This is in line with what we found using the default permutation in the paper (Appendix D.6, Fig. D-7). We note that these results must be interpreted with caution. There may be permutations for which NPE-PF performs significantly better or worse. However, identifying these permutations is very difficult due to the large number of possible permutations.

[1] Hermans et al. "A trust crisis in simulation-based inference? Your posterior approximations can be unfaithful", 2021
[2] Gloeckler et al. "All-in-one simulation-based inference", 2024