General Response

---

All reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on six datasets based on real-world data, with promising results that we detail next:

Known Graph: First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the .zip file in the supplementary material attached to the original submission for details):

Median MSE (CATE) - Known Graph

We find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.

RealCause: We also evaluate on four new datasets from the RealCause benchmark [5], using a version of Do-PFN (which we call v1.1) pre-trained on larger graphs with up to 60 nodes.

We observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. Due to space constraints we refer the Table “Median MSE (CATE) - RealCause” in our general response to reviewer Fczh for the precise MSE values.

These results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data.

Response to Reviewer Ww3g

---

Thank you for your feedback and interesting questions!

Paper lacks a clear understanding of where and to what extent uncertainty arises in the model’s predictions.

We thank you for encouraging us to further develop our theory regarding uncertainty!

The posterior predictive distribution that Do-PFN approximates can be factorized as follows: $p(y|x, do(t), D) = \int p(y|x, do(t), \psi) p(\psi|D) d\psi$, where $y$ denotes the outcome, $x$ a covariate vector, $\psi$ an SCM, and $D$ an observational dataset.

Let’s also consider the observational equivalence relation $\psi_1 \sim \psi_2 \iff p(y,x,t|\psi_1) = p(y,x,t|\psi_2)$, where a selector $r(\psi)$ maps any SCM $\psi$ onto the representative of its equivalence class $[\psi]$. Subsequently, this yields:

$p(y|x, do(t), D) = \int \int p(y|x, do(t), \psi) p(\psi|r) p(r|D) dr d\psi$

This factorization allows to distinguish three types of uncertainty:

• First, $p(y|x, do(t), \psi)$ represents the uncertainty in the conditional interventional distribution of the outcome $y$ assuming a fixed SCM $\psi$; all uncertainty about $y$ in $p(y|x, do(t), \psi)$ is caused by the noise distributions in the SCM $\psi$ and thus represents a form of aleatoric uncertainty.

• Second, $p(\psi|r)$ is the distribution over observationally equivalent SCMs assuming we are in equivalence class $[r]$, and thus represents uncertainty due to potential unidentifiability of $\psi$ from observational data.

• Finally, the distribution $p(r|D)$ quantifies the epistemic uncertainty about the Markov equivalence class of a ground truth SCM caused by a finite observational dataset $D$.

We show, in a newly added section of our paper, that for infinite observational data the uncertainty in $p(r|D)$ vanishes completely, but the other sources of uncertainty remain. More specifically, when the observational data $D$ comes from a ground-truth SCM $\psi_0$, the posterior conditional interventional distribution $p(y|x, do(t), D)$ converges to the posterior $p(y|x,do(t),[\psi_0])$ where $[\psi_0]$ denotes the set of all observationally equivalent SCMs.

More formally:

$p(y|x, do(t),D) \rightarrow p(y|x,do(t),[\psi_0]) = \int p(y|x, do(t), \psi) p(\psi|[\psi_0]) d\psi, \quad \text{for}\ |D| \rightarrow \infty.$

We formulate all those arguments in a rigorous way using measure theory and Doob’s theorem in a new theoretical section in the appendix of our manuscript.

Why is it the case that uncertainty is under-confident for theoretically identifiable cases?

This is a very interesting question!

As we show in Proposition 1, we minimize the forward KL-divergence $D_{KL}[p(y|x, do(t), D)||q(y|x,do(t), D)]$ between the posterior $p(y|x, do(t), D)$ and the model $q(y|x,do(t), D)$. This leads to $q$ being mass covering, i.e. overdispersed—especially when $q$ is a suboptimal approximation of $p$ [2,3]. We parametrize $q$ as a bar distribution, with fixed positions of the bars following TabPFN [4]. In the identifiable case the posterior is more concentrated leading to a slightly larger discrepancy between the posterior $p$ and the model $q$ potentially due to discretization error. This difference manifests itself in Do-PFN being slightly underconfident (please see our calibration curves), rather than overconfident due to the nature of the forward KL-divergence.

Please also note that underconfidence in case of approximation error is typically considered a very desirable property [3].

While I understand that the output of ACTIVA produces the shift in distribution for all variables after performing and intervention, wouldn’t this also inherit the shift in outcome variables distribution which Do-PFN is predicting? I am curious what are the primary differences that makes Do-PFN a stronger approach than ACTIVA, and accordingly where your paper's main source of novelty lies.

Thank you for the opportunity to point out the differences between Do-PFN and the contemporaneous ACTIVA method. We summarize some of the major differences below:

• Task: DoPFN is trained to learn how to predict the effect of a binary intervention on a univariate outcome variable, which is a fundamental yet practically relevant and widely researched task in causal inference with a broad set of available baselines and benchmarks [5]. ACTIVA considers the problem of learning the effect of multi-variable interventions on the multivariate distribution of all variables in an SCM simultaneously.

• Training data: ACTIVA is trained on data from fixed-sized, purely linear additive causal models (without any nonlinearities) and the SERGIO simulator (with 11 nodes) with knockout interventions. Causal sufficiency is assumed, i.e. covariates in an SCM are observed. In contrast, we vary the nonlinearities, graph sizes and noise distributions and also include unobservable variables. Conceptually, it would be possible to marginalize the output distribution of an ACTIVA-model after conditioning on only a single intervention in order to obtain predictions for a single variable. However, this would not bridge the gap between vastly different pre-training setups.

• Model: Do-PFN learns a univariate posterior predictive and employs the same architecture as TabPFN [4], as well as the same histogram-based posterior approximation, which has been found to be highly effective for predictive tasks. ACTIVA learns a multivariate posterior and employs a $\beta$-VAE with Vamp prior plus a mixture-of-Gaussian parametrization of the conditional interventional distributions and is based on transformer encoders and decoders with alternating row- and column attention.

• Evaluation: We compare Do-PFN against established methods for causal effect estimation besides observational baselines. ACTIVA is evaluated in terms of similarity of the predicted distribution compared to the ground truth interventional distribution (for instance evaluated in terms of MMD); we investigate the predictive performance of Do-PFN on specific causal graphs and semi-synthetic data.

This work considers only the class of single-node binary interventions, which is restrictive in practice. A consideration of multi-node continuous interventions would be much stronger.

We focus on this specific task since it is one of the most researched problems in causal inference, see e.g. [5]. We believe that, due to its relevance, this warrants studies that fully evaluate the performance of different methods on synthetic and semi-real benchmarks, as well as comparisons against numerous existing baselines. It’s a good starting point.

References

[1] Neal et al. "Realcause: Realistic Causal Inference Benchmarking." arXiv 2020.

[2] Murphy, Kevin P. "Probabilistic machine learning: Advanced topics." MIT press, 2023.

[3] McNamara, et al. "Sequential monte carlo for inclusive kl minimization in amortized variational inference." AISTATS 2024.

[4] Hollmann, Noah, et al. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." NeurIPS 2022.

[5] Yao, Liuyi, et al. "A survey on causal inference." ACM Transactions on Knowledge Discovery from Data (TKDD) 15.5 (2021): 1-46.

General Response

---

All reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on six datasets based on real-world data, with promising results that we detail next:

Known Graph: First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the .zip file in the supplementary material attached to the original submission for details):

Median MSE (CATE) - Known Graph

We find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.

RealCause: We also evaluate on four new datasets from the RealCause benchmark [5], using a version of Do-PFN (which we call v1.1) pre-trained on larger graphs with up to 60 nodes:

Median MSE (CATE) - RealCause

We observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. These results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data.

Response to Reviewer bczT

---

We thank you for your positive feedback and very meaningful questions!

How sensitive is Do-PFN to mismatches between the synthetic prior used during training and the test-time SCMs?

To address this, in addition to the strong real-world performance we observe above, we also perform several ablation studies regarding distribution mismatch. We also refer to our response to reviewer Fczh, where we provide OOD results in terms of graph size, and a theoretical argument regarding prior mismatch.

Our new out-of-distribution analysis shows the performance of Do-PFN on in-distribution samples from its prior, compared to out-of-distribution samples in terms of 1) graph size, 2) non-linearity type, and 3) noise distributions.

Median MSE - Out of Distribution

We find that Do-PFN is most robust to changes in the noise distribution and with respect to post-non-linearities, with its MSE in CID prediction changing by a magnitude of less than 0.001. Interestingly in some cases, such as the “ELU Non-Linearity” ablation, going out of distribution results in an increase in performance, possibly since ELU shares functional similarities to the ReLU activation which is represented in our prior.

How does the training time and inference speed compare to existing causal inference methods in settings with larger datasets or real-time requirements?

We thank you for the opportunity to discuss the relationship between training- and inference time of Do-PFN. Do-PFN is a foundation model of 7.3 million parameters (small compared to current LLMs, but large, compared to traditional causality methods). It is pre-trained for up to 48 hours, but its pre-training time is incurred once up-front. When applying DoPFN to a new dataset, predictions can be obtained in a single forward pass.

We now provide inference time results, calculating the median time in seconds of the .fit and .predict functions of Do-PFN and our baselines. We also provide versions of several baselines with HPO for 60 seconds per underlying base model, and run deep-learning-based methods on an RTX-2080 GPU.

Median Train/Optimize + Inference Time (s) - RealCause

In our runtime analysis, Do-PFN (run on CPU and GPU) has a faster inference speed than all other baselines. Do-PFN outperforms other baselines because there is no conventional model-fitting; and is slightly faster than the TabPFN meta-learning approaches (when both methods use a GPU), since Do-PFN has less parameters and does not do ensembling.

While the paper claims Do-PFN learns to “implicitly” adjust for confounding (e.g., back-door/front-door), it would benefit from deeper analysis.

While we empirically find that Do-PFN works very well across diverse causal challenges, on both carefully controlled synthetic setups (including front-door and back-door cases) as well as real world data, the precise inner workings of PFNs in general, and thus also Do-PFN, are rather unclear. We consider it unlikely that Do-PFN works analogously to classical multi-step approaches that would, e.g., perform causal discovery, then determine estimands, and finally carry out the appropriate adjustment, but the precise in-context-learning dynamics of Do-PFN remain to be explored.

In the field of LLMs, mechanistic interpretability tools have become popular and their applications to PFNs, and Do-PFN in particular, might allow to systematically trace specific procedures or algorithms Do-PFN utilizes internally. Based on your question, we will expand on this as a key area of future work in the discussion section.

References

[1] Neal et al. "Realcause: Realistic Causal Inference Benchmarking." arXiv 2020.

[2] Bereska, Leonard, and Stratis Gavves. "Mechanistic Interpretability for AI Safety-A Review." TMLR 2024.

General Response

---

All reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on six new datasets based on real-world data, with promising results that we detail next:

Known Graph: First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the supplementary material attached to the original submission as a .zip file for details). We also evaluate against six new baselines.

Median MSE (CATE) - Known Graph

We find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.

RealCause: Furthermore, we have pre-trained a version of Do-PFN (which we call v1.1) on larger graphs with up to 60 nodes, in order to evaluate on datasets from the RealCause benchmark [5].

Median MSE (CATE) - RealCause

We observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN.

These results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data.

Response to Reviewer Fczh

---

Thank you for your constructive feedback!

There is no systematic comparison with widely used causal-inference tools (e.g., T-learner, IPW, CFRNet, TARNet) on the same synthetic tasks.

In response, we have evaluated Do-PFN on our synthetic case studies against a strong set of meta-learning and double machine learning (DML) baselines with TabPFN (v2) as a base learner, following [1], who use only meta-learners and [2], who show TabPFN to be a strong base learner for meta-learning CATE estimators. We also include TARNet [3] and DragonNet [4] implemented in the CATENets library.

Median MSE (CATE) - Synthetic

Across our six synthetic case studies, we find that Do-PFN achieves the best performance on all cases except for “Observed Mediator” in which the two-model T- and X-learner strategies are best able to capture this highly binary treatment effect, due to the treatment being the root cause.

Can you quantify Do-PFN’s robustness when test data come from distributions not represented in your synthetic prior [..]?

Robustness to prior mismatch is an important topic for PFNs in general, but no theoretical results of the form suggested by the reviewer exist (also not for the non-causal TabPFN - this is a very hard problem). However, in addition to the strong real-world performance we observe above, we also perform several ablation studies regarding distribution mismatch where we vary core characteristics of the SCM-generated data Do-PFN is evaluated on.

Median MSE - Out of Distribution

We find that Do-PFN’s performance is robust to different noise distributions, and using post-non-linearities (instead of adding the noise after applying a nonlinearity). Using different nonlinearities can even improve performance while increasing the variance of the Gaussian noise drastically hurts performance.

While not a general form of Proposition 1, one can make the following argument regarding the mismatch between training and testing data. For the posterior $p(y|x, do(t), D)$ denote $x = (x,t, D)$, such that under abuse of notation $p(y|x, do(t), D) = p(y|x)$. Denote Do-PFN’s error as a function of $p$ as $E_{p(x)} D_{KL}(p(y|x)||q(y|x)) =: e(p)$. Assume that $e \in F$ for some function class $F$. Now let the testing data come from a different distribution $p’$ and write $e’(p) = E_{p’(x)} D_{KL}(p(y|x)||q(y|x))$. By definition of the integral probability metric $d_F$ wrt. $F$, we get: $|e(p) - e’(p)| \leq d_F(p, p’)$. In case $F$ is the space of 1-Lipschitz functions, $d_F(p, p’) = W_1$ by the dual representation of the 1-Wasserstein distance $W_1$.

Intuitively, when the loss of Do-PFN is well behaved and the real-world data distribution $p’$ has a difference of $C$ to the synthetic training distribution $p$, the same constant $C$ bounds how much the loss increases when taking inputs from $p’$ instead of $p$.

Having said that, while the (surprisingly) strong generalization of PFNs from synthetic training data to real-world evaluation datasets has been extensively empirically confirmed, the theory behind this phenomenon is still developing [6].

It remains unclear how the method performs on high-dimensional datasets.

We refer to a new experiment, in which we sample 500 synthetic datasets from our prior with graphs of up to 50 nodes.

Median MSE (CID) - OOD Graph Size

For larger graphs out of its prior-distribution, we observe a significant reduction in Do-PFN-v1’s performance, which decreases as the graph size goes further out of distribution. Do-PFN (v1.1), however, which was trained on graph sizes up to 60 nodes, achieves significantly better performance on large graphs with 21-50 nodes. This provides evidence towards Do-PFN’s ability to scale up to more complex and high-dimensional problems.

References

[1] Vanderschueren et al. "AutoCATE: End-to-End, Automated Treatment Effect Estimation." ICML 2025.

[2] Zhang et al. "TabPFN: One Model to Rule Them All?." arXiv 2025.

[3] Shalit et al. "Estimating individual treatment effect: generalization bounds and algorithms." ICML 2017.

[4] Shi et al. "Adapting neural networks for the estimation of treatment effects." NeurIPS 2019.

[5] Neal et al. "Realcause: Realistic causal inference benchmarking." arXiv 2020.

[6] Nagler. "Statistical foundations of prior-data fitted networks." ICML 2023.

General Response

---

All reviewers acknowledge that our paper presents a promising approach, but also raise the question of prior mismatch, especially when analyzing real-world data. While we believe it is crucial for a new method to study synthetic data with known ground truth, we agree that analyzing real world data is important. Therefore, we have meanwhile performed an analysis of Do-PFN's performance on six datasets based on real-world data, with promising results that we detail next:

Known Graph: First, we evaluate on the Amazon Sales and Law School Admissions datasets where the causal graph is known (please refer to the .zip file in the supplementary material attached to the original submission for details):

Median MSE (CATE) - Known Graph

We find that Do-PFN (v1) achieves the strongest performance on Law School and is not significantly worse than the best method on Amazon Sales, a dataset with a complex mediation structure.

RealCause: We also evaluate on four new datasets from the RealCause benchmark [5], using a version of Do-PFN (which we call v1.1) pre-trained on larger graphs with up to 60 nodes.

We observe that Do-PFN-v1.1’s extended pre-training leads to strong performance among competitive baselines on three out of four RealCause datasets. We note that these datasets perfectly satisfy unconfoundedness, giving our baselines a fundamental advantage against Do-PFN. Due to space constraints we refer the Table “Median MSE (CATE) - RealCause” in our general response to reviewer Fczh for the precise MSE values.

These results show Do-PFN to nevertheless be competitive with a strong set of CATE baselines, suggesting the strong transfer of Do-PFN's synthetic pre-training to complex, real-world data.

Response to Reviewer HtXo

---

Thank you very much for your feedback and comments!

Possibility of including more variables (e.g. number of confounders in your experiment)?

We are happy to provide more high-dimensional synthetic cases! We refer to a new experiment in our out-of-distribution (OOD) analysis, in which we sample 500 synthetic datasets from our prior with graphs of up to 50-nodes.

Median MSE (CID) - OOD Graph Size

For larger graphs out of its prior-distribution, we observe a significant reduction in Do-PFN-v1’s performance, which decreases as the graph size goes further out of distribution. Do-PFN (v1.1), however, which was trained on graphs up to 60 nodes, achieves significantly better performance on large graphs with 21-50 nodes. This provides evidence towards Do-PFN’s ability to scale up to more complex and high-dimensional problems.

The computational efficiency appears questionable, deploying a 7.3 million parameter model for datasets with only 5 variables raises concerns about scalability and whether such model complexity is warranted for these problem sizes.

First, we show, via the performance of Do-PFN (v1.1), that using the same architecture (and number of parameters) scales up to achieve strong performance on more complex tasks, as shown in the General Response provided above.

Median Train/Optimize + Inference Time (s) - RealCause

Second, in our runtime analysis, Do-PFN (run on CPU and GPU) has a faster inference speed than all other baselines. Do-PFN outperforms other baselines because there is no conventional model-fitting, and is slightly faster than the TabPFN meta-learning approaches (when both methods use a GPU), since Do-PFN has less parameters and does not do ensembling.

[...] why does the model exhibit underconfidence in identifiable cases?

This is a very interesting question, which we think can be best explained by two factors:

First, the training objective for Do-PFN minimizes a forward KL divergence $D_{KL}[P||Q]$ between the true posterior $P$ and Do-PFN’s approximation $Q$ (refer to Proposition 1 in our paper and [6]). This encourages $Q$ to be mass-covering when it cannot approximate $P$ perfectly, meaning $Q$ will be overdispersed in cases where there is a difference between $Q$ and $P$ [2,3].

Second, in the identifiable case, the posterior $P$ will be more concentrated (no need to account for unidentifiability), which means that our TabPFN-like approximation of the posterior, based on a bar-distribution [4], inevitably incurs more discretization error than in the unidentifiable case. This causes a larger KL-divergence relative to the unidentifiable case and thus, by the first point, leads to a slightly more conservative estimated distribution $Q$.

We also note that underconfidence in case of approximation error is typically considered a very desirable property [3].

[...] why it performs well specifically on small datasets

While performance on small datasets is a quality inherent to basically all PFNs and has been extensively empirically confirmed [4,5], from a theoretical perspective, this hasn’t been extensively explored. We believe that the best explanation is that PFNs are trained across millions of samples of small datasets, to minimize the CE loss on the test split of the dataset, which directly penalizes overfitting.

Can you run experiment[s] estimating ATE and compare with the same baselines?

We have evaluated Do-PFN and an extended set of baselines (please see our first response to reviewer Fczh for their descriptions) on the task of ATE estimation.

Relative Error (ATE) - Synthetic

We find Do-PFN to have the best performance in 5/6 case-studies, except for in the “Observed Mediator” case, in which the T and X-Learner baselines (with TabPFN v2 as a base model) is best suited to encapsulate the binary nature of this case study due to its two-model approach.

References

[1] Neal et al. "Realcause: Realistic Causal Inference Benchmarking." arXiv 2020.

[2] Murphy. "Probabilistic Machine Learning: Advanced topics. MIT press." 2023.

[3] McNamara, et al. "Sequential Monte Carlo for Inclusive KL Minimization in Amortized Variational Inference." AISTATS 2024.

[4] Hollmann, Noah, et al. "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second." NeurIPS 2022.

[5] McElfresh, et al. "When Do Neural Nets Outperform Boosted Trees on Tabular Data?." NeurIPS 2023.