A Meta-Learning Approach to Bayesian Causal Discovery

We thank the reviewer for their comments that "The paper is well-written and clear and includes a range of well-designed experiments." We address your questions point by point below.

Learning a distribution over causal mechanisms is crucial for many applications, such as causal inference and treatment effect estimation. Could the authors elaborate on potential applications of the proposed approach beyond the discovery of the causal graph?

Estimating causal effects is an important downstream application of causal discovery. In our case, causal effects can be estimated by sampling DAGs and then performing do-calculus using a chosen estimator. In fact, this is exactly the procedure to infer causal effects for all the methods in the paper you have cited [1] (Figure 1 and section 3.2 in [1]). Plenty of the methods tested in [1] also only infer the causal structure - Bootstrap-PC and DAG-GFlowNet to name a few. Furthermore, a lot of other (non-Bayesian) causal discovery methods also only infer the causal graph [2, 3, 4].

[1] Emezue, C. C., et al. (2023). Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation. [2] Rolland, Paul, et al. "Score matching enables causal discovery of nonlinear additive noise models." International Conference on Machine Learning. PMLR, 2022. [3] Spirtes, Peter. "An anytime algorithm for causal inference." International Workshop on Artificial Intelligence and Statistics. PMLR, 2001. [4] Bühlmann, Peter, Jonas Peters, and Jan Ernest. "CAM: Causal additive models, high-dimensional order search and penalized regression." The Annals of Statistics 42.6 (2014).

For instance, the representation of the posterior distribution as permutation and upper triangular matrices is also similar to [3]

We invite the reviewer to refer to the general comment for our summary of our main contribution.

It is true that plenty of methods use the permutation and upper triangular construction (discussed in L133), however the difference is the method used for inference. Whereas previous methods use sampling or variational inference with explicit Bayesian models, we use meta-learning to infer the posterior.

In contrast with other meta-learning approaches (which only parametrised a single graph), we target the full Bayesian posterior and hence include important properties (in Table 1). The permutation and upper triangular construction provides us with these properties so was included in the decoder construction (see L264) along with a novel loss to infer the posterior using meta-learning (section 3.3).

An important contribution of our work is that, with our changes, meta-learning approaches can better approximate the posterior over causal structure than explicit Bayesian models. This can especially be seen in Table 2, where with the same training data as our model (BCNP), previous approaches AVICI and CSIvA do not outperform the explicit Bayesian models (DiBS and BayesDAG), whereas our model (BCNP) does.

...using CPDAG E-SHD (as used in previous papers such as in BayesDAG) might be more reflective of the performance

The CPDAG is useful when the underlying causal model is known to be not identifiable (which is the case in sec. 6.1 of BayesDAG which is the only section where CPDAG E-SHD is used). The causal models that we generate data from have been previously used for testing discovery within a Markov equivalence class (functional mechanisms are similar to those used in [1, 2]). Hence, we believe that E-SHD is more appropriate.

[1] Goudet, Olivier, et al. "Learning functional causal models with generative neural networks." Explainable and interpretable models in computer vision and machine learning (2018).

[2] Dhir, Anish, Samuel Power, and Mark van der Wilk. "Bivariate Causal Discovery using Bayesian Model Selection." Forty-first International Conference on Machine Learning.

Do you have an intuition about how the proposed model performs in these [low data] particular settings?

The sample size of the training datasets can easily be changed to better reflect low data settings (see Appendix B.2). The performance of meta-learning models depends on the number of datasets used for training. As the training data is synthetically generated, any number of training datasets can be generated. Thus, we would still expect performance gains over other explicit Bayesian models.

We thank the reviewer for their comments that "The proposed model is shown to be competitive on synthetic and simulated datasets" and "The paper is written clearly and flows well". We answer your questions below.

The framework is not new and the only contributions I see are a change in how permutation invariance and acyclicity is incorporated.

Actually, we disagree with the reviewer on this point. We are actually the first method to use meta-learning to directly target an approximation of the full Bayesian posterior and yield accurate samples from this approximate posterior.

Causal structures can only be represented by acyclic graphs. Methods like CSIvA and AVICI do not guarantee acyclic graph samples. CSIvA also does not encode permutation equivariance. Further, AVICI only targets the marginal probabilities so outputs samples from the wrong posterior in very simple cases (section 4.1).

Please see our general response for more details on this point.

The ideas that the paper uses to impose acyclicity and permutation invariance seem applicable in general. Can existing meta-learning approaches be modified to get the same?

This was the exactly starting point for our work. We wondered about this too before we decided it was necessary to develop an alternative method.

AVICI encodes permutation equivariance with respect to node while the autoregressive decoder architecture of CSIvA prohibits direct encoding of permutation equivariance.

In general, there is no clear way to ensure that we can sample from the correct posterior and obtain acyclic samples for AVICI or CSIvA. As AVICI was only used to output a single graph, an acyclicity regulariser is used directly on the edge probabilities, which may lead to an acyclic final graph if the probabilities are thresholded (set probability of over 0.5 to 1). However, using this regulariser also biases the posterior probability estimates. In the example of sec. 4.1, as the two graphs are completely unidentifiable, the true posterior should have marginal probabilities $P(A_{12}) = 0.5$ and $P(A_{21}) = 0.5$. Directly using an acyclicity regulariser on the probabilities promotes one of the terms to go higher and the other to go lower, leading to an inaccurate posterior. The same applies to the output probabilities of CSIvA.

The experimental validation for the case when the true posterior is known, is done only for the two-variable case. Is it possible to scale this up?

The point of this experiment was to show the clear downsides of the other meta-learning approaches (speciically AVICI). This is best visualised with the two variable linear Gaussian case where the true posterior is obvious.

In general, the true posterior can be found for multiple variable cases, but only for the linear Gaussian model with a specific prior [1].

[1] Geiger, Dan, and David Heckerman. "Parameter priors for directed acyclic graphical models and the characterization of several probability distributions." The Annals of Statistics 30.5 (2002).

How can the posterior over the graphs be used for inference on that of the functional parameters?

We're not sure if the posterior over the causal graph can be used directly to infer the functional parameters.

The functional parameters for a specific causal structure can be found by sampling a causal structure and using a different estimator to infer the functional parameters.

For downstream tasks such as causal effect estimation, [1] provides an example of how the posterior over causal structures can be used.

[1] Emezue, C. C., et al. (2023). Benchmarking Bayesian Causal Discovery Methods for Downstream Treatment Effect Estimation.

Gene Regulatory Network simulated data like SERGIO

Thank you for this suggestion. Syntren is exactly a gene expression network generated from a simulator. Is this sufficient or is SERGIO completely different?

The paper says that the metrics average over the posterior. I believe this is a problem with any metric. Are there alternatives that the authors propose?

This is an interesting quesiton. The first two metrics AUC and Log probability do not average over the posterior, however they only take into account the marginal edge probabilities. We believe there is scope for further research into properly evaluating Bayesian causal discovery methods.

We hope we have answered the reveiwers questions. We're happy to discuss any point further or answer any other questions.

We thank the reviewer for their comment that our method "implements several key desiderata for a Bayesian meta-learning model more effectively than existing alternatives" and our paper is "well-written, allowing readers to easily follow the motivation". We address your questions below.

it is challenging to identify which components provide truly novel technical contributions

We invite the reviewer to refer to the general comment for our summary of our main contribution.

Whereas previous meta-learning approaches were only concerned with a single graph, we parametrise the full posterior over causal structures. This lead us to include key properties of the posterior that the other methods missed (Table 1, section 4.1). The key changes compared to the architecture of the previous approaches include the decoder (L264) to include the above properties and the loss function (section 3.3) to train the network using meta-learning.

Our change to the decoder to include the key properties of the posterior resulted in using the permutation lower triangular construction. Whereas many methods have used this construction (see L133), they use variational inference or sampling based methods with explicit Bayesian models to infer the posterior.

The promise of meta-learning is to approximate the posterior more accurately than explicit Bayesian models (see L103, L199). We show that previous methods (AVICI, CSIvA) fall short of this promise. With the same training data as our model (BCNP), previous approaches such as AVICI and CSIvA do not outperform explicit Bayesian models DiBS and BayesDAG (e.g. Table 2). With our changes, our model BCNP outperforms explicit Bayesian models.

From my perspective, learning P(X,G) is likely very challenging. Indeed, Appendix B.1 notes that, for a case involving two variables, the model was trained on 200,000 datasets. I understand that Section 4.1 is intended to evaluate the posterior, but in Appendix B.2, 500,000 datasets are used for training. This information would be better suited for the main text, and details on computational resources should be transparently reported there as well (they are not provided even in the appendix).

We used an A100 GPU to train these models. We will include this in Section 4. Note that the data used for training these models is entirely synthetic (see Appendix B.2), hence as many datasets as required can be generated. Once a model is trained, inference for a new dataset is simply a forward pass. We agree that learning $P(X,G)$ is challenging, however our results show that the meta-learning approach, if done properly, is a viable path for Bayesian causal discovery.

If you have any more specific questions regarding this, or any comments that you think will help improve the paper, we would appreciate it.

can we realistically expect access to hundreds of thousands of dataset-graph pairs?

All the experiments were carried out by training on synthetic data which was generated from a pre-defined causal model (and hence any amount can be generated, Appendix B.2).

Worth noting is that the results in table 2 (on syntren) did not use training data from the syntren generator, but from the causal models defined in Appendix B.2 (see L861 for exact details). This shows that training on synthetic data and testing on entirely different distributions still gives good performance.

Known causal relations from real data, which will be much less in number, may be incorporated into an already trained model to learn an even better prior, but we leave how to finetune these models for future work.

Do you have any other specific questions that might help assuage any concerns about the meta-learning approach?

For DiBS and BayesDAG, the authors mention that they “keep all other hyperparameters to their preset values.” What if these hyperparameters were tuned? Would the results differ significantly?

Note that we tune all the hyperparameters that the authors tune for their work (or use tuned values from the authors where relevant). In fact we tune more hyperparameters for these baselines than we do for our model, where we only tune the learning rate. We also tune the learning rate only once for our model but tune the hyperparameters once for each dataset type for the baselines. Hence we do not believe the performance difference can be attributed to hyperparameter tuning.

The preset hyperparameters are things like the architecture (residual connections, normalisation layers etc.) and training steps that we set to the value used by the authors in their work.

Does this imply that all datasets must be of the same size? What if the sample sizes vary, such as 100, 200, or 500?

This is an interesting question. The datasets do not have to be the same size. Variable node and sample sizes can easily be handled by padding approppriately. This is commonly done for attention models when processing variable length sequences.

We thank the reviewer for their comments that "The paper clearly outlines its unique contributions" and that our paper "positions BCNP as a strong advancement over existing Bayesian causal discovery methods". We address your questions below.

One of the main limitations of this paper is the assumption that there are no latent confounders between variables. This assumption may limit the applicability of the method to real-world datasets where unobserved confounders are common. Is it nontrivial to consider the setting where latent confounders between variables exist?

Actually, it is very nontrivial to consider latent confounders [Ch. 9, 1]. When and why we can identify graphs with latent confounders is currently an open problem. The few methods that claim to identify latent confounders make very strong assumptions when doing so.

[1] Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.

Unlike fully nonparametric approaches like the PC algorithm, which do not impose strict functional or noise assumptions, BCNP’s effectiveness depends on these choices being well-suited to the data.

Methods such as PC are limited as they only use conditional independence information and hence can only identify a causal structure upto a Markov equivalence class [Ch. 4, 1]. Bayesian methods can distinguish within a Markov equivalence class [2].

The effectiveness of BCNP, like any Bayesian method, depends on how good the prior is for the data, and how well the posterior is approximated (Section 5). The advantage here is that we can use any functional mechanisms with priors that allow for sampling, where the posterior might be hard to approximate.

When there is no knowledge of the data generating process, a wider prior should be considered (Section 5). This is the approach we follow in Section 4.3. Here, the training data is not informed by the syntren generator and we still outperform all other methods (details in B.3). Thus, section 4.3 shows that our model can perform well even without knowledge of the true data distribution.

[1] Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.

[2] Dhir, Anish, Samuel Power, and Mark van der Wilk. "Bivariate Causal Discovery using Bayesian Model Selection." Forty-first International Conference on Machine Learning.

The proposed framework may be sensitive to hyperparameter settings and architecture choices, such as the hyperparameters for the encoder-decoder network, or the types of prior distributions. Fine-tuning these parameters is often data-dependent, which could reduce the model's robustness and generalization capabiltiy.

For the meta-learning models, we only tune the learning rate once by looking at the loss on a synthetically generated validation set. The same hyperparameters was used for all experiments. Hence the hyperparameters used were not dataset dependent. This is contrast with the baseline methods (DiBS and BayesDAG) where several hyperparameters were tuned for each dataset type (details in B.3).

The performance of our method does depend on the types of causal models from which training datasets are sampled (as they form the prior). However, as we show in Section 4.3, when we have no knowledge of the data generating distribution, sampling training data from a mixture of causal models (representing a wider prior) performs well (L861 has exact details).

Is the proposed method computationally efficient when a function class is chosen as neural networks?

A nice feature of meta-learning is that the computational performance is independent of the function class that is used to generate the training data.

How sensitive the proposed framework is regarding to the sparcity of the graph?

We test this in our experiments. See C.1 for results on varying density of graphs. Our model performs better than other methods as the density increases.

Are there any interesting example where the Bayesian approach beats the Markov-equivalence-based method?

[Ch.4, 1] relates how causal discovery can be done within a Markov equivalence class and [2] relates to how the Bayesian approach is able to do it. In general, methods that can identify within a Markov equivalence class tend to do better than Markov-equivalence-based methods. This is because methods like PC cannot differentiate graphs like $X \to Y$ and $Y \to X$.

[1] Peters, Jonas, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.

[2] Dhir, Anish, Samuel Power, and Mark van der Wilk. "Bivariate Causal Discovery using Bayesian Model Selection." Forty-first International Conference on Machine Learning.

We hope we have answered your questions and would be happy to discuss any point further.