Modular Learning of Deep Causal Generative Models for High-dimensional Causal Inference

We are happy that the reviewer found our paper well-written and our work novel. Below we address the reviewer’s concerns.

Missing literatures:

"There is a large number of related literature that is missing"

We apologize if we missed some related works that solve high-dimensional interventional and counterfactual sampling problems. We intended to include all the recent works that solve similar problems with deep learning models such as GANs, normalizing flow, and diffusion models. We would be happy to include the ones that we missed if the reviewer would kindly point us to them.

For the reviewer’s convenience, here is a list of some recent papers we cited and qualitatively compared our approach against:

• Balazadeh et al (2022). Partial identification of treatment effects with implicit generative models.
• Louizos et al (2017). Causal effect inference with deep latent-variable models.
• Kocaoglu et al (2018). Causalgan: Learning causal implicit generative models with adversarial training.
• Pawlowski et al (2020). Deep structural causal models for tractable counterfactual inference.
• Xia et al 2023. Neural causal models for counterfactual identification and estimation.
• Nemirovsky et al (2020). Countergan: generating realistic counterfactuals with residual generative adversarial nets
• Xia et al 2021. The causal-neural connection: Expressiveness, learnability, and inference.
• Bica et al (2020). Estimating the effects of continuous-valued interventions using generative adversarial networks.

Datasets:

"The reviewer would suggest using a more established test dataset"

We believe that the performed experiments in our paper illustrate our motivation, contribution, and technical novelty. However, we would greatly appreciate it if the reviewer could point us to other real-world datasets that they believe is suitable to evaluate our method for high-dimensional causal inference. We would happily include them in our experiments.

For the reviewer’s convenience, here we provide the set of experiments we discussed in the main paper and in the appendix.

• In Section 5.1, we highlight our flexibility to utilize pre-trained models in a front door graph setting with the colored MNIST dataset.
• In Section 5.2, we experiment on the real-world COVIDx CXR-3 dataset with 30k samples that we associate with a front door graph.
• In Appendix F.4, we showed our performance on a Colored-MNIST experiment where the causal graph contains 7 discrete variables and 2 image variables.
• In Appendix F.5, we experiment with the ASIA causal graph (6 nodes) from the popular bnlearn repository ( bnlearn.com/bnrepository).
• In Appendix F.6, we show our algorithm’s performance on the real-world Sachs protein dataset from the bnlearn repository which includes both observational and interventional data.
• In Appendix F.7, we show performance on a synthetic experiment with a causal graph with 7 nodes.

Chosen neural network architecture:

"Why is GAN the chosen backbone?"

Even though we learn the desired distribution with GAN training, our modular training routine is not specific to GANs as we mentioned in Appendix H. Essentially, if there is a way to train multiple conditional generative models with the same noise, our results should apply. For example, conditional VAE can be used instead of GANs since its decoder is not different from the generator of a GAN in terms of sampling. It just requires a different way of training.

"Why not something like normalizing flows"

This is an interesting question. For a causal graph without any latent: $X \rightarrow Z \rightarrow Y$, we can use normalizing flow to get exogenous noise $n_X$, $n_Z$, and $n_Y$ by learning some invertible functions. However, suppose, there exists an unobserved confounder $U$ between $X$ and $Y$. The causal graph is $X \rightarrow Z \rightarrow Y; X \leftarrow U \rightarrow Y$. Now, obtaining $U$ from $X$ and $Y$ and getting back $X$ and $Y$ from $U$ might not be an invertible function. Thus, normalizing flow may not be directly applied. Although Pawlowski et al. (2020) proposed a normalizing flow-based method for interventional and counterfactual sampling, the solution was for causal graphs with no unobserved variables. In GAN or conditional VAE, this issue is dealt with by initializing some random Gaussian noise to represent U and feeding the same noise to multiple networks $X$ and $Y$ while training. As a result, when $X$ and $Y$ are trained to learn their mechanisms their output will be correlated since they use the same initial random noise. We will add this discussion on the challenges in using different generative models in the presence of unobserved confounders to a discussion section in the camera-ready.

We again thank the reviewer again for their constructive feedback. We hope we addressed all their concerns. We are looking forward to answering any other questions the reviewer might have.

We thank the reviewer for considering our work as a new contribution to the literature and appreciate their positive perspective about our work.

Modular training for c-component containing both low and high dimensional nodes

The reviewer mentioned that the challenge we face while matching the full joint distribution containing low and high dimensional variable still exists if a c-component contains both the image and low-dimensional variables.

Yes, we agree with the reviewer’s insight. However, please allow us to explain that even in these cases, there is utility in using our method compared to the existing work.

Suppose, a causal graph has $N$ variables. Without modularization, we have to match the joint distribution containing $N$ (might be large) number of low and high dimensional variables in a single training phase. Matching that joint distribution with deep-learning models, and a complicated confounded causal structure could be difficult since we are attempting to minimize a very complicated loss function for a very large neural network. Our proposed method allows us to reduce the complexity of this problem tremendously by modularizing the training process to c-components. The size of a c-component is generally a lot smaller than the whole graph. Thus, even though we have to train mechanisms in a c-component together and match a joint distribution involving high and low dimensional variables, the complexity will be much lower. Without our approach, there is no existing work that can modularize and simplify the training process for a causal graph with latents.

To back up our argument with empirical evidence, we have considered matching the joint distribution for the following graph.

$I_1 \rightarrow Digit \rightarrow I_2 \rightarrow Color ; I_1 \leftrightarrow Color \leftrightarrow Digit$.

Here $I_1$ and $I_2$ are image nodes and the rest are discrete. $I_1, Digit, Color$ belong to the same c-component.

Our baseline NCM will attempt to match $P(I_1, D, I_2, C)$ by training mechanisms of all variables at the same time. Whereas, we i) first train $I_2$ to match $P(I_2|D)$ and then ii) train $I_1, D,C$ to match $P(I_1, D, I_2, C)$. At the first step, $I_2$ trains well. In the 2nd step, since we don’t have to train $I_2$ anymore, matching the joint gets easier. Below we report the image quality of our baseline and our method.

FID scores representing the image quality of $I_1$ (lower is better):

FID scores representing the image quality of $I_2$ (lower is better):

The total variation distance for P(D, C) in both methods seems to be similar. But as we observe above, our method achieves better image quality than NCM.

"Still need to match $P(Z_1,Z_2,Z_3|X_1)$ and $P(X_1, X_2, Z_1, Z_3)$; does not simplify matching $P( X_1,X_2,Z_1,Z_2,Z_3)$"

We mention the causal graph here for the reviewer’s convenience: $Z_3 \rightarrow Z_1 \rightarrow Z_2; X_1 \rightarrow Z_1 \rightarrow X_2; Z_3 \leftrightarrow Z_1 \leftrightarrow Z_2; X_1 \leftrightarrow X_2.$

To match the full joint distribution $P(X_1, X_2, Z_1, Z_2, Z_3)$, the joint training (NCM) has to train mechanisms of all $X_1, X_2, Z_1, Z_2, Z_3$ together at the same time. Whereas, in our approach, to match the joint distribution, we first train $Z_1, Z_2, Z_3$ at the first step and train $X_1, X_2$ at the 2nd step. Based on the above argument, the training complexity to match the joint distribution reduces in both steps compared to NCM.

We thank the reviewer for acknowledging our novelty in causal sampling involving high-dimensional variables. We happily address each of the mentioned points by the reviewer.

Lack of discussions on assumption 3:

"Lack of discussions on assumption 3 and their violation"

We apologize to the reviewer for not clearly explaining it. We will add a simpler statement about this assumption in the paper. For the reviewer’s convenience, we restate the assumption here.

Assumption 3: for any h-node $H_k$ in the $\mathcal{H}$-graph, GAN training of DCM (our method) on dataset $\mathbf{D}$ converges to sample from the conditional distribution $P(H_k , \mathcal{A}_k | {pa(H_k , \mathcal{A}_k)})$ for all $H_k$, where $\mathcal{A}_k$ is from Algorithm 2: Modular Training.

Although the statement was written in a formal manner, assumption 3 simply says that we assume that GAN training converges and can match the conditional distribution after training on the given dataset. This is a common assumption used in many GAN papers in literature ([1], [2], [3], [5]). In our algorithm, we are training the set of all models modularly and matching corresponding conditional distributions. With sufficient representative power of the neural networks, and motivated by the success of deep generative models, we believe this assumption often holds in practice.

If this assumption is violated, i.e., if the generator models do not converge properly, our algorithm will output a near-optimal solution. We expect this to only smoothly affect the causal queries of interest but it is a very interesting future direction that we are interested in analyzing, i.e., how imperfect distribution matching affects the interventional samples.

Hidden assumptions in the text:

"Some assumptions are hidden in the text besides assumptions 1 to 3."

We agree with the reviewer that we stated some assumptions in the text. However, these are basically restatements, and were already stated in Section 4.2 on page 7 of our paper. For convenience, we restate those main assumptions here:

1. The causal graph is given
2. The causal model is semi-Markovian and
3. GAN training can correctly learn the desired conditional data distribution.

To address the reviewer’s comment, we will cross-reference in-text assumptions with those we listed above.

In section 4, we assumed that the neural networks have sufficiently many parameters to induce the observed distribution induced by the true SCM. Please note that this is basically assumption 3: GAN convergence.

Only in section 5.2, for the Covid X-ray experiment, we did not have access to the true causal graph (assumption 1 was violated). Thus, we had to fill its gap with some extra assumptions about the data distribution (no distribution shift and selection bias) and the causal relations among variables.

We are honored that the reviewer found the problem we are solving interesting and important. We also thank them for appreciating the soundness of our algorithm. Here we address the reviewer’s concerns below:

Choice of experiments:

“I would encourage to also test the method on a more complicated synthetic dataset with more than 3 nodes.”

We agree with the reviewer that the main paper considers causal graphs with 3 nodes. The intention was to illustrate the flexibility to utilize available pre-trained models (For example Covid-Xray model) and the significance of modularization. However, we would like to draw the reviewer’s attention to Appendix F of our paper where we also showed our algorithm performance on more complicated graphs. Here we list the causal graphs we used for our experiments more specifically:

• 3 nodes: in Section 5.1, we highlight our flexibility to utilize pre-trained models and illustrate our performance compared to baselines on the MNIST dataset.
• 3 nodes: in Section 5.2, we experiment on the real-world COVIDx CXR-3 dataset with 30k samples that we associate with a front door graph.
• 9 nodes: in Appendix F.4, we showed our performance on a Colored-MNIST experiment where the causal graph contains 7 discrete variables and 2 image variables.
• 6 nodes: in Appendix F.5, we experiment with the ASIA causal graph (6 nodes) from the popular bnlearn repository ( bnlearn.com/bnrepository).
• 4 nodes: in Appendix F.6, we show our algorithm performance on the real-world Sachs protein dataset that includes both observational and interventional data.
• 7 nodes: in Appendix F.7, we show performance on a synthetic experiment with a causal graph with 7 nodes.

We believe that our experiments clearly illustrate our motivation, contribution, and technical novelties. Having said that if the reviewer has any specific synthetic/real-world datasets in mind, we would be happy to perform more experiments and demonstrate our performance on them.

Quantifiable metrics for image results:

“The evaluation relies heavily on visual comparison of sampled data rather than quantifiable metrics”.

We agree with the reviewer that the evaluation relies on visual comparison. Since our aim was not to improve image quality but rather to consistently generate interventional and counterfactual samples, we modeled our synthetic dataset in such a way that visual inspection is enough to assess the correctness.

To measure the distance between our predicted distribution and the ground truth, we used the total variation distance ( TVD) and the KL divergence in all of our experiments. We would like to kindly point to our results in Appendix F (page 33), where we showed KL divergence as a metric for evaluation of our method.

However, to improve our paper according to the reviewer’s suggestion, we added the popular Fréchet inception distance ( FID) as our third metric that represents the quality of generated images. We compare our method with a baseline for the front door MNIST experiment (section 5.1).

This table shows that we achieve better image quality (lower FID) with more training epochs compared to the non-modular baseline.

Note that Mean squared error (MSE) is typically used in regression and is not directly applicable to our method since we measure the distance between the true and fake joint distributions. Although the average treatment effect (ATE) is suitable for binary variables, in this simulation we deal with larger support sizes. For variables with larger support sizes, TVD and KL metrics contain more information about the distance between true and fake distribution.