Neuro-Causal Factor Analysis

how sensitive the results are to the hyperparameters? and in practice, how should users choose them?

The main hyperparameter, lambda, is similar to the choice of latent dimension in any generative model: If lambda is too small, then we would run into the same kinds of problems training any generative model with insufficient capacity. It does not have to be tuned carefully in practice.

More precisely, lambda should be greater than the number of causal latents, but how much greater is not as important. In practice (and our experiments on real data), VAEs are likely to have high enough latent DoF that experimentally exploring this further is beyond the scope of our presentation of the NCFA framework. Also see our response to the third question of Reviewer WRCe for discussion of how this parameter should be chosen by users.

is NCFA scalable for larger datasets and denser casual connections? what's the running speed of this method?

Our real data application in Section 5 (especially Table 1, and further discussion in appendices F and C) demonstrates that NCFA scales to large data sets like MNIST, even with dense (fully connected) structures and up to a 1/4 of a million latent degrees of freedom in the VAE. The computational bottleneck is the NP-hard problem of finding an exact minimum edge clique cover, however our applications in Section 5 demonstrate that using polynomial-time ECC heuristics (based on the 1-pure-child assumption) bring the runtime to the order of standard VAE methods without sacrificing much of the accuracy of the exact method.

how does NCFA compare to other causal inference methods for deep generative models?

This is discussed in Section 2.3. Our results are complementary to existing results, and make different, sometimes weaker, assumptions. We are happy to clarify any more specific questions you have on this.

Thanks for providing clarifications. However, I still have concerns on limited experimental results and the positioning of the paper on the claim of causal structure discovery. I will have to keep the current score.

What does MCM stand for? Medil Causal Models? If so, please provide the full name in the main text too.

Yes, MCM stands for MeDIL causal model. We have added this to the main text.

What are the details of model structure and training, especially the VAE model constructed following the induced minimum MCM graphs? These do not appear to be included either in the main text or appendix.

These details are given in Appendix C.

For example, when we have a M_1 to M_5 forming a Markov chain. Based on the described algorithm, the UDG may just have a complete graph with 5-node clique, which does not really provide any additional causal understanding or constraints in NCFA.

Such graphs are not within the model class we are concerned with. Namely, your example is Markov to a DAG over the measurement variables, while we are explicitly concerned with causal factor models, which we motivate beginning in the second paragraph of Section 1.

The authors may want to clearly define what they meant by NCFA being 'non-parametric' and 'identifiable'.

Our statistical model is nonparametric (infinite-dimensional without functional restrictions), whereas the estimator is parametric (finite-dimensional). The latter is of course always necessary if it is to be implemented on a computer, e.g. as with a neural network. A similar phenomenon arises in density estimation: The model (usually H"older, Sobolev, etc.) is infinite-dimensional, but the estimator is finite-dimensional (e.g. kernel estimates). In other words, VAEs are parametric estimators of nonparametric models.

It may also not be fair to claim the advantages on identifiability of minimum MCM graphs in NCFA over the existing other causal structure discovery methods due to the assumed specific settings.

To the best of our knowledge, our identifiability results are the first of their kind for nonparametric causal factor models. Other methods make parametric assumptions and use this added power to relax their model class at the expense of stronger assumptions. Our approach has the advantage of not needing the stronger parametric assumptions, but of course if one is willing to make such assumptions for a given application, other methods are more appropriate.

Fig 2: it seems that all graphs can be aligned as the hierarchical structure. Is this an assumption or based on some theoretical results?

It is a well-known fact (due to Pearl) that any latent variable model can be represented minimally in this way. So this is not an assumption, and follows from known theoretical results.

Definition 3.5: why this is called "observationally equivalent"? How about calling “equivalent wrt d-separation”?

The phrase "observationally equivalent" is standard in the causal graphical models literature. For example, see Judea Pearl's seminal book "Causality", which uses the phrase in Theorem 1.2.8 (on page 19, in the printing from 2000).

Algorithm 1: it is clear that the partition of variables into "two groups" are critical to the performance of the algorithm. Is there any analysis about the efficiency, and the error cascade if an inaccuracy partition appears?

The algorithm does not partition variables. Line 2 of the algorithm along with Definition 3.2 rigorously describes how the latent variables are learned from the measurement variables. We do provide analysis about efficiency and robustness in the face of error—we describe this at the end of the "Synthetic data" analysis in Section 5, with the help of Figure 3, as well as in more detail and with more plots in Appendix E.

Clarity is one of the main issues of this work. Many terminology descriptions are missing or without explanation. For example, what is the full name of MCM graph and ECC model?

MCM abbreviates "MeDIL causal model", and ECC abbreviates "edge clique cover". The MCM graph is rigorously defined in Definition 3.2 and ECC-observational equivalence is rigorously defined in Definition 3.5.

The notations are confusing and the theoretical results in this paper are problematic. Theorem 3.6 states that the DAG G is identifiable with the conditions that, which is problematic. Since a DAG identifiable means that every direction for every edge in the DAG will be identified, which is impossible without any assumptions under the existence of a latent variable. In fact, this theorem relies on the theory of Markham & Grosse-Wentrup, 2020 which has several underlying assumptions and constrained which is missing in the statement of the theorem.

The underlying assumptions and theory required for Theorem 3.6 are clearly stated in the first sentence of the theorem, in the phrase "the data-generating distribution is Markov to a minimum MCM graph". Being Markov to a minimum MCM graph means that each clique in the ECC of U implies the existence of a latent variable in G (and this is clearly stated in the proof, which explicitly references all necessary supporting theory). This latent variable (by the assumption that the MCM graph is minimum), necessarily has exactly the edges directed to the measurement variables in its corresponding clique. Hence, the clearly stated assumption renders the latent DAG (and every direction of every edge in it) identifiable.

How can we ensure the identifiability of loading functions (f) and residual measurement errors?

Our focus is on identifying the causal structure. We have left identifiability of the parameters to future work (and would note that there is already extensive work on this problem). Our approach is to emphasize causal interpretability and accurate generative modeling as opposed to factor loading identification.

Could you provide an intuitive proof framework to demonstrate the correctness of Theorem 3.6? Are there any graphical conditions when UDGs have a unique minimum edge clique cover?

In Corollary 3.7 (and its proof in Appendix B), we prove that a UDG has a unique minimum ECC when it has equal intersection and independence numbers. We also show that this condition on UDGs is equivalent to the 1-pure-child condition on latent DAG models known in the literature.

In practice, how do we set the latent degree of freedom lambda?

The experiments show that this parameter does not need to be carefully tuned, and indeed this is a key contribution. Simply put, we need lambda to be larger than the true number of causal latents, and small enough to train in practice. In other words, lambda could be in the thousands and it does not matter very much. (If lambda is too small, then we would run into the same kinds of problems training any generative model with insufficient capacity.)

Section 2.2 on Page 4: which relax the Causal Markov Assumption? It should be the Causal Sufficiency Assumption, right?

The causal Markov assumption (CMA) is basically (as mentioned in the second sentence of Section 2.2) Reichenbach's common cause principle plus the causal sufficiency assumption (CSA), so relaxing the CSA is one (but not the only) way of relaxing the CMA.

• Figure 2: double l_2,2?

Yes, thanks for pointing out the typo! It should be l_2,1 and l_2,2.