Latent Variable Identifiability in Nonlinear Causal Models with Single-domain Data under Minimality Condition

We first clarify a potential misunderstanding here, before going into the details. This paper focuses on identification of latent variables, rather than causal structures, and there are abundant works focusing on the same topic (See “Latent variable identifiability” section in our Related Works). Also, we did not claim any new identification algorithm as our contribution, neither did most related works on latent variable identifiability theory (not those on structure identification).

For weakness 1:

Identification of the latent variables in an SCM is never an easy work. It is a consensus that there is no free lunch: to make a model identifiable, assumptions are necessary, and stronger results require stricter assumptions. The assumption “there is no direct path between observed variables” has been adopted by many works (see line 155-157 for related works), and is reasonable and necessary in our setting (see our reply to reviewer aNXT for an explanation, question 5). We do not aim to identify all latent variables in the original SCM, and this is not one of our main claims. Instead, the reduction process explains why we introduce the PBG structure, and builds a connection between the identification results of PBG-SCM and that of original SCM.

For the mentioned literatures, their results rely on stronger assumptions. For literature [1], it belongs to a branch (mainly for structure identification) which restricts the function class (linear non-Gaussian model). Such restriction is not acceptable in the field of disentangled representation learning, since the real functions are usually nonlinear. For literature [2], it introduces a strong structural assumption: guaranteed number of pure children, which is critical for successful structure identification but hard to satisfy for general SCMs. Remember that there is no free lunch.

For weakness 2:

We understand that you view the minimality assumptions as “not mild”, since it cannot be satisfied by an arbitrarily picked model. We respect your point of view, but please listen to ours. By proposing Proposition 5.1, we have shown that there always exists a valid model satisfying minimality assumptions, given invertibility and independence are already satisfied. We say “mild” since there are abundant models satisfying minimality assumption, and don’t need to worry about the existence of such models (find an invertible and independent one, and then apply Proposition 5.1).

About the necessity of our assumptions, we list some examples for explanation. Consider a basis model $v_1=[s_1, z], v_2=[z, s_2]$. A case only violating invertibility:

$\hat{s}_1=[s_1, c], \hat{z}=z, \hat{s}_2=s_2$, $c$ can be random noise;

A case only violating independence:

$\hat{s}_1=s_1+s_2, \hat{z}=z, \hat{s}_2=s_2$;

A case only violating minimality:

$\hat{s}_1=constant, \hat{z}=[z, s_1], \hat{s}_2=s_2$;

All above cases can lead to the same observational distribution, thus the assumptions are necessary (we omit the structural equations for $v_1$ and $v_2$ here, since they are easy to get).

For weakness 3:

This is not a weakness, on the contrary, it is a contribution. Previous works usually omit the discussion of the importance of known dimension and simply assume it in their experiments. Exactly, we point out by our theory that, known dimension should not be a default setting, and need be highlighted both in the experiment setup and the theory. Please notice, we did not propose new identification algorithm as our contribution.

For weakness 4:

Considering the limited space, we have not included a proof sketch in the main text. We will consider adding one after arranging other contents. For the experiment, we have added a new experiment on controllable generation of images in the updated version (see Appendix A.9). Further experiments on real data may be unrealistic currently, since new identification algorithm is required to tackle the unknown dimension problem, which would be a challenging future work. 

For question 1:

The definition of SCM has many equivalent versions, such as a tuple with 2 elements [1], 4 elements [2], or 6 elements [3]. We adopt an equivalent definition that emphasize the graph structure $\mathcal{G}$, while $\mathcal{F}$ are the functions in the assignments, and $\mathcal{P}$ is the joint probability. Note that $\mathcal{G}$ and $\mathcal{F}$ constitutes the original assignments.

[1] Kaddour et al. 2022. Causal Machine Learning: A Survey and Open Problems

[2] Moraffah et al. 2020. Causal Interpretability for Machine Learning - Problems, Methods and Evaluation

[3] Bongers et al. 2021. Foundations of Structural Causal Models with Cycles and Latent Variables

For weakness 1:

Our work is significantly different with [1]. As discussed in line 109-110, there is a main drawback in theory of [1], i.e., their subspace-span assumption is about a relation between two models, making it infeasible to guide the design the identification algorithm (since the ground truth model is unknown). In contrast, our assumptions are all about the properties of one model, and we suggest to design restrictions for the learned model accordingly. The basis model may look similar, since it is a basic building block for larger models. However, there are differences between our basis model and [1]: we assume independence among all latent variables, while [1] allows dependence between $z$ and $s_2$; we aim to identify all latent variables, while [1] only aims to identify $z$. For the general model, our work is completely different from [1]: while [1] focuses on hierarchical structure (grow mainly in height), we retain two layers and only expand the number of variables (grow only in width). Meanwhile, the approaches are also completely different, the iterative nature of the proof is just a common math technique, there is nothing similar in the details.

For weakness 2:

The expression $j$&$2^l\neq 0$ means, in the binary representation of $j$, the $l$-th digit is $1$ (from right to left). We have added a new figure in the updated version to help understanding, see Appendix A.1.

For question 1:

There is no directed path from $s_1$ to $v_2$ in the model, so that $s_1$ does not influence $v_2$.

For question 2:

There are totally 9 edges in Fig. 7 (right), and we prove that 6 of them do not exist. So that the only 3 edges left are $s_1\to \hat{s}_1$, $z\to \hat{z}$, $s_2\to \hat{s}_2$, which are what we expect for identification.

For question 3:

We don’t really understand this question. We are focusing on block-wise identification (multi-dimension) rather than element-wise (single-dimension), what do you mean by “component-wise”? We follow [1] to use $R^2$, which in our view is reasonable. For “mean correlation coefficient”, do you mean the averaged Pearson correlation coefficient (PCC)? PCC is not suitable here, since it can only measure linear correlation, while we allow for nonlinear correlation here.

For question 4:

We have explained this in line 362-364. We ensure the parameter matrix is of full rank, and use invertible function Tanh as activation, so that the invertibility of MLP is guaranteed.

For question 5:

Here is an example for explanation. Consider a model with observed variables $v_1$ and $v_2$, if there exists an edge from $v_1$ to $v_2$, i.e. $v_2=f_2(v_1, \cdots)$, then replace $v_1$ with its structural equation: $v_2=f_2(f_1(PA(v_1)), \cdots)$. Run this process recursively, till all inputs of this equation are latent variables. Now you get a new SCM with no edge from $v_1$ to $v_2$, while keeping the observational distribution unchanged. In conclusion, there is always an equivalent model without directed paths among observed variables. Allowing such paths thus makes the model unidentifiable, unless including more structural assumptions, which is out of the scope of this paper.

For weakness 1:

We don’t view this claim as overly strong. Acquiring multi-domain data is at least harder than single-domain data, this is exactly the reason why some works try to infer the domain label or propose solutions without the need of multi-domain data. The two references are supportive in our view, we list some original text from their corresponding papers:

Matsuura & Harada, 2020: “However, most datasets, such as those collected via web crawling, are a mixture of multiple latent domains, and it is difficult to know the domain labels.”

Creager et al., 2021: “such environment labels are unavailable at training time, either because they are difficult to obtain or due to privacy limitations.”

For weakness 2:

In our view, the explicit reduction process provides a constructive proof itself. We realize that it may not be convincing enough, so we provide a proof in the updated version of our paper, see Appendix A.1 for it.

For weakness 3:

Given an SCM and an observed variable set, the reduction is possible as long as there are no directed path among observed variables. The associated cost may be losing information of unobserved endogenous variables (since they are originally unknown and unlikely to be identified), as well as fine-grained structures (since similar exogenous variables are concatenated). To understand the real applications of PBG, we give a simple example here: given 2 observed variables (such as image and its caption), can we discover the common factors that influence both (semantic information), and factors that influence only one of them (painting style, writing style)? This leads to a requirement for identifying $2^n$ latent variables ($n=2$ in this case), distinguished by whether they influence each observed variable.

For weakness 4:

Independence across all dimensions of each latent variable is not required by our theorem. We admit that the use of word “presume” may cause confusions. In fact, we have proved in the first paragraph of the original proof of Thm. 1 that, for a model that does not satisfy dimensional independence, it can be converted into an equivalent one that satisfies, without influencing the identifiability result. Due to such confusion, we have revised this part in the updated version.

For weakness 5:

We did not include much discussion with works on element-wise ICA, since they are typically not applicable in our block-wise identifiability settings. We are willing but currently unable to make a detailed comparison with works discussing sparsity, since this requires expanding our theory to element-wise ICA, which in our view is a new work. As far as our current understanding is concerned, our PBG structure provides a different kind of sparsity, such that each latent variable can be separated by the “difference” operation over the parent sets of each observed variables, while previous sparsity condition uses the “intersection” operation. Since the “difference” operation is strictly stronger than the “intersection” operation ($A\cap B=A-(A-B)$), our PBG structure may allow for more identifiable cases.

For weakness 6:

We provide a new controllable generation experiment on images in our updated paper, you can find it in Appendix A.9.

For weakness 1:

We did not provide a proof since the result is relatively obvious in our view. As there is confusion about it, we have added a proof in Appendix A.1 in the updated version.

For weakness 2:

Viewing observable variables as children of latent variables is typical in the field of latent variable identifiability [1-2], since the goal is to discover the underlying factors that influence observable variables. The only exceptions may be those allowing directed paths among observable variables, which we regard as future work and do not weaken the contribution of this work. We also provide a new controllable image generation experiment in the updated version (Appendix A.9), see if it helps for your understanding.

[1] Kong et al. (2024) Identification of Nonlinear Latent Hierarchical Models.

[2] Brady et al. (2023) Provably Learning Object-Centric Representations.