Identification of Latent Confounders via Investigating the Tensor Ranks of the Nonlinear Observations

Thanks for your careful and valuable comments. We will respond to these issues point by point.

[W] The Mixture Oracle method, a relevant baseline, is not included in the experimental comparison, which limits the evaluation.

WQ1: We have added experimental results using the Mixture Oracle as a baseline. Since K-means is not suitable for discrete data, we adapt the Mixture Oracle implementation by replacing K-means with the K-modes algorithm [1]. We evaluate the accuracy in identifying the latent support and include the results in the updated experiments. As shown in Table 1 of https://anonymous.4open.science/r/TensorRank-E052/Experimental%20results.pdf

One can observe that the Mixture Oracle method does not effectively identify the support of latent variables, even when using a discrete clustering algorithm. This may be due to the fact that clustering yields only an approximate solution and lacks theoretical guarantees for recovering the true latent structure.

[Q1] Could the author clarify the key differences between the tensor rank condition in non-linear causal models and the tensor rank condition in discrete latent structure models? How does this work advance previous results, particularly in relation to (Chen et al. 2024)?

Q1A1: We would like to clarify our contributions in two main aspects—applicability conditions and identification boundaries—particularly in relation to the work by Chen et al. (2024).

1. Our work considers a more general setting for when the tensor rank condition holds. In particular, we allow the observed variables to be continuous and show that the latent structure remains identifiable when each latent variable has two sufficiently informative observed variables. In contrast, Chen et al. (2024) propose the tensor rank condition when all variables are discrete and each observed variable is assumed to be a sufficient measurement for its latent parent (i.e., the observed support has higher cardinality than the latent’s).

2. We also extend the tensor rank condition to conditional probability tables, which leads to a more general identifiability condition for latent structures, as described in Appendix D. While Chen et al. (2024) explore rank constraints based on the joint probability table under the assumption of a three-pure-measurement-variable model, our approach considers constraints on conditional distributions. This allows us to test conditional relationships among observed variables in the presence of latent confounding (i.e., impure structures), and further, to identify the causal structure among latent variables even under impure setting (Appendix D). To the best of our knowledge, this provides a novel identifiability condition for discrete latent variable models.

   To evaluate structure learning under an impure setting, we conduct the following simulation, as shown in Table 2 of https://anonymous.4open.science/r/TensorRank-E052/Experimental%20results.pdf. We compare our method (Appendix D.2) with the approach proposed by Chen et al. (2024). One can see that the method by (Chen et al. 2024) cannot learn the causal structure among latent variable, this is because their method only suitable for the pure measurement model.

   Moreover, in Appendix D.3, we show that only two purely measured variables per latent variable are sufficient to identify the measurement model when the latent structure is fully connected. This result also relaxes the structural assumptions required by previous work based on the three-pure-measurement-variable assumption.

[Q2] For the case of continuous observed variables and discrete latent variables, previous work—such as the Mixture Oracle method—has also explored identifiability. What are the key differences between your approach and the Mixture Oracle method?

Q2A2: We would like to clarify our contributions with two main points, especially in relation to the Mixture Oracle approach.

1. We offer a robust and testable method for structure learning. Unlike the Mixture Oracle method, which relies on identifying a mixture model (as discussed in our Appendix E) and provides only an approximate method for parameter estimation—an approach that we find can be unreliable and difficult to test (see Sec. 3.1 and App. B)—our method introduces a hypothesis test for the tensor rank condition. This test not only enhances robustness compared to the Mixture Oracle but also allows us to identify the latent support with theoretical guarantees (see W3A3 in response to Reviewer e9yn).

2. We introduce a novel and more general structural condition for the identifiability of discrete latent structures (refer to Point 2 in Q1A1). In contrast, the Mixture Oracle assumes that no edges between observed variables, which is a quite restrictive assumption.

We sincerely appreciate your thoughtful inquiry and hope this clarification helps. Please feel free to reach out if you have any further questions.

We appreciate your valuable comments and suggestions and thank you for your positive assessment of our work.

[W1] Estimating tensor rank in practice can be challenging, yet the paper does not provide sufficient discussion on this issue.

W1A1: As discussed in the Sec. 5 (Lines 402–404), we estimate matrix rank using the hypothesis test proposed by Mazaheri et al. (2023), and tensor rank following the approach in Chen et al. (2024). We will add more discussion on this point to improve clarity.

[W2] The data generation process in the experiments follows a mixture model rather than directly following Eq. (1). The author should clarify this.

W2A2: Thank you for the comment. To ensure that the generated data satisfies the completeness condition, we use a mixture model as a practical and effective way to simulate data. Although the data is generated from a mixture model, the resulting joint distribution still reflects nonlinear dependencies (due to the probability property of the mixture model), and thus remains consistent with the setting described in Eq. (1). We will clarify this point in the revised text.

[W3] The effectiveness of the discretization approach depends on accurately estimating the rank of the probability table. However, the paper does not discuss how the precision of rank estimation impacts causal structure learning.

W3A3: In our theoretical results, once the latent support size $r$ is identified, the causal structure can be recovered through rank-decomposition with specified $r$. Otherwise, an incorrectly estimated $r$ may lead to incorrect identification of latent structure, especially in the number of latent variables. To show that the latent support can be estimated consistently, we provide additional simulation results on estimating the rank of the probability table.

We consider the following settings: $G_1$: $L_1 \to \{X_1, X_2, X_3, X_4\}$; $G_2$: $L_1 \to L_2$, $L_1 \to \{X_1, X_2, X_3\}$, $L_2 \to \{X_4, X_5, X_6\}$. In both settings, the latent variables have support size 2, and the observed variables have support size 3. Each experiment was repeated 1,000 times with randomly generated data. Using the hypothesis test by Mazaheri et al. (2023), we find that the rank of the probability table can be accurately estimated in most cases. This suggests that the effectiveness of our structure learning method, which relies on tensor rank, is supported by the reliability of rank estimation.

[Q1] Previous works often assume sufficient observational support for latent variable identification. Why is the two-sufficient measurement condition enough for testing the tensor rank condition?

Q1A1: Thank you for the question. For each latent variable, two sufficient measurement variable $X_i, X_j$ are enough to identify the latent support by detecting the rank deficiency in the joint distribution $\mathbb{P}(X_i, X_j)$. Moreover, there is a key property of tensor rank: the rank of a tensor is at least as large as that of any of its subtensors, e.g., Rank$(\mathbb{P}(X_i, X_j, X_k))$ $\geq$ Rank$(\mathbb{P}(X_i, X_j, X_k=c))$.

For example, consider an case that $L$ d-separated $X_i, X_j, X_k$, where $X_i, X_j$ are sufficient measurement variables and $L$ has the support size $r$. If the rank condition holds for a subtensor, e.g., Rank$(\mathbb{P}(X_i, X_j, X_k=c)) = r$, then the full tensor $\mathbb{P}(X_i, X_j, X_k)$ must also have rank $r$, regardless of whether $X_k$ satisfies the sufficient measurement condition (i.e., even if its support is smaller than $r$).

Therefore, two sufficient measurements per latent variable are enough to test the tensor rank condition.

[Q2] Why does the identifiability of Mixed LSMs require that at least one set of observed variables is caused by a single latent parent? If this is a structural assumption, it should be explicitly stated in the main text.

Q2A2: Thank you for pointing this out. This requirement is included in the Three-Pure-Child-Variable Assumption in Definition 4.1, and we further discuss it in Remark D.5. Broadly speaking, this assumption is used to identify the support of latent variables when considering the n-factor structure. We will make this assumption more explicit in the main text.

Thank you for your careful review. We address each point below and have corrected the typos. Please feel free to reach out with any further questions (due to limited space).

W1W2: ... realistic dataset to test their approach.

A: Please see Appendix H.1.

W3: Novelty is between low and moderate.

A: Please see the Q1A1 response of #Review gVAe.

Q1: Do you allow edges from observed to observed variables?

A: We allow edges between observed variables. This highlights that the tensor rank condition applies beyond the pure measurement setting (e.g., impure case in Appendix D).

Q3: Assp. 2.1 (a): Do you mean for all $r\in \Omega$ here?

A: Yes, we mean that for any $r \in \Omega$ here — each marginal places non-zero mass on every element of $\Omega$.

Q4: Assp. 2.1 (b): What is its shape? Which dimension corresponds to which variable?

A: The conditional distribution can be represented as a contingency table— a $|V_i| \times |Pa(V_i)|$ matrix.

Q5.1: Theo. 3.2: ...what is meant by the rank of a tensor.

A: The rank of a tensor $\mathcal{X}$, denoted $rank(\mathcal{X})$, is the smallest number of rank-one tensors that generate $\mathcal{X}$ as their sum, where a $N$-way tensor is rank-one if it can be written as the outer product of $N$ vectors.

Q5.2: Theo. 3.2: So $X_p$ is just the set of observed variables here? Or is it a subset?

A: We modify it to $\mathbf{X}_p =\{X_1,\cdots,X_{p}\} \subseteq \mathbf{X}, p \leq n$.

Q5.3: Theo. 3.2: Can the conditional set $S$ intersect with $X_p$? Or should it be constrained to the latent variables?

A: Yes, we allow $S \cap X_p \neq \emptyset$ and allow observed variables included in $S$.

Q6: Lower case r is used both for the rank of tensor and for the cardinality of $\Omega$.

A: We will use distinct symbols for tensor rank and the cardinality of $\Omega$ in revision.

Q7.1: Cond. 3.3: ...only X_i that satisfy “for all L_j, $X_i \bot X\setminus X_i | L_j$”, correct?... Could you give an example? Q7.2: ...function $g:\Omega \to R$ is bounded since $\Omega$ is finite? Q7.3: ...no quantifier for $i$ and $j$ here.

A: We will clarify this in the revision: for any $X_i \in \mathbf{X}$ that has only one (latent) parent $L_j$, we assume the conditional distribution $\mathbb{P}(L_j|X_i)$ is complete. That is, for all measurable real function $g$ such that $\mathbb{E}(|g(l)|) < +\infty$， $\mathbb{E}(g(l)|x) = 0$ almost surely iif $g(l) = 0$ almost surely. As an example, consider $X_7, L_3$ in Fig. 2. Besides, we assume that the domain $\Omega$ is finite. In this case, any function $g:\Omega \to R$ is indeed bounded.

Q7.4: Cond. 3.3: It seems completeness is a property of the joint $P(L_j, X_i)$...Q7.5: App. F on cond. 3.3 didn’t help.

A: Yes, completeness is the property of the joint distribution. Due to the Bayesian formula, the completeness on the conditional distribution $P(L_j|X_i)=P(L_j, X_i)/ P(X_i)$ involves the joint distribution. In this paper, we adopt the conditional form for consistency with prior work (e.g., Cui et al., 2023). In Appendix F, we illustrate that the completeness condition is not overly restrictive and can arise naturally in practice. We’ve revised the appendix to clarify this intention.

Q8: Theo. 3.4: How do you define rank here?

A: As noted in Remark G.3, we treat the tensor as a theoretical representation of the joint distribution, with rank defined by its minimal rank-one decomposition. We will clarify this in the revision.

Q9: Def. 4.1: What’s a pure variable? It’s not defined here. What’s a “sufficient measured variable”?

A: Pure variables denote the variables that have only one latent parent, and no observed parents. For an observed variable $X$ with support $\mathcal{X}$ and latent parent $L$ with support $\Omega$, we define a sufficient measurement as $|\mathcal{X}| > |\Omega|$.

Q10: Line 244: The “tensor rank condition” ... is not properly defined anywhere...

A: The rank condition refer the rank deficiency of probability tensor. Besides, your statement is right, we would like to study when and how to use discretization to make this properity hold in continous data.

Q11.1: where we are assuming that each observed variable has a single latent parent?

A: This assumption follows from the model definition, where we adopt the standard pure measurement setting (Silva et al., 2006), in which each observed variable has a single latent parent. We will clarify this more explicitly in the revision.

Q11.2: Does this proposition mean that any discretization will work?

A: No, not all discretizations will work. We clarify this point in Remark 4.3 and illustrate it with an example in Appendix B.

Q12: Theo. 4.5: What’s k here? The iteration of some algorithm?

A: k refers to the number of discretization steps, as discussed in Remark 4.6.

Q14: Prop. 4.15: Isn’t it weird that $X_s$ does not appear in (i)...?

A: $X_s$ does not need to appear in (i) since it may not share a common latent parent with the other variables.