On the Parameter Identifiability of Partially Observed Linear Causal Models

Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.

Q1: Regarding writing suggestions about Thm 1, Remark 1, and other related discussions.

A1: Theorem 1 provides a sufficient condition (which consists of conditions (i) + (ii)) for parameters to be identifiable. In Remark 1, we have shown that condition (i) is provably necessary. Yet, (i)+(ii) is not a necessary and sufficient one, as there exist some cases that fall outside of condition (ii) yet remain identifiable (and they are not of Lebesgue measure zero). We genuinely appreciate the reviewer's valuable suggestion and have revised to add multiple examples to illustrate these cases, as you suggested. As for disjunctive graphical conditions, we totally agree that it could be helpful, e.g., adding a condition (iii) to capture these rare cases such that (i)+(iii) could also be sufficient for identifiability. However, characterizing these cases is highly challenging as it may involve tools from algebraic statistics. Thus, we formulate our theorem 1 in the current form and plan to address this gap in future research. Regarding other writing suggestions such as moving the discussion of necessity to a section later and moving Appx. D to the main text, and correction of some typos, we genuinely thank the reviewer for the valuable feedback and have revised them accordingly.

Q2: Test of identifiability conditions will be helpful.

A2: Yes, this would indeed be very helpful for practitioners. In this work, we focus on the problem of parameter identification, where the structure is assumed to be given. Thus, testing the conditions in Thm.1 is straightforward in our context.

On the other hand, we believe your question pertains more to structure identification, e.g., whether Conditions 1 and 2 in [17] can be tested. In practical applications, certain strategies can be employed to gain a preliminary understanding of whether the graphical conditions hold; one effective approach is to utilize domain knowledge. Yet, a rigorous test of these graphical assumptions still remains a significant challenge in the field of causal structure learning in general and will be a key area for future research. In this regard, researchers also focus on another critical question: identifying testable consequences when certain conditions are violated, as you mentioned. Specifically for Conditions 1 and 2, if certain kinds of violations exist, consequences can be detected, and a detailed discussion can be found in [17]. We thank the reviewer for the valuable question and have added this discussion to our revision.

Q3: Would it be possible to perform inference on the learned parameters? For instance, standard errors on linear coefficients?

A3: Yes, it is possible to perform inference in our framework. To be specific, as we use maximum likelihood estimation for the parameters, some standard techniques can be readily used. For example, bootstrap can be employed to provide standard errors on linear coefficients and Chi-square test can also be done to examine the fitness of the model. Thank you for the valuable suggestion and we have added this discussion to our revision.

Q4: How it fares against competitors such as GES or PC when latent variables are absent?

A4: Yes, it is possible to compare our method with GES - although GES is primarily used for structure identification, it estimates all the linear edge coefficients during the calculation of likelihood. In contrast, constraint-based methods like PC cannot be compared. Asymptotically speaking, when latent variables are absent, the parameters estimated by our method will be exactly the same as that of GES, as in this case only the set of true parameters can maximize the likelihood. We also conduct additional experiments to check finite sample cases (5k and 10k data points are concerned) and the result is aligned with the asymptotic analysis - for graphs that have no latent variables, the parameters estimated by our method are nearly the same as that of GES (due to minor difference of implementation details of likelihood).

We genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.

Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.

Q1: It would be interesting to have sufficient conditions also in structures that do not satisfy conditions 1 and 2: G may be identified even if it does not satisfy 1 and 2 or the structure is provided by a researcher.

A1: Thank you for the insightful question. We aim to explore conditions such that the whole causal model can be fully specified from observational data, and thus Conditions 1 and 2 have to be considered for structure identifiability. These conditions are not overly restrictive; rather, they are currently the most general or less restrictive ones for structure identifiability involving latent variables in the linear Gaussian setting, to our best knowledge [17].

At the same time, we also agree that in some cases the structure might be directly given by domain knowledge or experts, and the parameter identifiability for these cases can be interesting. We genuinely thank the reviewer for the suggestion and have added a related discussion to our revision, which can be summarized as follows. If the structure identifiability is not a concern (e.g., when G is given by an expert), a weaker sufficient condition for parameter identifiability can be used: Condition 1, along with conditions (i) and (ii) in Thm 1, are sufficient for parameter identifiability, and Condition 2 is not required.

Q2: The gap between the sufficient and necessary conditions?

A2: Theorem 1 provides a sufficient condition for parameters to be identifiable. The gap between this sufficient condition and a necessary and sufficient condition is further analyzed in Remark 1. Specifically, the sufficient condition in Thm 1 consists of two parts: condition (i) and condition (ii). In our paper, we have shown that condition (i) is provably necessary, meaning the gap arises only from cases that fall outside of condition (ii) yet remain identifiable. Characterizing these cases is highly challenging as it may involve tools from algebraic statistics, and thus we formulate our theorem 1 in the current form.

We appreciate the reviewer's insightful question and have revised the paper to include illustrative examples of these cases. We plan to address this gap in future research, and hope that these examples inspire further exploration in this area. After all, establishing a necessary and sufficient condition is always highly non-trivial and often requires significant time and multiple efforts (e.g., it takes around 10 years for the structure identification of latent linear non-Gaussian models).

Q3: Computational complexity of parameter identification.

A3: The optimization in Eq.3 is solved by gradient descent, which involves evaluating the LogDet and matrix inverse (for the gradient) terms (similar to continuous causal discovery methods based on Gaussian likelihood [33]). According to [58], the computational complexity is $O(td^3)$, where $d$ is the number of variables and $t$ is the number of iterations of gradient descent. Note that the computational cost is largely independent of sample size as we only need to calculate the sample covariance once and save it for further use. Thank you for the valuable suggestion and we have added this analysis on complexity to our revision. As for the empirical runtime analysis, please kindly refer to sec 5.4, where our method is shown to be very efficient - e.g., it takes only around 2 minutes to estimate parameters for a graph that contains 50 variables.

Q4: To what extent the sufficient and necessary conditions in Sec 3 are useful for (numerical) parameter estimation in Sec 4?

A4: Given observational data and any structure, we can always employ our method proposed in Section 4 to get an estimation of parameters. However, whether the estimated parameters are meaningful depends on the specific given structure and we need to rely on the theoretical results developed in Section 3 to answer this question. Specifically, if the given structure satisfies the sufficient conditions proposed in Theorem 1, then we can conclude that the estimated parameters are meaningful in the sense that they will be the same as the true underlying parameters asymptotically (as in this case only the true parameters can maximize the likelihood). On the other hand, if a given structure does not satisfy the necessary conditions in Corollary 1 or 2, then it can be guaranteed that the true parameters cannot be recovered (by any estimation method as there is just not enough information). In such cases, multiple (infinite number of) parameter sets can yield the same maximum likelihood. We thank the reviewer for the insightful question and have added this discussion to our revision.

Q5: The motivation for considering edges / parmeters from observed to latent variables and between latent variables

A5: Thank you for the insightful question. From one perspective, if we do not explicitly consider the edge coefficients from observed to latent variables and edge coefficients between latent variables, then in many cases we cannot correctly recover the edge coefficients between observed variables either. From another perspective, the recovered edge coefficients that involve latent variables are themselves practically meaningful. Take our psychometric result in Figure 4 as an example. We may be interested in how Agreeableness influences Extraversion in human personality, even though none of them can be directly observed. In this case, the edge coefficient from L3 to L2 is informative; the value of +0.39 not only indicates a positive influence but also shows that the magnitude is considerable.

We genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.

[58] Toledo, Sivan. "Locality of reference in LU decomposition with partial pivoting." SIAM Journal on Matrix Analysis and Applications, 1997.

Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.

Q1: Regarding group sign indeterminacy.

A1: The group sign indeterminacy is rather minor with reasons as follows. (i) In practice, we can always anchor the sign of some edges according to our preference or prior knowledge in order to eliminate the group sign indeterminacy. For example, in Figure 4, if we expect that L2 should be understood as Extraversion instead of non-Exterversion, we can add one additional constraint during our parameter estimation such that the edge coefficient from L2 to E1 ("I am the life of party.") will be positive (as we believe E1 should be positively related to Extraversion). (ii) On the other hand, there are some application scenarios that are not influenced by the group sign indeterminacy, such as causal effect estimations between certain variables.

Q2: What if the additive noise follows another continuous distribution?

A2: Thank you for the insightful question. If we do not assume Gaussianity, the proposed asymptotic identifiability result still holds. The reason lies in that we only make use of the second-order statistics of the distribution and thus the additive noise can follow any other continuous distribution. We thank the reviewer for the insightful question and have revised to make it clear in our revision.

Q3: Will the inverses in Proposition 1 always exist?

A3: Yes, they always exist. Thanks for asking this question which helps improve the clarity of Proposition 1. In light of your suggestion, we have updated Proposition 1 to state that all matrix inverses exist. We have added the proof to our revision with a sketch as follows.

Sketch of proof. Note that matrices $I-D$ and $I-F$ are invertible because structure $\mathcal{G}$ is acyclic. This implies $\det(I-F)\neq 0$ and $\det(I-D)\neq 0$. Define

U=\\begin{pmatrix} I & 0 \\\\ -(I-D)^{-1}C & I \\end{pmatrix},

which implies

(I-F)U=\\begin{pmatrix} M & B \\\\ 0 & I-D \\end{pmatrix}

and thus

$$\det((I-F)U)=\det(M)\det(I-D).$$

Since $\det(U)=1$ and $\det(I-F)\neq 0$, we have

$$\det((I-F)U)=\det(I-F)\det(U)\neq 0.$$

By the statement above and $\det(I-D)\neq 0$, we have

$$\det(M)=\frac{\det((I-F)U)}{\det(I-D)}\neq 0,$$

which implies that $M$ is invertible. Similar reasoning can be used to show that $N$ is invertible.

Q4: It would be useful to also show how accurate the estimates of the noise covariance matrix are.

A4: We note that once the edge coefficient matrix $F$ is determined, the noise covariance matrix $\Omega$ can also be uniquely determined - we have $(I-F^T)\Sigma_{\mathbf{V}}(I-F)=\Omega$ where the left hand side only depends on $F$ when variables have unit variance. Thus, a small MSE of $F$ usually implies a small MSE of $\Omega$. In light of your suggestion, we conducted additional experiments to empirically validate this point: under the GS case, the MSE of $\Omega$ using our Estimator-TR are 0.003, 0.001, 0.0004 with 2k, 5k, and 10k sample size, respectively. We thank the reviewer for the valuable question and have included this additional result in our revision.

We genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.

Thank you for the time dedicated to reviewing our paper, the insightful comments, and valuable feedback. Please see our point-by-point responses below.

Q1: Whether the graph will be dense regarding Eq.3?

A1: We note that Eq.3 concerns the estimation of causal coefficients F, given the data and the structure G, and the entries of F that do not correspond to an edge in G are constrained to be zero during the optimization (as in lines 289-290). As the graph is given, sparsity constraints are not needed in Eq.3, which is in contrast to continuous-optimization-based structure learning methods such as NOTEARS.

On the other hand, you are totally right that we do expect that the given structure is parsimonious/sparse. For example, if a graph contains latent variables and all variables are fully connected, then according to our identifiability theory the parameters are provably not identifiable. Roughly, a sparse/parsimonious graph has a better chance of satisfying the condition for parameter identifiability.

Q2: Is it possible to restrict the MSE calculation to edges among observed nodes to compare with any baseline that does not allow the presence of latent variables?

A2: Thank you for your insightful suggestion. Yes, it is possible to restrict MSE to edges among observed to compare with methods that do not allow latent variables. For example, GES with BIC scores can be considered as it estimates all the linear edge coefficients during the calculation of likelihood. In light of your suggestion, we conduct additional experiments to compare with GES using your proposed restricted version of MSE. Specifically, GES achieves restricted MSE of 0.41, 0.35, and 0.35 with 2k, 5k, and 10k sample size, while our Estimator-TR achieves 0.008, 0.002, and 0.001, respectively. It is observed that, even when we restrict the MSE to edges among observed variables, methods that do not allow the presence of latent variables cannot perform well. The reason is that, if two observed variables have latent confounders, we cannot correctly recover the edge coefficients among them without explicitly modeling/considering the existence of latent variables. We thank the reviewer again for the valuable comment and have added the additional result together with this discussion to the appendix.

Q3: Discussion of uncertainty estimation?

A3: Many existing works on the parameter identification problem focus on the asymptotic identifiability [18,19], and thus our theoretical analysis follows this spirit, because this type of asymptotic identifiability result is needed before one can provide results on uncertainty estimation. At the same time, uncertainty estimation is certainly possible under our framework. To be specific, as we use maximum likelihood estimation for the parameters, some standard techniques can be readily used. For example, bootstrap can be employed to provide standard errors on linear coefficients. Chi-square test can also be done to examine the fitness of the model. Thank you for the valuable suggestion and we have added this discussion to our revision.

Q4: What does it mean by QF and F share the same support.

A4: Your understanding is correct. It means QF and F share the same set of non-zero entries. We have added this sentence to our revision as you suggested.

Q5: Some mention of how MSE up to orthogonal transformation is done would be helpful.

A5: MSE up to orthogonal transformation is calculated by solving the optimization problem in line 350. We use PyTorch with SGD to solve this problem where an orthogonal matrix Q can be directly parameterized and optimized. We thank the reviewer and have added it to our revision.

Q6: GPU acceleration?

A6: Yes, our optimization problem in Eq.3 is solved by gradient descent using pytorch (or any other automatic differentiation framework) and it can certainly be further accelerated by using GPU. A very related discussion can also be found in [33]. Our current implementation is based on CPU as it is already fairly fast - it only takes around 2 minutes to handle a quite big graph that has 50 variables (as discussed in section 5.4). Thank you for the valuable suggestion and we have added this discussion to our revision.

Q7: Regarding the rare cases of parameters, probably some justification is needed to articulate why this should be rare in real causal systems. Does any prior literature speak to this?

A7: This is the typical setting in parameter identification literature where generic identifiability is concerned [5,19] (a similar spirit is shared by causal discovery literature that assumes faithfulness [48]). These cases are rare in the sense that they have Lebesgue measure zero. In real causal systems, if all the edge coefficients are randomly generated (from an absolutely continuous distribution), these cases are not a concern because they are of Lebesgue measure zero. At the same time, if edge coefficients in a causal system are deliberately or adversarially designed, then identifying these parameters using only observational data can become extremely difficult (similar to violation of faithfulness assumption in causal discovery in the sense that typical constraint-based algorithms would fail).

Q8: Implications of the diagonal covariance matrix $\Omega$ of $\epsilon_{\mathbf{V}_\mathcal{G}}$ ?

A8: Do you mean why it is diagonal? As our framework explicitly models all the latent variables, all the $\epsilon_{\mathbf{V}_i}$ are mutually independent, and thus $\Omega$ is diagonal (in contrast, the ADMG framework does not explicitly model latent variables, and thus their $\Omega$ is not necessarily diagonal).

Q9: Regarding, "As variables are jointly Gaussian, asymptotically our observation can be summarized as population covariance over observed variable". Wouldn't this statement also apply in finite samples under Gaussianity?

A9: In finite samples under Gaussianity, our observation can be summarized as the empirical covariance, which is an estimation of the population covariance.

Q10: If actual interpretation of the edge weights is going to be done in practice, group sign indeterminancy would limit applicability. Would it help to anchor the sign of one edge based on prior science? Guidance?

A10: Yes, we can always anchor the sign of some edges according to our preference or prior knowledge in order to eliminate such indeterminacy. For example, in Figure 4, if we expect that L2 should be understood as Extraversion instead of non-Exterversion, we can add one additional constraint during our parameter estimation such that the edge coefficient from L2 to E1 ("I am the life of party.") will be positive (as we believe E1 should be positively related to Extraversion). Thank you for your insightful question and we have added a related discussion to our revision.

Q11: In Figure 4, are circulate nodes latent and nodes denoted by [LetterNumber] observed? If so, clearly articulate this in the figure label

A11: Yes. We have revised the caption of Figure 4 as you suggested.

Regarding the suggestions on writing such as additional shadings for examples, more precise statements for Remark 1, moving definition 7 into the main text and some explanations to the front of corresponding definitions, and correction of some typos, we thank the reviewer and have revised them accordingly. We genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.