Identifiability Analysis of Linear ODE Systems with Hidden Confounders

Thank you for your valuable comments. We have addressed each of your comments as follows. Additionally, we have revised our manuscript in accordance with your suggestions, and we believe that the quality has been significantly enhanced as a result of your insightful input.

Answers to weaknesses:

W1: Explain the symbols $\boldsymbol{x}_0'$ and $A'$.

A: Thank you for pointing this out. The symbols $\boldsymbol{x}'_0$ and $A'$ are first introduced in our paper in Definition 2.1. To enhance clarity, we have included an explanation in the subsequent paragraph, which reads as ``We use $\boldsymbol{x}_0'$ and $A'$ to distinguish other system parameters from the true system parameters $\boldsymbol{x}_0$ and $A$; $\boldsymbol{x}_0'$ and $A'$ can represent any $d$-dimensional initial conditions and any $d\times d$ parameter matrices, respectively."

W2: Provide intuitive explanations for the assumptions and theorems.

A: Thank you for your valuable suggestion. We have provided intuitive explanations for each assumption as they are introduced. The added explanations are as follows (please refer to Table 1 in the uploaded PDF file for the summary of the proposed conditions):

1. For condition A1 in Theorem 3.1 and condition B1 in Theorem 4.1: These conditions are the same as the one stated in Lemma 2.1 for the fully observable ODE system (1), but with a different initial condition $\boldsymbol{\beta}$ or $\boldsymbol{\gamma}$. We have added the intuitive explanation of this condition in the paragraph following Lemma 2.1, since it is where this condition was first introduced. The added explanation reads:

   "From a geometric perspective, the set of vectors stated in Lemma 2.1 being linearly independent indicates that the initial condition $\boldsymbol{x}_0$ is not contained in an $A$-invariant proper subspace of $\mathbb{R}^d$. Intuitively, this means the trajectory of this system started from $\boldsymbol{x}_0$ spans the entire $d$-dimensional state space. That is, our observations cover information on all dimensions of the state space, thus rendering the identifiability of the system."

2. For condition C1 in Theorem 4.2: The added explanation reads:

   "Under the latent DAG assumption, we can transfer the ODE system (3), which includes hidden confounders, into a $(d+p)$-dimensional fully observable ODE system (1) through the augmented state $\boldsymbol{y}(t)$. Condition C1 indicates that our observations span the entire $(d+p)$-dimensional state space, thus rendering the system identifiable."

3. For conditions B2, B3, B4 in Theorem 4.3: Condition B2 is the same as B1, for which the explanation has been added. Conditions B3, B4 are straightforward and do not require additional intuitive explanations.

4. For condition C2 in Theorem 4.4: This condition is the same as C1, for which the explanation has been added.

In addition, we have included a table to summarize all the notations (please refer to our comment to reviewer Te42) and another table to outline the proposed identifiability conditions (available in the uploaded PDF file). We believe these additions significantly enhance the clarity and readability of our manuscript, aiding readers in better understanding the essence of our proposed theories.

Answer to question:

Q1: Practical validity of the proposed theoretical results.

A: Thank you for your question. The primary assumption in our manuscript is the latent DAG assumption, which is standard in causality studies. For a detailed discussion on the practicality and reasonableness of this assumption, please refer to our response to W2 from Reviewer 1wF2.

In addition to the latent DAG assumption, the other assumptions are quite mild:

1. Conditions in Theorem 3.1, 4.1, 4.2: These conditions are both sufficient and necessary and cannot be relaxed further in our linear ODE setup. As we have mentioned conditions A1 and B1 align with the condition stated in Lemma 2.1, which is the set of vectors $\\{\boldsymbol{x}_0, A\boldsymbol{x}_0, \ldots, A^{d-1}\boldsymbol{x}_0\\}$ being linearly independent (denote it as condition A0). This condition is generic as noted in [40], meaning that the set of system parameters that violate this condition has Lebesgue measure zero. Intuitively, condition A0 is satisfied for almost all combinations of $\boldsymbol{x}_0$ and $A$. Once condition B1 is satisfied, then one can always find observations on the same trajectory satisfying condition C1, rendering condition C1 also mild.
2. Conditions in Theorem 4.3 and 4.4: These conditions are sufficient but not necessary. Identifying parameter matrices $B$ and $G$ requires additional observations (trajectories) and assumptions. Condition B2 and C2 mirror the mild conditions B1 and C1, respectively. Condition B3 is experimentally controllable and trivially satisfied, while condition B4, similar to A0, is generic. Thus, all proposed conditions are practical and not restrictive.

Regarding real-world dataset experiments, we were unable to include them due to the unavailability of suitable data. However, we have added two real-world linear ODE examples (see the comment to Reviewer 1wF2) and additional higher-dimensional simulation cases to better support our theoretical results. The corresponding simulation results are presented in the uploaded PDF file. These simulations align with current studies on theoretical identifiability of linear ODE or SDE systems, which also rely solely on simulations (see [27, 33, 40, 41]).

Here, we provide the added table for summarizing all the notations.

Thank you for your valuable comments. We have addressed each of your comments as follows. Additionally, we have revised our manuscript in accordance with your suggestions, and we believe that the quality has been significantly enhanced as a result of your insightful input.

Answers to weaknesses:

W1: Include tables that summarize the notations and identifiability conditions.

A: Thank you for your valuable suggestion. In accordance with your recommendation, we have included a table to summarize all the notations (please refer to the following comment) and another table to summarize all the proposed identifiability conditions (available in the uploaded PDF file). We believe that these additions significantly enhance the clarity and readability of our manuscript.

W2: Scale the simulations to a more complex system.

A: Thank you for your comment. We would like to clarify that, theoretically, our proposed identifiability conditions are applicable to any finite dimensions $d \geqslant 1$ and $p \geqslant 1$. In practical scenarios, as the system dimensions increase, the complexity of the system also escalates. Consequently, larger sample sizes and more advanced parameter estimation methods may be required to achieve satisfactory parameter estimates.

In response to your suggestion, we have included additional simulation examples with increased complexity in our updated manuscript. For single trajectory identifiability validation (Theorems 4.1 and 4.2), we have added an example with $d=5, p=5$ (i.e., 5-dimensional observable variable and 5-dimensional latent variables, totalling 10 dimensions). For $p$ trajectory identifiability validation (Theorems 4.3 and 4.4), we have added an example with $d=10, p=5$ (i.e., 10-dimensional observable variables and 5-dimensional latent variables, totalling 15 dimensions). The results of these simulations are presented in the uploaded PDF file and provide empirical evidence supporting the validity of our proposed identifiability conditions in more complex systems.

Answer to question:

Q1: Elaborate on how the agent obtains matrix $G$.

A: Thank you for your question. As discussed in our manuscript, the identifiability of matrix $G$ is established under the conditions stated in Theorem 4.3 and Theorem 4.4. The proof of Theorem 4.3, detailed in Appendix B.4, provides a comprehensive derivation of how matrix $G$ can be obtained. Due to the complexity and extensive use of notations, it is challenging to fully elaborate on this proof in a few plain sentences.

To summarize, obtaining matrix $G$ requires not only the latent DAG assumption but also the assumptions B2, B3, and B4 outlined in Theorem 4.3. A crucial aspect of the latent DAG assumption is that it allows the matrix $G\in \mathbb{R}^{p\times p}$ to be permuted into a strictly upper triangular form, such that $G^k = 0$ for all $k \geqslant p$, where $p$ is the dimension of the latent variables. Consequently, the states of hidden variables can be expressed as polynomial functions of time $t$, from which the identifiability conditions are derived.

We hope this brief explanation provides some insight into the latent DAG assumption. For a detailed derivation of matrix $G$, we encourage you to refer to our proof in Appendix B.4.

Dear Reviewer Te42,

Thank you so much for raising your confidence score. Your recognition of our work means a great deal to us. In addition, regarding the simulations, we have added an additional simulation inspired by Reviewer 1wF2's comment. Specifically, we set different ground-truth parameter configurations by using different seeds. Due to time constraints, we have currently applied 10 different configurations to the single and multiple trajectory experiment with $d=3, p=3$. We will update these results to include 100 different configurations in our final manuscript, and we will also include higher-dimensional cases.

The simulation results are provided in the comment to Reviewer 1wF2. Through these additional simulations, we have increased the diversity of our ground-truth examples, providing more convincing empirical support that our proposed theoretical results are suitable for any system parameter configurations that meet the proposed identifiability conditions. We believe that our simulation has been greatly improved by adding this set of simulations.

Thank you again for increasing your confidence score and for your valuable comments.

Sincerely,

The Authors

Example 1: damped harmonic oscillators model

Consider a one dimensional system of $D$ point masses $m_i (i = 1, \ldots, D)$ with positions $Q_i(t) \in \mathbb{R}$ and momenta $P_i(t) \in \mathbb{R}$. These masses are coupled by springs characterized by spring constants $k_i$ and equilibrium lengths $l_i$, and are subject to friction with a coefficient $b_i$, all while the end positions are fixed.

The dynamics of this system are described by the following linear ODE system [24]:

$

\begin{split} \dot{P}_ i(t) &=k_i(Q_{i+1}(t)- Q_i(t) -l_i)-k_{i-1}(Q_i(t) -Q_{i-1}(t)-l_{i-1}) - b_i P_i(t)/m_i \\ \dot{Q}_i(t) &= P_i(t)/m_i \end{split}

$

where $Q_0(t) = 0$ and $Q_{D+1}(t) = L$ represent the fixed boundary conditions. External forces $F_j(t)$ (e.g., wind force or a varying magnetic field) can influence the entire system of coupled harmonic oscillators. These external forces can be modelled as latent variables with a constant derivative. Consequently, the system can be expressed as:

$

\begin{split} \dot{P}_ i(t) &= k_i(Q_{i+1}(t) - Q_i(t) -l_i)-k_{i-1}(Q_i(t) -Q_{i-1}(t)-l_{i-1}) - b_i P_i(t)/m_i + \sum_{j}\alpha_{ij} F_j(t)\\ \dot{Q}_i(t) &= P_i(t)/m_i\\ \dot{F}_j(t) &= c_j \end{split}

$

where $\alpha_{ij}$ is a constant determining the effect of the external force $F_j(t)$ on the $i$-th mass, and $c_j$ is the constant rate of change of the external force $F_j(t)$. This model aligns well with our ODE system (2). An example causal graph illustrating this model is provided in the uploaded PDF file.

Example 2: population model

The growth of a population $P$ can be described by a linear ODE:

$

\dot{P}(t) = a P(t),

$

where $a$ is a constant representing the growth rate of the population. The system can also be influenced by latent variables $L_i$, such as environmental factors and food supply. Incorporating these latent influences, the system can be modelled as:

$

\begin{split} \dot{P}(t) &= a P(t) + b L_1(t) + c L_2 (t)\\ \dot{L}_1(t) &= l L_2(t)\\ \dot{L}_2(t) &= m
\end{split}

$

where $a, b, c, l$ and $m$ are constants. Here, $L_1(t)$ represents the food supply, influenced by the environmental factor $L_2(t)$. $L_2(t)$ corresponds to an environmental factor, such as temperature or pollution levels, which changes steadily over time. This model aligns well with our ODE system (3).

Thank you for your valuable comments. We have addressed each of your comments as follows. Additionally, we have revised our manuscript in accordance with your suggestions, and we believe that the quality has been significantly enhanced as a result of your insightful input.

Answers to weaknesses:

W1: Provide practical examples.

A: Thank you for your comment. Since both this and your first question (Q1) address the practical value of our paper, we will respond to them together here.

In response to your suggestion, we have included two real-world linear ODE examples. Due to the 6000-character limit, detailed descriptions of these two models are provided in the following comment, and the corresponding causal graphs are presented in the uploaded PDF file. As illustrated by the graphs, the structure of these models aligns well with our structural constraints.

Regarding the applicability of the proposed identifiability condition and real-world dataset experiments, we kindly refer you to our detailed response to Q1 from Reviewer LnTR, which addresses this query comprehensively. Thank you for your understanding.

W2: Feasibility of the latent DAG assumption.

A: Thank you for your comment. In response to your second question, we believe there may be some confusion regarding how the causal graph is defined within the context of an autonomous (time-invariant) ODE system. To address this, we will first clarify the causal graph in the context of an ODE system, which we hope will provide a clearer understanding of the feasibility of the latent DAG assumption.

To enhance understanding, consider a general fully observable time-invariant ODE system $\dot{\boldsymbol{x}}(t) = f(\boldsymbol{x}(t))$. In such systems, the derivatives of state variables $\dot{\boldsymbol{x}}(t)$ do not explicitly depend on time $t$. For an ODE system like this, we state there is a direct causal relationship from variable $x_j$ to variable $x_i$ if $\dot{x_i}$ is dependent on $x_j$, expressed as $\dot{x}_i(t)= f(x_j(t))$. As detailed in [24], the causal graph for such an ODE system is defined such that each node represents a variable $x_i$, and there is a direct edge from $x_j$ to $x_i$ if and only if $\dot{x}_i$ depends on $x_j$. This causal graph remains invariant over time in a time-invariant ODE system. Both ODE systems (2) and (3) in our manuscript are time-invariant, and the graphs in Figure 1 are well-defined within this context.

We assert that the latent DAG assumption is reasonable for several reasons:

1. Consider a particle of mass $m$ in a uniform gravitational field where the gravitational field exerts a constant force $F$ on the particle. The evolution of the particle's velocity (denoted as $v$) and position (denoted as $r$) can be described by a linear time-invariant ODE system:

   $\dot{v}(t) = F/m, \dot{r}(t) = v.$

   The corresponding causal graph of this ODE system is $v \rightarrow r$, which is a DAG. Hence, a DAG is a reasonable causal structure to describe ODE systems.

2. Since we focus on time-invariant ODE system analysis, the causal graph remains invariant with respect to time. Therefore, treating the causal graph as static and making a DAG assumption is a natural extension of traditional static causal studies.

3. Deriving identifiability conditions for causal models is a challenging problem. This is why the DAG assumption is adopted in static setups, even in fully observable cases. Similarly, deriving the identifiability conditions for linear ODE systems with hidden variables is difficult, and this field of study is still in its early stages. To our knowledge, our work is the first to systematically derive identifiability conditions for such ODE systems. Referring to well-established assumptions in traditional causal studies is a prudent starting point. Additionally, we allow for cycles and self-loops among observable variables and only assume that latent variables follow a DAG structure, which is relatively less restrictive compared to the classic DAG assumption in static setups. Without the latent DAG assumption, it is currently not feasible to derive identifiability conditions for linear ODE systems with hidden confounders.

W3: Regarding randomness of the simulation study.

A: Thank you for your comment. Upon reviewing our uploaded codes in the supplementary material, you will see that all parameters in our simulation study are entirely randomly generated. Specifically, we used randint(-2,3) to generate the parameters for simplicity.

We chose to set the initial parameter values close to the true parameter values with uniform perturbations $U(-0.1,0.1)$ and $U(-0.3,0.3)$ because the Nonlinear Least Squares (NLE) loss function associated with our simulation is non-convex. Initializing the parameters close to the true values helps the NLE converge to the true global minimum. Introducing more randomness into the initial parameter values would necessitate using a computationally intensive and time-consuming global optimization technique or another parameter estimation method.

The primary objective of our simulation is to validate the proposed theoretical results rather than check or develop parameter estimation methods or techniques for ODEs. We believe that the current simulation results robustly support our theoretical claims. However, to address your concern and further substantiate our findings, we have included two additional higher-dimensional cases.

Answers to questions:

Q1: Provide practical examples.

A: Please refer to our response to W1.

Q2: Explain causal graph for an ODE system.

A: Please refer to our response to W2.

Q3: Increase dimension in simulations.

A: Thank you for your question. Due to the 6000-character limit, we kindly refer you to our response to W2 from Reviewer Te42, as it addresses the same query in detail. We appreciate your understanding.

Dear Reviewer 1wF2,

Thank you for your response. We appreciate your earnest and responsible approach to reviewing our paper.

However, we believe there may have been some misunderstanding regarding the setup of our simulations. Our simulation methodology follows a standard and widely accepted approach for verifying proposed theories like ours. Specifically, we set a single ground-truth parameter configuration. To ensure reliable and robust parameter estimation results, we then run multiple replications (100 in our simulation) of experiments with different initial parameter values and report the mean and variance of the metric of interest (MSE in our simulation).

We encourage you to take a few minutes to review the simulation settings in the recently accepted JMLR paper [41] and NeurIPS paper [40], which also study the identifiability analysis of linear ODEs and linear SDEs. These papers employ the same simulation setup as ours.

To address your concerns, we have conducted additional simulations incorporating various ground-truth parameter configurations by utilizing different seeds. The simulation results strongly affirm the validity of our proposed identifiability conditions. For further details, please refer to the following comment.

Thank you again for your consideration. If you have further questions or need additional clarification, please do not hesitate to contact us.

Sincerely,

The Authors

Dear Reviewer 1wF2,

We would like to reiterate that our theoretical results are applicable to any system parameter configurations that meet the proposed identifiability conditions.

To further address your concerns, we have added two additional simulations inspired by your comments. Specifically, we set different ground-truth parameter configurations by using different seeds. Due to time constraints, we have currently applied 10 different configurations to the single and multiple trajectory experiment with $d=3, p=3$. We will update these results to include 100 different configurations in our final manuscript, and we will also include higher-dimensional cases.

The simulation results are provided in the following tables, and they offer strong empirical evidence supporting the validity of our proposed identifiability conditions. Through these additional simulations, we have increased the diversity of our ground-truth examples, providing more convincing empirical support that our proposed theoretical results are suitable for any system parameter configurations that meet the proposed identifiability conditions. We believe that our simulation has been greatly improved by adding this set of simulations.

Table1: MSEs of the $\boldsymbol{\eta}$ -(un)identifiable cases of the ODE (3) with $d=3, p=3$ and different parameter configurations

Table2: MSEs of the $\\{\boldsymbol{\eta}_i\\}_1^p$ -(un)identifiable cases of the ODE (3) with $d=3, p=3$ and different parameter configurations

Thank you again for your valuable comments. We hope this additional set of simulations addresses your concerns. If you have further questions, please do not hesitate to ask us.

Sincerely,

The Authors