Doubly Robust Structure Identification from Temporal Data

We appreciate the reviewer for evaluating our paper and acknowledging its novelty. We address each of your concerns in the following and look forward to further discussion if you have additional feedback. We are committed to making the necessary revisions to enhance the clarity, accessibility, and overall coherence of our writing.

Generally speaking, the paper is hard to follow and the goals of the proposed method are unclear given the entire paper. The approach is motivated as to be for (time-varying) causal feature selection for a target variable. However, it is claimed in the paper that it can also be used for full causal discovery, but it's not clear the evaluation is for either causal feature selection or full causal discovery.

Thank you for your comment! If you have specific suggestions on how to improve the write-up, we will gladly incorporate your suggestions in our manuscript.

To answer your question, in the experiments, we consider both causal feature selection and causal discovery. The reason why we consider both scenarios is to ensure a fair comparison with competitors. Note that causal feature selection for timeseries data is relatively underexplored. Hence, we compared our method with causal structure learning algorithms to showcase performance of our method with stronger evidence. In Section 6.1 we focus on causal discovery, whereas in Section 6.2 / Appendix J.1 we focus on causal feature selection. We are happy to make this point clearer in our submission.

The task performed in the evaluation section is not described at all. Presumably the task should be causal feature selection, but all the reader is told is the metric used for evaluation is AUC, which doesn't sound like we're evaluating the correct set of causal features. Furthermore, the methods used in the evaluation section do not appear to be causal feature selection methods and are different from the related methods mentioned earlier in the paper.

Yes, thank you for pointing it out! Our experimental framework essentially replicates previous related work. In our experimental section, we report on all state-of-the-art benchmarks for the DREAM3 datasets. Furthermore, the use of the AUC is consistent with previous related work [1] (state-of-the-art). There are several reasons for the usage of AUC other than other metrics like accuracy, F1 score, recall, etc, when comparing the performance of statistical algorithms that involve statistical tests. All of those metrics need a p-value thresholding which may lead to the following potential problems in the comparisons: vulnerability to class imbalance, task-specific threshold choices, model sensitivity to false positives and negatives, sensitivity to class probabilities, etc.

Aside from the above confusion about the evaluation section, the empirical work is minimal in general and standard errors are not included.

Yes, thank you for pointing it out. Our experimental framework considers a variety of baselines, and it is consistent with previous related work [1,2,3,4]. Throughout our experimental investigaion, we did not find that standard errors play a significant role, as it was also observed in related work [1]. Please inform us of other experiments/datasets/plots that you would think that our manuscript can benefit from, so we can incorporate your comments in our experiment section.

Can the authors explain the evaluation task? Is it causal feature selection? What is the actual target that AUC is reported for?

In the semi-synthetic experiment involving the Dream3 dataset (section 6.1, Appendix J.2), our objective is full causal discovery. Each of the five tasks within Dream3 involves observing the evolution of a system comprising 100 genes over time, subjected to 46 different perturbations. The ground truth causal graph delineating the relationships among these genes is provided. Out of the $100*99$ possible directed edges among the genes, some are indeed present, indicating direct causal influence, while others are not. This forms the actual target for AUROC calculations.

In the additional synthetic experiments (Appendix J.1), the focus shifts to causal feature selection. Here, the ground truth used in the AUROC metric is represented by the $\Delta \times m$ binary adjacency matrix $\Sigma^{\mathbf{Y}}$, where $\Delta$ signifies the lag, and $m$ represents the number of potential causes."

We express our gratitude for reviewer's interest and constructive feedback regarding the paper’s clarity. We address your concerns in the following. We look forward to your further feedback to make sure the contributions of the paper are clarified clearly.

The paper focuses on time series data but there is no statistical analysis focusing on time series data. For example, on page 7, it is stated that "Under mild conditions on the convergence of $g^0_j$, $g^i_j$, and $\alpha_j^0, \alpha_j^i$, the quantity $\theta^0 - \theta^i$ has $\sqrt{{n}}$-consistency" and that "We refer the reader to Chernozhukov et al. (2022; 2018) for a proof of the $\sqrt{{n}}$-consistency for estimates as $\theta^0$ and $\theta^i$." I do not think the cited references deal with time series data directly. It is disappointing that the paper does not provide any extensive treatment of time series analysis.

Please note that, even though the data has time series structure, but i.i.d. trajectories of the data are provided and each trajectory can be treated as a single draw from an i.i.d. distribution. Hence, the strong consistency of our method follows directly from Chernozhukov et al. (2022; 2018). In our paper, we only provide an informal discussion on this point, to avoid a complicated notation that would have made the paper difficult to read. However, we are happy to provide a formal statment and an in-depth discussion of this point, if required. It is important to note that double ML can be extended to handle data of stochastic nature (Markovian non-i.i.d.) by applying it iteratively in a backward order, starting from the end of the process and moving towards the beginning. At each time step, following the doubly robust estimation, residuals of causal effects are computed to update the auxiliary random variables from the preceding time step. It can be demonstrated that the resulting scores derived from these auxiliary random variables satisfy the Neyman Orthogonality condition. This way the Markovian structure of the stochastic process at each step is cancelled out. The algorithm then proceeds to the previous time step. For further details, please refer to [1]. A similar approach can be employed here with appropriate adjustments, but delving into those modifications is beyond the scope of the main message conveyed in this manuscript.

Lemma A.1 claims that conditional mean independence in part 1 is equivalent to the conditional dependence in part 2. This seems mainly driven by Axiom (A) where the error is exogenous independent noise. I feel that this is a rather restricted setting. For example, suppose that Y is the time series of financial returns (e.g., S&P 500) and X is the causal factor that does not affect the conditional mean of returns but does affect the conditional variance of returns (typically called volatility in finance). It seems that the framework in the current paper excludes this kind of scenario. It is unclear to me what sense Axiom (A) is necessary; related to this point, Appendix A.3 is difficult to understand (see question 1 below).

Axiom (A) essentially requires that no confounding effect exists between the target variable Y and the causal parents. This assuption is essential for identifiability. As you correctly observe, this setting is restrictive. However, without any assumption we simply cannot get identifiability. Conditional mean independence is a necessary assumption in this setting. In other words, if this assumption fails, as in the example that you provide, than no method will provably be able to recover the true underlying causal structure due to identifiability issues. Please refer to appendix A.3 for an example of this identifiability problem without Axiom (A).

Appendix A.3 is difficult to understand. What are roles of $W_t$ and $Z_t$? $\Sigma$ is not a positive definite matrix here and seems too irregular. Some further comments would be useful.

$W_t$ and $Z_t$ are simply two time series, much like $X_t$ and $Y_t$. We use different notation, to highlight that the time series are generated differently. The matrix $\Sigma$ is a 2x2 positive semi-definite matrix, with eigenvalues lambda=2 and lambda=0. We hope this clarifies the point?

We sincerely appreciate the reviewer’s recognition regarding the significance of the contributions made by this research in the challenging domain of causal structure learning for time-series under highly general assumptions.

In the contribution, the authors claim that the proposed algorithm can be used for full causal discovery under some assumptions. The related discussion in section 5.2 is limited without details.

Thanks a lot for point this out! This simple reduction was not included in the interest of space and it goes as the following:

If the underlying causal structure doesn't have hidden confounders (noted as "fully-observed" in the manuscript) and doesn't contain any cycles (noted as "acyclic" in the manuscript), then we can iteratively select each node of the causal graph as our target variable and apply our causal feature selction algorithm to find its direct causes. These two conditions are necessary for the Axiom (A) to be true when any node of the causal graph could be selected as the target variable. In the end, the whole causal structure will be discovered.

We will make sure to clarify further in the final version of the manuscript.

In principle, the approach adheres to steps (1) through (4) in section 4.2, yet the practical algorithm has been adjusted to account for the time-consuming nature of "large instances." While the approach outline aligns with the proven theorem, a gap exists between the outlined approach and the modified algorithm. Is it feasible to implement the approach strictly in smaller instances, adhering to steps (1) through (4)? Furthermore, what does "large instances" imply in this context?

Thanks a lot for your precise and detailed question! When we mention "large instances", we are specifically referring to cases where the number of nodes, denoted as $m$, is substantial. The principled approach necessitates $\mathcal{O}(m)$ regressions, which is linear in the number of nodes. However, by incorporating the zero-masking heuristic, we can optimize the process, reducing the required regressions to just $\mathcal{O}(1)$. This proves particularly advantageous in situations where computational resources are limited, but the causal graph is huge. It's essential to highlight that, as demonstrated in Appendix A.7, there is no discernible difference between the principled approach and the one utilizing the heuristic when the underlying estimator is linear regression.

There is no real-world application provided in the paper.

Indeed, thank you for highlighting this consideration. In previous related works [2,3,4,5], Dream3 was employed as a gold-standard benchmark which is of systems' biology nature. However, we acknowledge that over-tuning causal discovery algorithms exclusively for a single dataset could present potential issues. As you aptly observe, other datasets might be pertinent in this context. We are open to the inclusion of any additional datasets you may suggest.

Regarding the baselines, from my understanding, some of them are designed for full causal discovery, encompassing the detection of causes for target variables and beyond. In contrast, the proposed algorithm primarily focuses on feature detection. In the experiment section, are there any specific modifications necessary to ensure a fair comparison?

There are no modifications necessary to ensure fair comparison. The reason is that our algorithm naturally extends to full causal discovery, without any modifications, by simply treating each variable as a target. The only subtle issue that could arise is the following: since a two-sample Student’s t-test is used to determine whether a particular covariate is a causal parent of the target, then a multiple hypothesis testing issue could arise, because we apply this procedure for every pair of variables. However, we do not face such a problem here because, according to the standard practice, the AUROC metric is used for evaluation in the Dream 3 benchmark. For this metric, we simply sort the predictions about existence/absence of an edge; that is, only the relative order matters.

"In equation 3, what is n? Should it be k?" Sorry that I made a mistake, and it should be equation 4.

Thank you for the detailed clarification; my questions have been well answered.

Thank you for pointing this out. Indeed, $\theta^0$ and $\theta^i$ are sample estimates of the true parameters $\theta^0_0 = \mathbb{E}[Y_T \cdot g^0_0 ]$ and $\theta^i_0 = \mathbb{E}[Y_T \cdot g^i_0 ]$ respectively and of course their distribution depends on $n$. The only reason we did not make this dependence explicit in the notation was simplicity, in order not to overload variable symbols with too many subscripts and superscripts.

Thank you very much for taking the time to review our work and recognize its merits!

The paper did not analyze or give an intuition why the proposed method allows the existence of cycles or hidden confoundings. Why does the proposed method allow the existence of cycles or hidden confounding?

To give an intuition of why we can handle hidden confounding and cycles, denote with $Y$ the target time series and with $\\{ X^1, \dots, X^n \\}$ the observed time series that are potential causes for $Y$. Using Axioms $A$-$C$, we prove that in this case, performing an intervention on all the variables $\\{ X^1, \dots, X^n \\}$ is equivalent to conditioning (this fact is specific to our causal model and it is not true in general). This type of conditioning "cancels out" the effect of cycles or hidden confounding on $\\{ X^1, \dots, X^n \\}$.

Thank you for enumerating the strengths of our work. We address each of your concerns in the following and look forward to further discussion if you have additional feedback.

Regarding the "stationary causal relation" assumption, you mentioned that the results could potentially apply to models that do not meet this axiom. Have you formally demonstrated this claim in any specific section, or are you implying that the proof of Theorem 4.1 does not rely on this assumption?

Under the stationarity assumption, we can test direct causal effects, by focusing on a target $Y_T$ for a fixed time T. If stationarity does not hold, then we will have to perform the test on $Y_t$, for all time steps t. This fact follows directly from our proof.

How do you identify cyclic structures? Does Algorithm 1 have the capability to detect cyclic structures, and does this imply the presence of confounders?

Our algoirthm only identifies direct causal effects on a target variable of interest. It does not identify cycilc structure. However, as detailed in Axioms (A)-(B), our algorithm works even under confounding or cycles among the covariates. In other words, even if hidden confounders and cycles are present, we are still able to uncover the direct causes.

In your method, is the time lag $k$ fixed, or does it remain stationary but vary among different variables?

In general, we only assume stationarity, and $k$ may vary among different variables. However, in our experimental framework we set $k$ to always be the same for simplicity. Specifically, in all our experiments, time lag is equal to 2. This is quite a standard value accross the literature [1,2,3,4].

[1] Gong, Wenbo, Joel Jennings, Cheng Zhang, and Nick Pawlowski. “Rhino: Deep Causal Temporal Relationship Learning with History-dependent Noise.” In The Eleventh International Conference on Learning Representations. 2022.

[2] Saurabh Khanna and Vincent YF Tan. Economy statistical recurrent units for inferring nonlinear granger causality. arXiv preprint arXiv:1911.09879, 2019.

[3] Roxana Pamfil, Nisara Sriwattanaworachai, Shaan Desai, Philip Pilgerstorfer, Konstantinos Geor- gatzis, Paul Beaumont, and Bryon Aragam. Dynotears: Structure learning from time-series data. In International Conference on Artificial Intelligence and Statistics, pp. 1595–1605. PMLR, 2020.

[4] Alex Tank, Ian Covert, Nicholas Foti, Ali Shojaie, and Emily Fox. Neural granger causality for nonlinear time series. stat, 1050:16, 2018.