Causal Discovery from Conditionally Stationary Time Series

We sincerely appreciate the reviewer’s feedback. Below, we address key concerns.

There is room for the clarity/simplification of the notations.

We appreciate this point and will revise the manuscript for clarity.

• Subscripts $\psi$ and $\phi$ denote parameters of the generative model and variational approximation, respectively. The generative model should always be indexed by $p_{\psi}$
• The outgoing structure $C_k^{i}$ contains variable indices, $1, \dots, N$, while $PA_{s_{t-1}}(t-1)$ refers to function arguments (e.g. $x_{t-1}^{(i)}$). For time series, we consider the time index when computing densities at each time step $t$, and this is why we use $t-1$ as an argument to $PA$.

We will make sure the above is revised and address all noted typos.

but I am concerned with the contributions.

Our theoretical results encompass three levels of generality.

• Markov Switching Models (m1-m3): Thm. A.6
• Conditionally stationary time series (m1-m4): Thm A.7
• SDCI (m1-m5): Cor. A.8.

Our paper explicitly studies the applications of SDCI to address nonstationarity. However, our scope in terms of theoretical results is broader, and our results from Thms. A.6 and A.7 could be of independent interest. Identifiability in regime-dependent time series is a very challenging problem with only very recent contributions [1], and importantly, our work makes no assumptions on state dependence.

Regarding empirical contributions, Gene Regulatory Network (GRN) inference methods are typically validated with semi-synthetic data that mimics the underlying biological system, as validation through real-world cell systems is very expensive. SDCI design agrees with GRN literature and its ability to model gene activation/deactivation could be impactful in this domain.

[1] Balsells-Rodas et al., "On the identifiability of switching dynamical systems." ICML 2024.

Theoretical Claims and Q1

All proofs are original. Theorem A.6, leverages Yakowitz & Spragins (1968) [2], also used in [1]. We note key differences:

• We allow dependencies from $x_{1:t}$ to $s_t$, strictly prohibited in [1].
• Our weaker assumptions require a novel proof strategy resolving permutation equivalence issues from Yakowitz & Spragins (1968) result.
• Our strategy addresses identifiability from conditional distributions $p(x_t| x_{1:t-1})$, while [1] considers the joint distribution $p(x_{1:T})$.

Identifiability in finite mixtures relies on function linear independence [2]. For Gaussian families, this holds if means or covariances are distinct. We ensure this by assumption (a1). We will clarify the above in the appendix, and plan to include a discussion comparing other works in the main document (see Reviewer Fbj4).

[2] Yakowitz and Spragins. "On the identifiability of finite mixtures." AMS 1968.

Reviewer’s questions:

Q2: Conditional effect labels improve scalability in conditionally stationary time series. SDCI, inspired by interactive systems (e.g. NRI, ACD), learns diverse graphs from limited types of interaction types $n_{\epsilon}$. Example: in sports data, different samples may have distinct graphs, but all interactions could be categorized as "no interaction," "aggressive," or "defensive."

Q3:

• Eq. (5): $f_{s_{t-1}}^{(j)}$ in is indexed by $s_{t-1}$ leading to $K^N$ functions definitions, which is prohibitively expensive.
• Eq. (8): We use GNNs with pairwise interactions and an aggregator function, reducing complexity, and we prove identifiability of the labels under this setting.
• SDCI considers (m1-m5), but we also prove identifiability considering weaker cases (m1-m4), allowing alternative solutions.

Q4:

• We require no knowledge of $K$, unlike most mixture model results on time series.
• This allows finding $K$ via model selection (assuming convergence to MLE), which is theoretically impossible if $K$ requires to be known.
• In practice, standard techniques such as the “elbow method” (used in K-means) can be applied. See the figure below for model selection with true $K=2$ in the springs and magnet data, where reconstruction MSE plateaus at $K=2$.

https://postimg.cc/k2rFZkBp

We plan to expand on selecting $K$ in the Experiments section, showcasing also $K=4$ in the springs and magnets data.

Actions

To improve our manuscript (see other Rebuttals for details), we aim to:

• Clarify notation and revise typos.
• Provide intuitions from [2] in our Proof for Thm. A.6.
• Contrast our method assumptions with prior works, emphasizing our broader theoretical scope.
• Expand the discussion on assumptions and implications to real-world data.
• Provide details for $K$ selection, and clarify for NBA data.
• Clarify scalability and include runtime analysis in Appendix.
• Include figures for increasing data sizes on fixed graph data in Appendix.
• Modify Figure 5b to include additional variables.

We sincerely appreciate the reviewer’s insights and believe these revisions will significantly improve the paper.

We sincerely appreciate the reviewer's thoughtful comments. Below, we address the key concerns raised in the review.

It is unclear how the proposed method is different from existing methods...

While prior works [1-3] also consider discrete latent variables for modeling nonstationary time series, our method has three key advantages:

• No assumption of known number of states:
  • Most previous works, including HMM-ICA [1] and CtrlNS [2], assume the number of states is known.
  • Our work leverages Yakowitz & Spragins (1968) [4], which enables identifiability without knowing the number of states. This allows for model selection to determine $K$, an aspect theoretically impossible in prior approaches.
• No assumptions on state dependency:
  • Prior works often assume that state transitions independent of observations (e.g., HMM-ICA [1], [3]).
  • Recently, NCRTL [2] models direct observation-to-state dependencies, but requires stronger assumptions (e.g., mechanism sparsity, known $K$).
  • Our proof technique requires no assumptions in terms of state dependencies, allowing feedback from observations.
• Regime-dependent identifiability
  • While we provide identifiability for SDCI in the main text (m1-m5), our results provide a broader scope with general identifiability results for regime-dependent time series (m1-m3) in Theorem A.6.
  • In Thm. A.6 we present a novel proof technique leveraging Yakowitz and Spragins (1968) [4]. This result can be of independent interest.
  • Strategies based on [1] cannot solve this problem, as their proof strategy prohibits explicit autoregressive dependencies from $x_{t-1}$ to $x_t$.

We will explicitly incorporate this discussion in the revised manuscript.

[1] Hälvä, Hermanni, and Aapo Hyvarinen. "Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series." UAI 2020.

[2] Song, Xiangchen, et al. "Causal temporal representation learning with nonstationary sparse transition." NeurIPS 2024.

[3] Balsells-Rodas, Carles, Yixin Wang, and Yingzhen Li. "On the identifiability of switching dynamical systems." ICML 2024.

[4] Yakowitz, Sidney J., and John D. Spragins. "On the identifiability of finite mixtures." AMS 1968.

it is still unclear how to ensure the assumptions hold?

Aligning data with assumptions is a key challenge in causal discovery and ICA. Our assumptions are as follows:

• (m1-m5): Our model design is inspired by interactive systems, and physical models (e.g. sum of forces).
• (a1) Unique Causal Structures Across States: Each variable must have distinct causal interactions per state.
• (a2) Functional Faithfulness: Discussed in lines 248–253. It can be achieved by using smooth functions (e.g. analytic functions), which are commonly assumed in EEG or Gene expression data.

For semi-synthetic datasets, the assumptions are simple to verify due to access to the groundtruth dynamics. In real-world data, expert knowledge is required. Assumption violations can break identifiability, which would invalidate interpretability. We discuss these issues in Section 6.3 and will expand on them in revision.

does the proposed algorithm satisfy all the required assumptions

Yes, SDCI is designed to incorporate these assumptions:

• (m1-m5) & (a2.1): Enforced via a GNN architecture and the no-edge effect (i.e. $w_{ijk}=0 \to f_0:= 0$).
• (a1): Date-dependent, independent of algorithm design.
• (a2.2): Ensured via analytic activation functions (e.g., Softplus, Cosine) in the edge-type mechanisms.

We will enhance the discussion on how SDCI aligns with these assumptions in the final version.

The number of variables (N=3, 5) is too small

See the figure below for an additional column with $N=10$ in Figure 5b. Increasing $N$ in time series causal discovery is challenging due to the temporal component. In our setting, the total data dimensionality is $N \cdot d$ ($N\cdot 4$ in springs and magnets), but complexity also increases with $K$ and $T$.

https://postimg.cc/mzznGYhz

Actions

To improve our manuscript (see other Rebuttals for details), we aim to:

• Clarify notation and revise typos.
• Provide intuitions from [2] in our Proof for Theorem A.6.
• Contrast our method assumptions with prior works, emphasizing our broader theoretical scope.
• Expand the discussion on assumptions and implications to real-world data.
• Provide details for $K$ selection, and clarify NBA data.
• Clarify scalability and include runtime analysis in Appendix.
• Include figures for increasing data sizes on fixed graph data in Appendix.
• Modify Figure 5b to include additional variables.

We believe these revisions will strengthen the paper and appreciate the reviewer’s constructive feedback.

We sincerely appreciate the reviewer’s time and thoughtful feedback. We are glad they find our framework innovative and acknowledge its significance in handling nonstationary time series. Below, we address the key concerns raised in the review.

... if the actual data cannot meet the premise, may lead to model failure.

SDCI is explicitly designed for conditionally stationary time series, with applications in interactive systems (e.g., sports) and Gene Regulatory Networks (GRNs). However, our theoretical results are broader, covering:

• Markov Switching Models (m1-m3): Theorem A.6.
• Conditionally stationary time series (m1-m4): Theorem A.7.
• SDCI (m1-m5): Corollary A.8.

We develop theoretical results starting from a very general case to our specific implementation, SDCI. These results, A.6 and A.7, provide a general foundation for regime-dependent causal discovery.

We would like to note that identifiability in regime-dependent time series is a very challenging problem with only recent contributions [1]. Importantly, our results make no assumptions on state dependencies, and introduce a novel proof technique based on classic finite mixture model theory (Thm. A.6). We will incorporate a discussion emphasizing our theoretical scope in contrast with related work (see Reviewer Fbj4 for additional details).

[1] Balsells-Rodas, et al. "On the identifiability of switching dynamical systems." ICML 2024.

… scalability of experimental validation is not fully demonstrated.

We acknowledge this concern and will include additional runtime analysis in the appendix to illustrate SDCI’s linear scaling with respect to $K$. We will also compare state inference times across “determined” and “recurrent” scenarios.

NBA data are only mentioned in context, lacking detailed experimental design and outcome analysis

We provide details on training in Appendix B.4 (lines 1097-1099) and dataset preprocessing in Appendix C.3. These details were deferred to the appendix to focus on the interpretability of learned structures.

… this paper does not need to know the number of K mentioned in Remark 4.2, but in experiment 3, the movement trajectory needs to do two groups …

Our identifiability results do not require knowledge of $K$, which implies we can determine $K$ via model selection (assuming convergence to MLE). This is a significant contribution, as other related works require knowledge of $K$ (with exception of [1]) and cannot rely on this idea.

However, selecting $K$ in practice is a well-known challenge even in non-temporal settings. We can use the “elbow method” (as in K-means), or similar heuristics. Below we show results for synthetic springs and magnets with $K=2$ where reconstruction MSE plateaus for $K=2$.

https://postimg.cc/k2rFZkBp

Our discussion regarding results on $K=2$ and $K=4$ aimed to show that both values provide meaningful interpretations. Figure 9 (forecasting results) shows similar performance for $K=2$ and $K=4$, suggesting $K=4$ overfits the data in this scenario.

We acknowledge selecting $K$ is an important challenge, and will

• include a discussion on how to select $K$ in practice, showing figures similar to the above, including a setting with $K=4$.
• discuss how this applies to NBA data.

the targeted validation is not designed in the experiment (such as the convergence test under different data quantities) …

Figure 4 shows that SDCI achieves 100% $F_1$ score when the graphs are fixed, demonstrating consistency empirically. For samples with different graphs, the challenge relies on the approximate posterior design $q(W|x_{1:T})$. The identifiability results support structure learning from purely unsupervised data, which for this setup is very challenging.

We provide dataset size variation results for observed states in Appendix E.1, Figure 14. Below, we show similar results for fixed graphs on "Recurrent states" with $N=5$ and $K=2$, which will be included in the appendix:

Actions

To improve our manuscript (see other Rebuttals for details), we aim to:

• Clarify notation and revise typos.
• Provide intuitions from [2] in our Proof for Theorem A.6.
• Contrast our method assumptions with prior works, emphasizing our broader theoretical scope.
• Expand the discussion on assumptions and implications to real-world data.
• Provide details for $K$ selection, and clarify NBA data.
• Clarify scalability and include runtime analysis in Appendix.
• Include figures for increasing data sizes on fixed graph data in Appendix.
• Modify Figure 5b to include additional variables.

We appreciate the reviewer’s constructive feedback and believe these revisions will further improve the paper.