Exploring Neural Granger Causality with xLSTMs: Unveiling Temporal Dependencies in Complex Data

Dear Reviewer K18p,

Thank you for recognizing the work as clearly written and original.

We gladly clarify the raised comments and questions:

Q1a: Granger causality is not yet solved. For instance, looking at Tab. 2, we see that few methods even surpass just 70%, and all fall far short of the desired detection accuracies of 90+%. GC-xLSTM contributes toward reaching this by integrating xLSTM modules into a novel optimization procedure specifically designed to discover sparse inputs.

Q1b: GC-xLSTM improves the detection accuracy of Granger causal graphs, as the varied empirical results shown specifically in Tab. 1 and 2 confirm. However, this does not yet show which of the two major contributions, (a) integrating xLSTMs and (b) the joint optimization procedure for sparse feature selection, contributes how much. We thus perform a respective ablation study on two datasets. Results are shown below.

First, we replace the xLSTM block in the architecture with a standard LSTM layer, keeping the rest of the architecture unchanged. We can clearly see from (base) -> (a), that the performance drops substantially without the modeling capabilities of the xLSTM blocks.

Second, we employ the baseline of Group Lasso (Simon and Tibshirani, 2012) instead of our novel procedure for optimizing for strict input sparsity. Again, we can see from (base) -> (b) that our strong results indeed stem from the contribution of the new optimization procedure. This new procedure is a contribution beyond GC-xLSTM and could, in the future, be used in other models too.

Results on Lorenz-96 ($F=40$):

Results on fMRI:

Q2: An overview would help maintain a more complete image. We thus added the following table to Appendix C, amending the missing numbers:

Q3: We agree that scaling behavior is one of the key questions for any Granger causality method. Usually, though, the main challenge lies in scaling the number of variates $V$ (cf. ll. 97ff). We, therefore, specifically analyze it in Appendix F and already successfully apply the flexible GC-xLSTM framework to time series lengths between $T=56$ and $T=4136$ (see table directly above). Lastly, we also consider a wide range of samples from $N=1$ to $N=2527$. Note that here larger numbers of variates $N$ can actually be beneficial, since it reduces the likelihood of overfitting on smaller datasets.

Thank you again for your thoughtful review. We hope it answered all outstanding questions. Specifically, having provided additional results on the main concern (significance), we would greatly appreciate it if you considered raising your score to reflect this.

Best

The Authors

References:

• Simon, N., & Tibshirani, R. (2012). Standardization and the group lasso penalty. Statistica Sinica, 22(3), 983–1001. https://doi.org/10.5705/ss.2011.075

Dear Reviewer TkdF,

Thank you for recognizing the work as well-written, as well as the diverse evaluation and superior performance.

We gladly clarify the raised comments and questions:

W1: One strength of the theoretical analysis is that parts of it, indeed, apply to other works as well. You are correct in that models such as cLSTM would also, in theory, be able to learn GC relationships, which we never question. We merely posit that due to the increased modeling capabilities of xLSTMs, more complex relationships can be learned (Beck et al., 2024). We validate this hypothesis with empirical results. We also note that a major contribution besides the xLSTM is the novel optimization scheme, going beyond cLSTM etc.

W2: Comparing very different algorithms by “integrating out” their hyperparameter choices, such as the sparsity penalty $\lambda$, is, indeed, challenging. They are comparable if the range of HPs each is complete, i.e., that the range covers both extremes where the behaviour saturates. This is the case for our AUROC computation. Also, note that the scores for cLSTM and cMLP were obtained analogously to our work. However, particularly when omitting this metric, GC-xLSTM clearly outperforms all baselines (cf. Tab. 1 & 2).

W3/Q1: We agree that an ablation study to more clearly isolate the empirical contribution of these two components would be very valuable. We thus perform an ablation study of (a) the use of xLSTMs and (b) the joint optimisation strategy. The results are provided for two datasets and shown below.

First, we replace the xLSTM block in the architecture with a standard LSTM layer, keeping the rest of the architecture unchanged. We can clearly see from (base) -> (a), that the performance drops substantially without the modeling capabilities of the xLSTM blocks.

Second, we employ the baseline of Group Lasso (Simon and Tibshirani, 2012) instead of our novel procedure for optimizing for strict input sparsity. Again, we can see from (base) -> (b) that our strong results indeed stem from the contribution of the new optimization procedure.

Results on Lorenz-96 ($F=40$):

Results on fMRI:

W4: Beyond the innovative and effective use of xLSTMs, we also propose a novel optimization procedure (Sec. 3.2, Alg. 1, Code), which we discuss in detail and validate empirically.

Minor typos: Thank you, they are now fixed.

Q2: Choosing hyperparameters is a common challenge in machine learning. However, $\lambda$ is rather well-behaved in two regards. Firstly, due to the adaptive joint optimization of both the sparsity projection and the forecast xLSTM, in many cases, results are rather stable across choices of $\lambda$. This really shows with the rather high AUROC scores of 88.0 and 99.3, which are computed over sweeps of $\lambda$ (cf. ll. 244ff). Secondly, when $\lambda$ does have a large effect, we can often visually inspect the resulting graph and coordinate with domain experts to determine the desired level of sparsity. As Fig. 4 shows, choosing $\lambda$ in this setting is not about model performance, but instead about the desired connectedness of the resulting graph, and therefore, an inherent user preference much like the threshold in other Granger causal detection methods.

Thank you again for your thoughtful review. We hope it answered all outstanding questions. Specifically, having clarified the questions regarding the theory, the AUROC metric, and the hyperparameter choice, as well as providing the suggested ablation study, we would greatly appreciate it if you considered raising your score to reflect this.

Best

The Authors

References:

• Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael K. Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended Long Short-Term Memory. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, November 2024.
• Simon, N., & Tibshirani, R. (2012). Standardization and the group lasso penalty. Statistica Sinica, 22(3), 983–1001. https://doi.org/10.5705/ss.2011.075

Thanks for engaging with the rebuttal.

• Regarding the AUC, we agree that a bit of non-linearity can be important, and that is why the metric of choice is the balanced accuracy. The AUC metric was included for completeness reasons.

• To choose the hyperparameters for group lasso, we do a sweep over lambda values. We also introduce a separate validation set for picking the best weights to prevent overfitting.

We agree that the optimization procedure is central to our approach, and thus we highlight it as a major contribution (points 2 and 3 in key contributions on p. 2). Furthermore, please take a look at Fig. 8a in Appendix E (p. 16) that demonstrates the ability of our model to self-select a version of weights that gives the best GC accuracy.

Dear Reviewer w7eG,

Thank you for recognizing the work as well-written and technically sound, as well as the diverse evaluation of the method and baselines.

We gladly clarify the raised comments and questions:

Regarding Significance and Originality: We want to point out that the contribution of this work is twofold: While the exploration of the xLSTM architecture of GC detection is one part, as identified by the reviewer, the other part is the novel optimization scheme introduced and analyzed in detail in Sec. 3.2. It is summarized in Algorithm 1 and implemented in the provided code. The latter is a contribution beyond just xLSTMs. To more clearly isolate the empirical contribution of these two components, we perform ablation studies of (a) the use of xLSTMs and (b) the joint optimisation strategy. The results are provided for two datasets and shown below.

First, we replace the xLSTM block in the architecture with a standard LSTM layer, keeping the rest of the architecture unchanged. We can clearly see from (base) -> (a), that the performance drops substantially without the modeling capabilities of the xLSTM blocks.

Second, we employ the baseline of Group Lasso (Simon and Tibshirani, 2012) instead of our novel procedure for optimizing for strict input sparsity. Again, we can see from (base) -> (b) that our strong results indeed stem from the contribution of the new optimization procedure.

Results on Lorenz-96 ($F=40$):

Results on fMRI:

Q1: When choosing model families, there often is a trade-off between efficiency and fidelity. Indeed, state space models (SSMs) make for highly efficient time series models. However, xLSTM-based models tend to generally provide better modeling accuracy while still limiting the compute necessary (see, e.g., Beck et al., 2024). This is also confirmed empirically by the effectiveness of GC-xLSTM. We will ensure to make this discussion more explicit and balanced in the final version of the paper, where § ll. 30ff currently only discusses why no Transformers have been employed.

Q2: We first note that the xLSTM is a recurrent model, where forecasting 10 steps into the future auto-regressively is quite typical. While a look-back of 10 steps is indeed little compared to many long-term time-series forecasting methods, it is a lot in the regime of (Granger) causal discovery we contribute to. For instance, the seminal work of Tank et al. (2022) introducing Neural Granger Causality mostly uses a lookback of 2 or 5. Marcinkevičs and Vogt (2021) use values of 1 or 5. We did not scale beyond that, as higher numbers were not necessary to sufficiently forecast the datasets we investigated. Also, not that this regime is beneficial in many real-world datasets, where limits in length do not allow for substantially larger look-backs or forecast horizons.

Q3: Firstly, regarding outperforming other baselines: We quantitatively compare the results of GC-xLSTM to those of baselines on the Lorentz-96 dataset in Tab. 1 and on the fMRI dataset in Tab. 2. For Lorentz-96, GC-xLSTM always wins in both Accuracy and Balanced Accuracy while staying very competitive for AUROC. We further reiterate that this AUROC score is computed by sweeping over different sparsity configurations $\lambda$ ll. 244ff, essentially removing that hyperparameter. For fMRI, GC-xLSTM outperforms all five baselines.

Thank you again for your thoughtful review. We hope it answered all outstanding questions. Specifically, having clarified the questions regarding significance and originality, as well as providing the suggested ablation study, we would greatly appreciate it if you considered raising your score to reflect this.

Best

The Authors

References:

• Tank, A., Covert, I., Foti, N., Shojaie, A., & Fox, E. B. (2022). Neural Granger causality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8), https://doi.org/10.1109/TPAMI.2021.3065601.
• Marcinkevičs, R., & Vogt, J. E. (2021, January). Interpretable models for Granger causality using self-explaining neural networks. In International Conference on Learning Representations.
• Simon, N., & Tibshirani, R. (2012). Standardization and the group lasso penalty. Statistica Sinica, 22(3), 983–1001. https://doi.org/10.5705/ss.2011.075
• Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024). xLSTM: Extended long short-term memory. Advances in Neural Information Processing Systems, 37, 107547-107603.

Dear Reviewer dBaa,

Thank you for recognizing the clear description of the novel optimization scheme and comprehensive evaluation on a diverse set of benchmarks.

We gladly clarify the raised comments and questions:

Q1: We agree that this is an important ablation and since added it to the paper for the datasets Lorenz-96 and fMRI. Specifically, we employ the baseline of Group Lasso (Simon and Tibshirani, 2012) instead of our novel procedure for optimizing for strict input sparsity. We can see from (base) -> (ablated) that our strong results indeed stem from the contribution of the novel optimization procedure. It provides substantial and consistent improvements over the prior Group Lasso procedure.

Results on Lorenz-96 ($F=40$):

Results on fMRI:

Q2: The motivation for employing xLSTM-based models is more than just capturing long-term dependencies, and also their generally improved modeling accuracy while still limiting the compute necessary (see, e.g., Beck et al., 2024). This is also confirmed empirically by the effectiveness of GC-xLSTM. Specifically, for the Company Fundamentals dataset, the model uses 40 time steps (10 years) instead of 10 steps (2.5 years), which we deem as already a long-term span in (Granger) causal discovery. See also the next paragraph for empirical proof of the benefit of using xLSTM modules over the classic LSTMs.

Q3: To specifically show the benefit of this improved modeling capability, we ablated GC-xLSTM by replacing the xLSTM with a classic LSTM. The following table shows the results of replacing the xLSTM block in the architecture with a standard LSTM layer, keeping the rest of the architecture unchanged. We can clearly see from (base) -> (ablated), that the performance drops substantially without the modeling capabilities of the xLSTM blocks, especially in the case of the real-world data set of fMRI.

Results on Lorenz-96 ($F=40$):

Results on fMRI:

Thank you again for your thoughtful review. We hope it answered all outstanding questions. Specifically, having clarified the questions regarding the long-term dependencies, as well as providing the suggested ablation study, we would greatly appreciate it if you considered raising your score to reflect this.

Best

The Authors

References:

• Simon, N., & Tibshirani, R. (2012). Standardization and the group lasso penalty. Statistica Sinica, 22(3), 983–1001. https://doi.org/10.5705/ss.2011.075

I want to thank the reviewers for their thorough review. In particular, I appreciate the the additional experiments that, in my sense, provide enough evidence for the value of the xLSTM and the joint optimization. I encourage the authors to include it in their final manuscript. I'm updating my score accordingly.

Dear Reviewer jaiu,

Thank you for recognizing the clear writing as well as the wide set of experiments on synthetic and real-world data.

We gladly clarify the raised comments and questions:

W1: In the causal discovery community, guarantees on identifiability are indeed cornerstones of many important works. However, we want to point out that (classic) causality and our focus, Granger causality, are different fields closely related in purpose, not methodology (see also ll. 26-29). Granger causality is a statistical test that does not yield Pearlian causal graphs and is centered on the prediction task. Thus, Granger causality can indeed be helpful in identifying potential causal relationships, but it doesn't necessarily imply true causal mechanisms in the classical sense. There has been some work on equating Granger causality with classical structural causality, such as by Bodik and Paluš (2024), where, however, the focus was on extreme events. Thus, we hope that the reviewer will appreciate that proving “identification” in case of Granger causality is an open, non-trivial problem and warrants a completely separate study.

In GC, identifiability is defined (see Def. 1) to be given when we can correctly learn the underlying time-evolving process $g$. This is the case whenever we use models with sufficient capacity and flexibility (universal approximators), which we discuss being the case for GC-xLSTM in Sec. 3.3 and extend on in Appendix B. We further note that even the seminal work of Tank et al. (2022), having introduced Neural Granger Causality, recognizes this analysis as the limit of current theoretical guarantees in their conclusion section.

We do agree that this is indeed important future work. However, it goes beyond this work and overarches the entire family of Neural Granger Causality models, such as cMLP, cLSTM, GC-KAN, and our GC-xLSTM. We thus mention it in ll. 338-341.

W2: Adding the publication year to Tab. 1 is a very good suggestion. We thus added the following column and will reorder the rows to follow an ascending timeline.

As can be seen, we compare against recent works, such as cLSTM, cMLP, and GC-KAN. We will extend the manuscript with a more detailed description of the methods and how they differ from xLSTM-Mixer.

W3: This appears to be a misunderstanding. We never claim that models such as cMLP “fail to capture interactions”. Instead, we state that they “may not capture interactions between time series and external factors as effectively as xLSTMs [emphasis added]” in ll. 48f. This improved modeling capability was also shown by Beck et al. (2024). This modeling capability is important for uncovering the data-generating process with sufficient fidelity, as discussed in W1 and Sec. 3.3. The improvement is most clearly shown in the results of Tab. 2, where the benchmark is sufficiently challenging to differentiate the models based on that. There, methods with simpler models (specifically, GVAR, VAR, cMLP, and cLSTM) discover far fewer correct edges than more advanced models such as the attention-based TCDF and our GC-xLSTM (overall best).

Thank you again for your thoughtful review. We hope it answered all outstanding questions. Specifically, having clarified the questions regarding identifiability, choice of baselines, and modeling complex temporal dependencies, we would greatly appreciate it if you considered raising your score to reflect this.

Best

The Authors

References:

• Tank, A., Covert, I., Foti, N., Shojaie, A., & Fox, E. B. (2022). Neural Granger causality. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(8), https://doi.org/10.1109/TPAMI.2021.3065601.
• Beck, M., Pöppel, K., Spanring, M., Auer, A., Prudnikova, O., Kopp, M., Klambauer, G., Brandstetter, J., & Hochreiter, S. (2024). xLSTM: Extended long short-term memory. Advances in Neural Information Processing Systems, 37, 107547-107603. -Bodik, J., Paluš, M. & Pawlas, Z. Causality in extremes of time series. Extremes 27, 67–121 (2024). https://doi.org/10.1007/s10687-023-00479-5