What Makes a Good Time-series Forecasting Model? A Causal Perspective

• The theoretical results are not very clear to me. Please see the question part. Also, please add more background information, like how you define the generalization error.
• Can the method be applied to the classification setting?
• Do you make some assumptions on the distribution of the time series? Like no time-lag effect, the conditional distribution of V_i given its parents is invariant.
• Would the method still have advantages when $v_c | (v_i,v_s)$ is changing?

Despite the good explanation of the intuition, I think there were some areas over simplified (e.g., Section 4.2):

• how much does these method scale? Causal discovery problems often struggle with dozen of variables; Is this a limitation to training the transformers? is this a limitation on the applications (as the authors mentioned practical applicability ).
• What if the relationship between variables is not linear? Is it realistic to always make this assumption?
• The formal definition of Markov boundary claims that a set of variables is a markov boundary if the markov blanket is minimal, meaning no variable can be dropped without losing information. Hence, if the authors can find a new variable that can be dropped (based on colliders), isn’t that a contraction that the previous set was markov boundary?
• I appreciate the empirical evaluation, but there is some information missing that would help demonstrate the superiority of the proposed method;

Main argument

The role of causal relationships among variables in multivariate time series has not been well proved in this article. The obtained Markov boundary has been simply and rigidly used as mask mechanisms for attention scores. Although in section 4.2, it has been roughly proved that the hypothesis space can be constrained through conditional independence relationships to achieve better generalization, this still belongs to variable selection on the level of correlation relationships. Except for the DAG obtained by using the PC algorithm, the integration of the model with causality is not tight. The trained attention is considered to be able to learn which tokens are more worthy of attention and which ones need to be ignored. This study does not bring any new insights.

The experimental results cannot well prove that the method has a significant improvement, and some details are missing. For example, the model scale of CAIFormer is not mentioned in the paper. The content of the experimental part is insufficient, and it is difficult to draw conclusions.

There are many missing details:

1. It is important to express the existence of "collider provides additional independence relationships" before proposition 1. However, in proposition 1, when the condition set is given, Y is unexpectedly independent of X_i. Is this a conflict?
2. The differences in the attention matrix under different operations need to be considered (the original trained attention and the masked attention).
3. Why are the experimental results of this article a subset of the experimental results of iTransformer? The experimental results in different environments must be inconsistent. The experimental results are not rigorous.
4. The names of the variables in DAG are not given, lacking reliability to some extent.
5. the Markov boundary obtained according to the DAG does not consider its own variables, but from an autoregressive perspective it is important.
6. The difference in the ablation experiment effect is not significant, and more detailed results are needed in the experiment. For example, the average results of different prediction lengths should be shown separately.

The paper has many imprecise parts, here are a few:

1. For the experiment, a more detailed description of the settings is required.
2. The proof in section 4.2 needs to be more detailed.
3. There is a lack of comparison curves between the actuals (ground truth) and the predictions, and the performance cannot be intuitively presented.
4. What is the purpose of using PC and FCI for causal graph learning in the appendix? Does it help with the results of the paper?

Things to improve the paper that did not impact the score:

1. The experimental analysis needs to be more detailed.
2. The operator should be Hadamard product between the DAG and Attention, instead of add in figure 2.

1. while the paper deals with multivariate-time series forecasting, the causal framework presented and used is the one for iid data. Additionally the used algorithms, PC and FCI are algorithms for iid data mainly (while they can be adapted to time-series, by taking lag variables). The section on causality describes tools that are mainly used for iid data. For time series, they can be expanded but additional frameworks exist. For instance, the causal graph for a time-series problem can obviously be cyclic since feedback loops can appear. Under some simplifying assumptions, DAG and SEM can be used to describe causal graphs in a time-series setting by unwrapping and explicitly representing lagged variables. I feel that a proper discussion of causal discovery methods in time series settings is missing. Some references:

• The book "Elements of causal inference: foundations and learning algorithm" by Peters, Janzing, and Schölkopf. has a section on time series extension.
• "Discovering causal relations and equations from data" Camps-Valls et al. 2023 contains a review of time series causal discovery methods.

2. Also, a recently published paper (Neurips) https://arxiv.org/abs/2402.09891 tackles a similar problem, reaching a different conclusion. I understand that the theoretical argument is that causal information should help generalization and this is somehow a known fact. But how much this is useful in practice, especially with non-linear systems, is debatable. The only known method that tackles generalization that i know is anchor regression which is based on linearity assumptions. https://arxiv.org/abs/1801.06229.

3. the experiment section is somehow limited and the results are missing uncertainty.