Rethinking Channel Dependence for Multivariate Time Series Forecasting: Learning from Leading Indicators

Thanks for your time and effort in reviewing our paper! We have provided more explanations below to better clarify our work.

Q1: The experiment shows better performance and improvement for CI methods compared with CD methods. Could you provide possible reasons?

A1: Let "$>$" mean "outperforms". The following is the rationale for the experimental results.

$\text{Why CI backbones} > \text{CD backbones}?$

Crossformer focuses on variables that share similarities at the same time but ignores leading indicators that look different at the same time (as illustrated in Figure 1) due to a large leading step. MTGNN learns a fixed set of indicators and overlooks the varying leading steps. Thus MTGNN can easily overfit training data, of which the lead-lag relationships are different from test data (see Figure 7-8). Other CD backbones (e.g., Autoformer and FEDformer) simply mix up all variables and suffer from noise.

$\text{Why CI+LIFT}>\text{CI} > \text{CD+LIFT}?$

This happens when $CI \gg CD$ and the performance gap is too large for LIFT to fill. The reason is that LIFT is affected by the error from the backbone. As defined in Eq. (5), the shifted window of leading indicators is padded by a part of the backbone's forecasts when the leading step is smaller than the forecast horizon.

$\text{Why LIFT achieves higher relative improvement on CI than CD}?$

As both CD backbones and the LIFT module exploit dependence between variables, their leveraged dependence information can overlap. By contrast, CI backbones treat each variate independently and hence benefit more from the channel dependence exploited by LIFT.

---

Q2: Provide a formal definition of locally stationary lead-lag relation. Is a lead-lag relationship inferred whenever we observe relatively larger cross-correlation?

A2: Formally, there is a locally stationary lead-lag relationship between variate $i$ and $j$ at time $t$ if \begin{aligned} \exists \delta >0,\ &D(\mathcal X^{(j)} _{t+1:t+H}, \mathcal X^{(i)} _{t+1-\delta:t+H-\delta})\approx D(\mathcal X^{(j)} _{t-L+1:t}, \mathcal X^{(i)} _{t-L+1-\delta:t-\delta})\approx 0 \\ \wedge\exists t'> t,\ &D(\mathcal X^{(j)} _{t'+1:t'+H}, \mathcal X^{(i)} _{t'+1-\delta:t'+H-\delta}) \not\approx 0, \end{aligned} where $D(\mathcal W,\mathcal W')$ is the distance between two windows $\mathcal W$ and $\mathcal W'$, and $D(\cdot,\cdot)$ can be estimated by a neural network or a non-parametric metric (e.g., $1-\text{cross-correlation coefficient}$). Furthermore, if there exists $i\neq j$ such that $D(\mathcal X^{(j)} _{t+1:t+H}, \mathcal X^{(i)} _{t+1:t+H})\approx 0$, we dismiss variate $i$ as it is not a leading indicator and offers no advance information.

Since most time series are not extremely non-stationary, we assume the lead-lag relationship keeps stationary across the lookback and horizon windows, i.e., $\mathcal X^{(j)} _{t-L+1:t}$ and $\mathcal X^{(j)} _{t+1:t+H}$ has the same leading indicator $i$ with the leading step close to $\delta$.

A lead-lag relationship is not inferred by the correlation-based estimation alone. The motivation of using cross-correlation is to estimate all variates as fast as possible and filter out most noisy variates. A large correlation between two lookback windows just suggests that the indicator can serve as a potential reference for the horizon window. We rely on the Lead-aware Refiner which is the key component to exploit the advance information from potential leading indicators. The Lead-aware Refiner is trained to learn how much we can trust the suggestion and how we leverage the potential leading indicators.

---

Q3: Is the algorithm's applicability confined to variables that share a linear relationship?

A3: We do not constrain the relationship to be linear. We just use cross-correlations to roughly yet quickly select a subset of variates. Then, our Adaptive Frequency Mixer, a non-linear neural network, is capable of modeling complex relationships within the small subset, instead of costly modeling relationships among all variates.

Besides, the correlation-based estimation also allows some non-linear relationships between two windows $\mathcal W$ and $\mathcal W'$, e.g., $\mathcal W=(\mathcal W')^3$, as long as the variates share similar trends. In case the subset overlooks arbitrarily complex relationships, e.g., $\mathcal W=(\mathcal W')^2$, an alternative implementation of the distance function $D(\cdot, \cdot)$ is using another neural network (e.g., $D(\mathcal W, \mathcal W')=f _{1}(\mathcal W)^\top f _2(\mathcal W')$, where $f _{1}$ and $f _{2}$ are two different MLPs and trained by gradient descent).

Dear Reviewer 4A7m,

Thank you again for your dedication to reviewing our paper! As the discussion period is coming to an end very soon, we are looking forward to any feedback you have to our response. We will highly appreciate it if you let us know whether your previous concerns have been adequately addressed.

Thanks for your time and valuable comments! We are happy to see that you found our design intuitive and smart. We have provided more explanations and experiments to address your concern.

Q1: The proposed estimation of leading indicators cannot easily capture highly complex relations among variates and may not be accurate.

A1: Our procedure is to quickly filter out most noisy variates, coarsely obtain advance information from potential leading indicators, and then finely model relationships within a narrowed scope that benefits the forecasting. The motivation is that many MTS contain numerous variates (e.g., the Traffic dataset has 862 variates), while the complexity of estimating all cross-variate relationships at one time step is $\mathcal O(C^2)$ ($C$ is the number of variates). This requires making the estimation as efficient as possible, since we need to restart estimations at each prediction time.

It is not necessary for our lead estimation to be perfect, because our proposed Adaptive Frequency Mixer, a non-linear neural network, shoulders the responsibility for filtering out noisy information from the selected indicators.

Existing MTS forecasting methods pay little attention to the dynamics of leading steps, making their test performance susceptible to variations of channel dependence. Going a step further, one of our contributions is that we propose target-oriented shifts to keep variates aligned dynamically, allowing test-time adaptation.

---

Q2: The locally stationary lead-lag relationship should be static only in a short period, and may not work well in long-term forecasting.

A2: Nice comment! We provide additional data analyses on long-term forecasting benchmarks, demonstrating that the discovered lead-lag relationships generally last over the large horizon window with $H$ steps, where $H$ is set to $720$. Empirically, at each time $t$, we lookback the previous $336$ steps to select the $4$ most possible leading indicators for each target variate $j$. And for each leading indicator $i$, we obtain its leading step $\delta^{(j)} _{i,t}$. Then, we compute the future cross-correlation coefficient $R^{(j)} _{i,t+H}$ between the ground-truth horizon window $\mathcal X^{(j)} _{t+1:t+H}$ and the shifted sequence $\mathcal X^{(i)} _{t+1-\delta^{(j)} _{i,t}:t+H-\delta^{(j)} _{i,t}}$. With the horizon window sliding along the dataset, we repeat discovering relationships on the corresponding lookback window and computing $R^{(j)} _{i,t+H}$ for each $j$ and $i$ at different time $t$. We report the average of $R^{(j)} _{i,t+H}$ on each dataset as listed below.

As shown in the table, the large values indicate the lead-lag relationships generally hold over the large horizon window in long-term forecasting tasks.

There are two rationales behind the empirical results and the good performance of LIFT. (1) As most time series are not extremely non-stationary, there are relationships that last over a certain period in favor of channel dependence modeling. (2) Existing long-term forecasting methods lack prior knowledge of channel dependence and try to learn arbitrarily complex dependence, increasing difficulty in optimization and risking overfitting issues. By contrast, our method dynamically learns from the discovered lead-lag relationships, which are much simpler and remain effective during test time.

---

Q3: Fair comparison of the computation cost.

A3: Following LSTF-Linear [1], we compare the backbones and LIFT per sample by the number of parameters, the number of multiply-accumulate operations (MACs, $\approx\frac{1}{2}$FLOPs), the GPU memory consumption, and the inference time in the following table.

Our empirical study shows that LIFT additionally requires an average of 10.7% parameters, 1.7% MACs, 2.5% memory, and 14.2% inference time more than its backbone. It is noteworthy that our Lead Estimator is non-parametric and can precompute estimation only once on the training data, reducing practical training time.

---

[1] Zeng & Chen et. al. Are Transformers Effective for Time Series Forecasting? AAAI 2023.

Dear Reviewer cNYF,

Thank you once again for your dedication to reviewing our paper.

In our early response, we have included more explanations about our method, along with detailed data analyses and cost studies.

We would like to know whether our response has addressed your concerns. If you have any further questions, please do not hesitate to let us know, so that we can respond to them timely.

Thank you for your time and efforts in reviewing our paper. The followings are our responses to your specific questions.

Q1: The effect of instance normalization and denormalization.

A1: As a popular preprocessing method in time series forecasting, instance normalization and denormalization is used to avoid disturbance from value ranges. Particularly in our method, instance normalization is an essential prerequisite for computing cross-correlation coefficients as defined in Eq. (1).

$$R^{(j)} _{i,t}(\delta) = \frac{\operatorname{Cov}(\mathcal X^{(i)} _{t-L+1-\delta:t-\delta}, \mathcal X^{(j)} _{t-L+1:t})}{\sigma^{(i)}\sigma^{(j)}} = \frac{1}{L}\sum _{t^\prime=t-L+1}^{t}{\frac{\mathcal X^{(i)} _{t^\prime-\delta}-\mu^{(i)}}{\sigma^{(i)}}\cdot\frac{\mathcal X^{(j)} _{t^\prime}-\mu^{(j)}}{\sigma^{(j)}}}$$

Intuitively, it is infeasible to directly measure lead-lag relationships by the raw series which have varying value ranges (e.g., one ranging from 1 to 10 and another ranging from 100 to 1000). The advance information from a leading indicator is the shared general up/down trends rather than the specific values. After refining the normalized series of lagged variates by the advance information, it is necessary to restore them to the original value range via denormalization, yielding the final forecasts.

---

Q2: In Figure 3, “all layers in the grey background” refer to which?

A2: The grey rectangles refer to normalization, denormalization, Lead Estimator, Fourier transform, and the inverse Fourier transform. We would like to highlight that we avoid repeating Lead Estimation every epoch but precompute it only once on training data since our Lead Estimator is non-parametric.

---

Q3: Inappropriate expression: “one of the hottest” -> “one of the most popular”.

A3: Thanks for your advice. We have modified the expression in the re-uploaded paper.

Thanks for your time and interesting questions. We have provided more explanations below to better clarify our work and address your concern.

Q1: What if two or more time series jointly influence another time series? Misspecifying such actions interaction could be why CD methods underperform CI.

A1: To handle the complex cases, we design the Lead-aware Refiner to jointly model all the selected leading indicators for a target variate. Specifically, in Eq. (9), we utilize $\boldsymbol{R} _t^{(j)} \in \mathbb{R}^K$, the $K$-dimensional correlation vector of the $K$ leading indicators, to generate filters that reweight the indicators' influence. In this way, we learn from each leading indicator according to the overall situation of the $K$ selected indicators.

Existing CD backbones also jointly model the complex interactions among all variates. The main reason of their inferior performance is the overfitting issue, which is caused by their unawareness of the dynamic variation in leading indicators and leading steps. Particularly, LIFT alleviates overfitting by decomposing CD modeling into two steps:

1. An efficient Lead Estimator roughly ranks potential leading indicators by cross-correlations, and then the indicators are shifted by leading steps;
2. A learnable Lead-aware Refiner jointly models the complex interactions between the selected subset of leading indicators and the target variate.

---

Q2: When the influence structure is $X^{(1)} \rightarrow X^{(2)} \rightarrow \cdots \rightarrow X^{(C)}$, LIFT will not pick $X^{(C-1)}$ if it has the lowest cross-correlation.

A2: The chain structure $X^{(1)} \rightarrow X^{(2)} \rightarrow \cdots \rightarrow X^{(C)}$ indicates that the leading steps of the $C-1$ indicators w.r.t. $X^{(C)}$ are in a descending order, i.e., $\delta _1^{(1)}>\cdots>\delta _{C-2}^{(C)}>\delta _{C-1}^{(C)}$. On the one hand, a leading indicator with a larger leading step offers more advance information. On the other hand, a higher cross-correlation coefficient indicates a more possible lead-lag relationship. Thus, it is better to pick $\{X^{(i)}\} _{i=1}^{C-2}$ which are more helpful and more reliable than $X^{(C-1)}$.

---

Q3: Is it possible to discover causal relationships (e.g., Granger causality) among time series via modifications of LIFT?

A3: A possible way to compute a time-dependent causal graph via the modifications of LIST is the following. Initially, we build a graph based on the lead estimation results at time $t$, where each node (a variate $j$) is connected to $K$ nodes (its potential indicators $\mathcal I _t^{(j)}$), i.e., $\{X _{t-\delta _{i, t}^{(j)}}^{(i)} \rightarrow X _t^{(j)} \mid 1 \leq j \leq C, i \in \mathcal I _t^{(j)}\}$. Then, we prune the graph as follows.

1. Replacing the sum pooling in Eq. (11) by mean pooling, we rewrite our frequency mixing as

$$\widetilde{V}^{(j)} = g\left({V}^{(j)}\parallel \frac{1}{|\mathcal K^{(j)}|}\sum _{k\in \mathcal K^{(j)}} {\widetilde U}^{(j)} _k \parallel \frac{1}{|\mathcal K^{(j)}|}\sum _{k\in \mathcal K^{(j)}} {\widetilde \Delta}^{(j)} _k \right),$$

where $\mathcal K^{(j)}=\{1,2,\cdots,K\}$ denotes the indices of $\mathcal I _t^{(j)}$.

2. We train LIFT on the training data and calculate the forecast loss on the validation data.

3. For each target node $j$ in $[1, C]$, we repeat removing the most useless indicator by the following iteration algorithm until the loss of forecasting $j$ cannot be decreased.

   • For each $k$ in $\mathcal K^{(j)}$, we specially dismiss the $k$-th indicator, re-conduct frequency mixing, and calculate a new forecast loss by using the remaining $|\mathcal K^{(j)}|-1$ indicators. A lower loss indicates that the $k$-th selected one is just a noise.

   • Then, we update $\mathcal K^{(j)}$ as $\mathcal K^{(j)}-\{k^{\prime}\}$ if dismissing the $k^{\prime}$-th indicator leads to the lowest forecast loss.

---

Q4: Why there is subscript $U$ and $\Delta$ in equation (9)?

A4: To simplify notations in the following discussion, we leave out the superscript $(j)$ and the subscript $k$. Let $U$ denote the information of one leading indicator, $V$ denote the prediction of the target variate, and $\Delta=U-V$. Eq. (9) generates different filters $r _U$ and $r _{\Delta}$ for $U$ and $\Delta$, respectively.

The reason is that we consider two kinds of relationships:

• $V$ is directly influenced by $U$, and the ground-truth $V$ contains a degree of $U$, e.g., $V _{true} \approx V + r _U\odot U$;

• $V$ is similar to $U$ when they are both influenced by a latent factor, and the ground-truth $V$ is the interpolation between $V$ and $U$, e.g., $V _{true} \approx (1- r _{\Delta})\odot V + r _{\Delta}\odot U=V+ r _{\Delta}\odot\Delta$.

Note that both situations may exist in practice. We thus use different layers to generate $r _U$ and $r _{\Delta}$. Our ablation study in Figure 5b demonstrates the effectiveness of modeling both kinds of influence.

Dear Reviewer MEf1,

Thank you again for your dedication to reviewing our paper! In our early response, we have included more elaboration on our method. We will highly appreciate it if you let us know whether your previous concerns have been adequately addressed.