Structured Matrix Basis for Multivariate Time Series Forecasting with Interpretable Dynamics

Q1 The proposed model may be complex to implement due to the use of structured matrix bases and singular value decomposition, potentially leading to a steep initial learning curve in practical applications. Does the appendix indicate similar results for datasets other than Electricity?

We would like to clarify that our method actually is very easy to implement. Note that the singular value decomposition only serves to offer a parameterization form, we do not need to implement it in practice. Instead, we only need to implement the parameterization of two orthogonal matrices $\mathbf{U}$ and $\mathbf{V}$ via the Clay map. As we mentioned in the paper, the Clay map can be implemented in a numerically stable way in PyTorch with one line code by the torch.linalg.solve function. The learning curves of our proposed method are also very stable across all six datasets, and we will add more of them to the revised manuscript.

Q2 While the model has been validated on various datasets, it remains to be seen how well it performs on very specialized domain data or data with unclear characteristics (i.e., weak temporal dependencies or spatial correlations). How does the model address these scenarios?

Thanks for your comments. To our knowledge, the temporal dependency assumption is necessary for all time series forecasting methods. If a dataset has only weak temporal dependencies, it will impede the performance of all existing time series forecasting methods. For datasets with weak spatial correlation, it means that different channels (series) are independent and our proposed method will consequently learn an identity matrix for the spatial structure. This will not cause any negative impact on the forecasting performance. We will add the corresponding experiments to verify this by synthesizing the datasets with independent channels in the revised manuscript.

Q3 Including more analysis and comparison with graph-based spatio-temporal models would enhance the paper.

Thanks for your suggestion. In Table 1 of our manuscript, MTGNN, MegaCRN, iTransformer, Crossformer, Card, ESG, FourierGNN, and TPGNN are all GNN-based methods. First, we note that the graph-based spatio-temporal methods often produce better results than the methods without considering the spatial structures. Second, the dynamic graph-based methods such as iTransformer, Card, and ESG often give rise to better performance than the static graph-based methods such as MTGNN and MegaCRN. However, the existing graph-based spatio-temporal models all rely on a two-stage spatial structure learning process that is prone to yielding high variance and impeding the final performance, which is verified by the experiments. We will add more comparisons and analyses in the next version of our paper.

Q1 The obtained performance gain (Table 1) was overall marginal (up to 2-3 decimal points). Adding statistics to the results over different random seeds of experiments is important. Clarification on the observed margin of performance improvements and statistics/error bars to the results will be appreciated.

Thank you for your comments and suggestions. We would like to point out that the performance gains of 2-3 decimal points are often considered to be notable improvements in time series forecasting literature, e.g., the prior works iTransformer [1] and Card [2] also achieve similar gains. To further demonstrate it more clearly, we provide the improvements (percentage) over the best baseline methods in the following table.

In summary, our proposed method achieves an average performance improvement of 2.35% over the best baselines. This enhancement is significant in the field of time series forecasting and verifies the effectiveness of our proposed model. Regarding the effect of random initialization, as mentioned in line 263 of our manuscript, our reported results are actually averaged over three runs with different random seeds. We will further add the statistics (standard deviation) into the tables as well as error bars in our revised manuscript. In addition, the error bars are also provided in Figure 1 in the supplied PDF file.

[1] iTransformer: Inverted Transformers are Effective for Time Series Forecasting, ICLR 2024.

[2] Card: Channel Aligned Robust Blend Transformer for Time Series Forecasting, ICLR 2024.

Q2 In switching dynamic systems, it is also common to model the transition matrix over time as a linear combination of several global matrices, with time-varying mixing coefficients. Better clarification and empirical results on relation with switching dynamics systems will be appreciated.

Thank you for your suggestion. In switching dynamic systems, the transition matrix is used to model the temporal dependencies. The recent representative works include Hippo [1], LSSL [2], S4 [3], and Mamba [4]. In contrast, the structured matrix basis is used to capture the spatial structures in our proposed method. We have conducted the additional experiments to compare with S4 in Table 2 in the supplied PDF file and will also add the experimental results as well as the corresponding discussion in our revised manuscript.

[1] Hippo: Recurrent Memory with Optimal Polynomial Projections, NeurIPS 2020.

[2] Combining Recurrent, Convolutional, and Continuous-time Models with Linear State Space Layers, NeurIPS 2021.

[3] Efficiently Modeling Long Sequences with Structured State Spaces, ICLR 2022.

[4] Mamba: Linear-time Sequence Modeling with Selective State Spaces, arXiv 2023.

Q3 Based on the results in Fig 3, it appears that increasing M from 1 to 2 in general introduced small difference in performance except in the Electricity dataset. Stronger clarification on performance improvement and adding error bars to the results in Fig 3 will be important for verifying the contribution of the paper.

Thank you for your suggestion. To further demonstrate the effectiveness of our proposed dynamic spatial structure, we present the improvements (percentage) in the table below as $M$ increases from 1 to 2. The results show that our method achieves an average performance improvement of 5.66% across six different datasets. This significant improvement underscores the effectiveness of our dynamic spatial structure design. We also include the error bars in Figure 2 in the supplied PDF file to show these more clearly.

Q4 What is the computational cost difference in the ablation with or without common U-V basis (i.e., vs. w/ Um,Vm)?

Let $M$ be the dimension of the matrix basis, $N$ the number of nodes, $K$ the rank, and $D$ the feature dimension. As shown in Equation 10 of our manuscript, if each matrix basis has specific coordinates $\mathbf{U}_m$ and $\mathbf{V}_m$, then $\mathbf{V}_m^T \mathbf{x}$ and $\mathbf{U}_m \mathbf{y}^\prime$ will be calculated $M$ times. Consequently, the computational costs are $\mathcal{O}(MNKD)$ and $\mathcal{O}(NKD)$ for the models without and with a common $\mathbf{U}$-$\mathbf{V}$ basis, respectively. The computational cost difference will be included in our revised manuscript.

Q5 The authors did not include sufficient discussion about the limitation or potential negative societal impact of the presented work.

Thanks for your suggestion. We will add more discussion on the limitation including, the model performance on long-term forecasting tasks and hyperparameter selection.

Q1 The forecasting horizon in the experiments is quite different from the setting in some baselines, such as PatchTST and FEDformer. While long-term forecasting is not major claim of the paper, I wonder if the method scales well with prediction length.

Thanks for your comments. Indeed, long-term forecasting is not the main focus of our proposed method, and the present version of Sumba is not aimed at addressing long-term predictions. But we do conduct experiments with a forecasting horizon of 96 and as Table 1 in the supplied PDF file shows, Sumba still gives rise to favorable performance in comparison to PatchTST, FEDformer, and iTransformer which are designed specifically for long-term forecasting. We will leave it for our future work to further enhance its long-term forecasting capability and add this discussion to the limitation section.

Q2 Why is TCN selected for temporal encoding?

Since the main focus of the paper is to effectively capture the spatial structures under the GCN framework, TCN is chosen for temporal encoding due to its simplicity and can be easily integrated with the GCN operation. In addition, TCN architecture (with appropriate modifications) [1] also proved very effective in time series analysis recently. We are also exploring the possibility of integrating our proposed spatial structure modeling with other temporal encoders such as Transformers, Structured State Space models, etc.

[1] ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis, ICLR 2024.

Q3 Is there any issue that the learned bases degenerate to be trivial or converge to be close?

We do not empirically run into this issue in our experiments (conducted on six public datasets from various domains). We hypothesize that the degenerated issue may arise when the spatial structures are static and can be represented by a single graph structure. In such a case, the model would only attend to one of the matrices in the basis (that is, all weights are concentrated in one entry while the others are zeros in the coefficient $\alpha$). This will not have a negative impact on the model's performance.

Q4 Limitations are discussed, and I agree that the selection of $M$, i.e. the number of bases, is highly empirical and ad-hoc.

Yes, it will be our future work to explore the adaptive selection of $M$ as well as enhancing the model’s long-term forecasting capability.