Ada-MSHyper: Adaptive Multi-Scale Hypergraph Transformer for Time Series Forecasting

Comment:

Many thanks to Reviewer VVbq for providing the insightful reviews and comments.

Q1: In long-range, short-range, and ultra-long-range time series forecasting, does Ada-MSHyper perform better on one type compared to the others? What might be the reasons for this?

Thanks for your valuable feedback and scientific rigor. Compared to long-range and ultra-long-range time series forecasting, short-range time series forecasting has better performance (reducing prediction errors by an average of 10.38%). The reason may be that we use PEMS datasets in short-range time series forecasting, which are influenced by human activities and have obvious multi-scale pattern information. This also explains why Ada-MSHyper also performs better on Electricity and Traffic datasets for long-range time series forecasting (achieving the best performance on all forecasting horizons).

As observed by the reviewer, the cause of the above phenomenon is the design of the node constraint, which is used to cluster nodes with similar semantic information. To investigate the impact of node constraint, we have newly added ablation studies on Electricity dataset by carefully designing the following variant:

• -w/o NC: It removes the node constraint.

The experimental results are shown in Table 1, which has also been included in the $\underline{\text{revised paper}}$.

Table 1.

From Table 1 and the experimental results of -w/o NC in $\underline{\text{Section 5.3 of the original paper}}$, we have the following observation: (1) The performance drop of -w/o NC is smaller compared to other variants (e.g., -w/o NHC) on ETTh1 dataset, possibly because there are weaker temporal patterns on ETTh1 dataset, and on that dataset Ada-MSHyper may focus more on macroscopic variation interactions rather than detailed group-wise interactions between nodes with similar semantic information. (2) Compared to its performance on ETTh1 dataset, the performance of -w/o NC shows a significant drop on Electricity dataset. The reason may be that Electricity dataset has obvious multi-scale pattern information, making the NC mechanism used to cluster nodes with similar semantic information appear to be more important. (3) Ada-MSHyper still performs better than -w/o NC in almost all cases, showing the effectiveness of node constraint.

Q2: How do the authors compare their method with frequency domain analysis techniques and can Ada-MSHyper easily adapt to the 2D spectrogram data in time-frequency domain?

To solve the problem of temporal variations entanglement, some frequency domain analysis methods (e.g., TimesNet [1], FiLM [2], FEDformer [3], and Autoformer [4]) adopt simplistic series decomposition with frequency domain analysis techniques to differentiate temporal variations at different scales. We have compared Ada-MSHyper with these methods and added analysis in $\underline{\text{Section 5.2 of the paper}}$.

In addition, Ada-MSHyper may not be directly applicable to 2D data. To investigate the performance of Ada-MSHyper in the frequency domain, we flatten the 2D spectrogram data into 1D features and conduct ablation studies by carefully designing the following variant:

• w/ STFT: It converts the multi-scale subsequences into 2D spectrogram data using Short-Time Fourier Transform (STFT) and then flattens it into 1D feature representations before send to the AHL model.

The experimental results on ETTh1 dataset are shown in Table 2, which has also been included in the $\underline{\text{revised paper}}$. From Table 2 we can observe that:

-w/ STFT performs worse than Ada-MSHyper. The reason may be that flattening 2D spectrogram data into 1D features will mix frequency domain features with time domain features, thereby impacting the forecasting performance of the model. However, the valuable feedback from the reviewer inspired us to consider that time series features need no to be limited to the form of "scalars" or "vectors". The "tensors" (e.g., 2D spectrogram data) may offer a better representation for time series forecasting. Modifying the model to support 2D spectrogram data instead of flattening it into 1D features may have better performance. Considering the scope of our paper, we would like to leave the exploration in the future work.

Table 2.

[1] Wu H, Hu T, Liu Y, et al. TimesNet: Temporal 2d-variation modeling for general time series analysis. ICLR, 2023.

[2] Zhou T, Ma Z, Wen Q, et al. FiLM: Frequency improved legendre memory model for long-term time series forecasting. NeurIPS, 2022.

[3] Zhou T, Ma Z, Wen Q, et al. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. ICLR, 2022.

[4] Wu H, Xu J, Wang J, et al. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. NeurIPS, 2021.

Comment:

Many thanks to Reviewer HJMp for providing the insightful reviews and comments.

Q1: The pipeline can be improved and some expressions in the paper can be more formal.

Thanks for your valuable suggestions and scientific rigor. We have improved the color of the pipeline, see Figure 1 in $\underline{\text{global response}}$. In addition, we have performed thorough proofreading and used more formal expressions.

Q2: Are your Euclidean distance normalized? Do some ablation studies to verify whether using $\alpha D$ as hyperedge loss will introduce redundant information.

We do not normalize the Euclidean distance $D$. The cosine similarity $\alpha$ and normalized Euclidean distance are indeed similar as both of them compare the direction of two vectors in feature space, reducing the impact of feature scale differences. However, relying solely on cosine similarity neglects differences in hyperedge representations regarding relative scale and distance (i.e., the magnitude of two vectors). For instance, one hyperedge connects group-wise nodes with larger values indicating a "peak variation", while another hyperedge connects group-wise nodes with smaller values indicating a "trough variation". After normalization, the differences between hyperedge representations would become less noticeable. To address this, we add Euclidean distance and use $\alpha D$ as hyperedge loss.

To investigate the effectiveness of the hyperedge loss, we conduct ablation studies by carefully designing the following three variants:

• -w/o $\alpha$: It removes cosine similarity and uses Euclidean distance as the hyperedge loss.

• -w/o $D$: It removes Euclidean distance and uses cosine similarity as the hyperedge loss.

• -Wass: It uses Wasserstein distance as the hyperedge loss.

The experimental results on ETTh1 dataset are shown in Table 1, which has also been included in the $\underline{\text{revised paper}}$. From Table 1 we can observe that: (1) -w/o $\alpha$ and -w/o $D$ perform worse than Ada-MShyper, showing the effectiveness of Euclidean distance and cosine similarity in hyperedge loss, respectively. (2) Ada-MSHyper performs better than -Wass, which demonstrates the effectiveness of the hyperedge loss in the adaptive hypergraph learning module.

Table 1.

Q3: Explain the reasons for the different designs of the intra-scale and inter-scale modules. Can they just use the same method for updating features?

The intra-scale interaction module is used to capture detailed group-wise interactions between nodes with similar semantic information, while the inter-scale interaction module focuses on capturing the interactions of macroscopic variations through hyperedges. If both modules use hypergraph convolution attention for updating features, there are two limitations for the inter-scale interaction modules: (1) There is no hypergraph structure regarding the relationships between hyperedges. (2) Hyperedges already represent group-wise interactions by connecting multiple nodes. Modeling group-wise hyperedges interactions through the hypergraph structure may introduce redundant information and cause the overfitting problem.

If we want to use the same method for updating features, one direct way is to use attention for both modules. However, this would cause the intra-scale interaction module lack the ability to capture group-wise interaction between nodes with similar semantic information and pair-wise attention would result in $\mathcal{O}(N^2)$ computation cost.

To investigate the feasibility of using the same method for updating features, we conduct ablation studies by carefully designing the following variant:

• -r/ att: It replaces the hypergraph convolution attention with the attention mechanism used in the inter-scale interaction module to update node features.

The experimental results on ETTh1 dataset are shown in Table 2, which have also been included in the $\underline{\text{revised paper}}$. From Table 2 we can observe that: Ada-MSHyper performs better than -r/ att, which demonstrates the effectiveness of the hyperedge convolution attention used in the intra-scale interaction module.

Table 2.

Q4: Whether there are some other distances for modeling the pattern interactions we can try? Like Wasserstein distance.

Other distances can also be introduced as constraint for modeling pattern interactions. As per your suggestion, we normalize the hyperedge features and use Wasserstein distance as the hyperedge loss. See response to Q2 for detailed analysis.

Comment:

Many thanks to Reviewer Y3pA for providing the insightful reviews and comments.

Q1: About some different long-range prediction results from those in the DLinear and iTransformer paper.

Thanks for your careful check and scientific rigor. As for the long-range time series forecasting, we have two kinds of settings, i.e., multivariate settings and univariate settings. Due to some methods (e.g., iTransformer and Crossformer) do not have results under univariate settings, we use their official codes and fine-tune their key hyperparameters. We have added an explanation about the Crossformer results in the $\underline{\text{revised paper}}$. As for the multivariate settings, some newly added baselines are rerun by us and other results are from iTransformer. We have carefully rechecked the experimental results and addressed the aforementioned issues in the $\underline{\text{revised paper}}$. See Table 1 in $\underline{\text{global response}}$.

Q2: Is each time point with multiple features in the time series mapped to a node in the hypergraph?

Yes, to be precise, Ada-MSHyper maps the input sequence into subsequences at different scales. The features of these subsequences are then treated as nodes in the hypergraph.

Q3: It would be beneficial to explain the model's efficiency from theoretical complexity or provide results on longer input sequence and additional datasets.

To provide a more comprehensive evaluation of the model's efficiency, we have included the theoretical complexity analysis and added additional computation cost results for longer input sequence and additional datasets. The results are shown as follows, which has also been included in the $\underline{\text{revised paper}}$.

Theoretical complexity analysis: For the MFE module, the time complexity is $\mathcal{O}(Nl)$, where $N$ is the number of nodes at the finest scale and $N$ is equal to the input length $T$. $l$ is the aggregation window size at the finest scale. For the AHL module, the time complexity is $\mathcal{O}(MN+M^2)$, where $M$ is the number of hypergraphs at the finest scale. For the intra-scale interaction module, since $\mathbf{D}_v$ and $\mathbf{D}_e$ are diagonal matrices, the time complexity is $\mathcal{O}(MN)$. For the inter-scale interaction module, the time complexity is $M^2$. In practical operation, $M$ and $l$ is the hyperparameter and is much smaller than $N$. As a result, the total time complexity is bounded by $\mathcal{O}(N)$.

Computation cost results: We have newly added computation cost on Traffic and Weather datasets with the 720-96 input-output length. The experimental results are shown in Table 1 and Table 2, from which we can observe that:

• Although Ada-MSHyper has a large number of parameters, it achieves lower training time and lower GPU occupation due to the matrix sparsity strategy in the model and the optimization of hypergraph computation provided by torch_geometry in PyTorch.

• Ada-MSHyper can maintain better performance even with longer input length. Considering the forecasting performance and computation cost, Ada-MSHyper demonstrates its superiority over existing methods.

Table 1.

Table 2.

Q4: In the ablation experiment section, Include longer prediction lengths to further evaluate the model's performance.

We have added additional ablation studies on ETTh1 dataset to verify the performance of Ada-MSHyper on longer prediction lengths. The results are shown in Table 3, which has also been included in the $\underline{\text{revised paper}}$. From Table 3 we have the additional observation:

• For longer prediction length, -w/o NC has smaller performance degradation than other variations. The reason may be that when the prediction length increases, the model tends to focus more on macroscopic variation interactions and diminishes its emphasis on fine-grained node constraint.
• Ada-MSHyper performs better than -w/o NC and -w/o HC even with longer prediction length, showing the effectiveness of node constraint and hyperedge constraint, respectively.

Table 3.

Comment:

Many thanks to Reviewer HQ9E for providing the insightful reviews and comments.

Q1: Graph-Transformer methods should be included in Related Works and used for comparisons.

Thanks for your valuable suggestions and scientific rigor. We have added two latest Graph-Transformer methods, i.e., MSGNet (AAAI 2024) and CrossGNN (NeurIPS 2023) for comparison. The descriptions of these methods and their long-range time series forecasting results are shown as follows, which have also been included in the $\underline{\text{revised paper}}$.

• MSGNet: MSGNet leverages frequency domain analysis to extract periodic patterns and combines an attention mechanism with adaptive graph convolution to capture multi-scale pattern interactions.
• CrossGNN: CrossGNN uses an adaptive multi-scale identifier to construct multi-scale representations and utilize a cross-scale GNN to capture multi-scale pattern interactions.

The long-range time series forecasting results under multivariate settings are shown in Table 1 of the $\underline{\text{global response}}$ and the following tendencies can be discerned:

• MSGNet and CorssGNN are the state-of-the-art graph learning methods that use graph learning modules to capture multi-scale pattern interactions. However, they can only capture pair-wise interactions instead of group-wise interactions and get worse performance than Ada-MSHyper in most cases.

Q2: Will the sparsity strategy introduce any loss of useful information?

The sparsity strategy is employed to reduce subsequent computation costs and noise interference. However, the effectiveness of the sparsity strategy is influenced by the hyperparameter $\eta$. When $\eta$ is set to a smaller value, some useful information may be filtered out. We have performed parameter studies to measure the impact of $\eta$. The results are shown in Table 2, which has also been included in the $\underline{\text{revised paper}}$. From Table 2 we have the additional observation:

• The best performance can be obtained when $\eta$=3. The reason may be that a small $\eta$ may filter out useful information and a large $\eta$ would introduce noise interference.

Table 2.

Q3: Study and quantify the computational cost reduction by the above approach.

To investigate the effectiveness of the sparsity strategy, we conduct ablation studies by carefully designing the following variant:

• -w/o SO: It removes the sparsity strategy in the AHL module.

We compare the computation cost on Electricity dataset under 96-96 input-output settings, the experimental results are shown in Table 3, which has also been included in the $\underline{\text{revised paper.}}$

Table 3.

From Table 3 we can observe that Ada-MSHyper can achieve better performance with faster speed and lower GPU occupation compared to -w/o SO, which demonstrates the effectiveness of the sparsity strategy.

Q4: Is the linear layer for forecasting a linear regression?

Yes, the linear layer used for forecasting can be considered as a form of linear regression as it maps the updated multi-scale features to the final predictions through a linear relationship.

Q5: Datasets should include financial data, e.g., Stock Market.

We have added Nasdaq 100 Stock dataset for comparison. The detailed descriptions of the public dataset are shown as follows:

• Nasdaq: This dataset includes the stock prices of 82 major corporations, which are sampled 390 times every day from July 2016 to December 2016.

The full long-range time series forecasting results on Nasdaq dataset will be included in the $\underline{\text{revised paper}}$. Due to time and space limitations, we list the comparison results between Ada-MSHyper and three latest baselines. The experimental results are shown as follows:

Table 4.

From Table 4 we can observe that Ada-MSHyper achieves the best performance in almost all cases. The experimental results demonstrate the effectiveness of Ada-MSHyper on stock market dataset.

Q6: Analyze the datasets to check and quantify the presence of "multi-scale pattern interactions".

We have added weight visualization results, see Figure 2 in $\underline{\text{global response}}$, the detailed analysis has been included in the $\underline{\text{revised paper}}$.

Q7: The reason for comparable accuracies of WITRAN on ETTm2 dataset.

WITRAN employs a Recurrent Acceleration Network within its framework. For ETTm2 dataset (high forecastability), it can make effective predictions with fewer parameters and a shallower network structure. However, recurrent structure may face underfitting risks when handling more challenging datasets (low forecastability), e.g., ETTh1 and ETTh2. This is why WITRAN achieves comparable accuracies on ETTm2 but performs worse on other datasets (e.g., ETTh1 and ETTh2).