PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecasting

W3: How PSformer differs from the separated modeling methods like Crossformer and iTransformer; Visualizing the advantages of PSformer in modeling time and channel.

• SegAtt vs. Separate Modeling Methods Unlike methods such as Crossformer[2] and iTransformer[3], which treat temporal and channel information as independent entities for separate modeling, PSformer takes a different approach. Separate modeling not only increases the total number of network parameters but can also degrade predictive performance when aggregating information from both dimensions. In contrast, PSformer introduces Spatial-Temporal Segment Attention (SegAtt), which unifies the treatment of temporal and channel information within spatio-temporal segments. This simplifies the model structure and leads to more structured attention expressions in shallow layer, mitigating the risk of overfitting caused by unnecessary channel mixing. Additionally, our experiments demonstrate that PSformer also performs competitively on single-channel sequences and scenarios with weak inter-channel correlations (as discussed in our response to Reviewer 5(MoxF)'s W2).

• Visualizing PSformer’s Advantages We provided further visualizations of attention weights in the README of the anonymous repository to demonstrate the model's ability to capture local spatio-temporal information. To facilitate understanding, we added auxiliary lines to the attention maps to distinguish cross-channel and cross-temporal components. Additionally, we separated single-channel attention submaps and cross-channel attention submaps, analyzing their correspondence with specific time points in the respective time series segments.

In the Visualization section of the README, the Figure 1.1 and Figure 1.2 show that the local spatio-temporal relationships in the attention weight maps are structured and organized. It is evident that regions of concentrated attention weights and gradual changes in attention weights across spatio-temporal dimensions can be observed. This suggests that the model captures temporal consistency between variables, local stationary features. Further verification was conducted by comparing the attention submaps with the corresponding time series variates. From Figure 2.1 to Figure 4.2, we visualized the time series positions corresponding to the high attention weights assigned to univariate time series. These visualizations reveal that the model places higher attention on critical time steps along temporal dimension fo univariate. From Figure 5.1 to Figure 7.2, we visualized the cross-channel attention weights between two time-series variables and their corresponding time series. These analyses demonstrate that PSformer assigns higher attention to important cross-channel patterns, such as large inverse changes between channels, locally continuous temporal patterns, and shared local temporal troughs.

W4: Why do the PatchTST results in Table 1 differ significantly from the original?

As discussed in Appendix A.1, we referred to the PatchTST[3] results from the iTransformer paper, as our work draws inspiration from SAMformer, which adopted same PatchTST results. In addition, the original PatchTST experiments inherited a known bug where drop_last=True was set in the test set Dataloader. This bug may notably impact the predictive performance on many datasets. To better compare with PatchTST, we correcting the bug in the official PatchTST code to set drop_last=False (as noDL) and re-ran its experiments. The results are presented in our response to W5.

W5: Add PDF and Time-LLM for comparison.

Thank you for suggesting additional baseline methods like PDF[5] and Time-LLM[6] for comparison. We appreciate the recommendation and have the following response:

• Comparison with PDF and PatchTST. While PDF demonstrates strong predictive performance, as shown in its paper, it also relies on a more complex hyperparameter search process. Additionally, PDF ad PatchTST set drop_last=True in the test set dataloader, which results in the model inadvertently using incomplete batches during evaluation. To ensure a fair comparison, we modified PDF and PatchTST by correcting this DL issue (setting drop_last=False) and set the input length to 512 to align with our experimental setup. Under these adjustments, we have included the updated results in the table below, which provide further insights into PSformer performance under fair conditions.

  We conducted tests on the following five datasets. From the experimental results, both PSformer and PDF significantly outperform PatchTST in predictive performance, and PSformer achieving better predictions than PDF. However, the average prediction loss reduction relative to PDF is not substantial. Therefore, we consider PSformer and PDF to exhibit equally excellent predictive performance under the same settings.

To be continued...

[1] Vaswani, A. (2017). Attention is all you need. Advances in Neural Information Processing Systems.

[2] Zhang, Y., & Yan, J. (2023). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The eleventh international conference on learning representations.

[3] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[4] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations.

[5] Dai, T., Wu, B., Liu, P., Li, N., Bao, J., Jiang, Y., & Xia, S. T. (2024). Periodicity decoupling framework for long-term series forecasting. In The Twelfth International Conference on Learning Representations.

[6] Jin, M., Wang, S., Ma, L., Chu, Z., Zhang, J. Y., Shi, X., ... & Wen, Q. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. In The Twelfth International Conference on Learning Representations.

[7] Huang, Q., Shen, L., Zhang, R., Ding, S., Wang, B., Zhou, Z., & Wang, Y. CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement. In Thirty-seventh Conference on Neural Information Processing Systems.

• Comparison with Time-LLM. For Time-LLM, We listed the predictive performance from the original paper of the model in the table below. However, we check its offical repository and find Time-LLM also faces DL issue in the test set dataloader, which may affects its reported results. Besides, due to the computational demands of large-scale models, we decided not to execute its code directly in our experiments. When selecting the baseline large models, we considered both MOMENT and TimeLLM. We ultimately chose MOMENT for two main reasons: the MOMENT paper includes a direct comparison with TimeLLM, and it is relatively lightweight (1 billion parameters). In summary, despite the significant difference in parameter scale and the DL issue present in TimeLLM, PSformer still achieves equal or better average predictive performance than TimeLLM on 3 out of 5 datasets.

• Comparison with Crossformer. For Crossformer, we used experimental results based on the CrossGNN[7] paper, as there is an inconsistency in prediction lengths between the original Crossformer paper and current mainstream work. Therefore, we aligned the experimental setup with the CrossGNN results to ensure consistency in the comparison. From the results, PSformer performs better than Crossformer.

ETTh1

ETTh2

ETTm1

ETTm2

Weather

W6: There are writing errors.

Thank you for pointing out these writing errors. We will correct these errors in the revised version of the manuscript.

To be continued...

We appreciate the thoughtful and valuable comments from the reviewer. Below, we offer detailed responses to each of your concerns.

W1: The paper overall lacks innovation; Parameter Sharing is simple module parameter-sharing technique to reduce complexity; Spatial-Temporal Segment Attention simply merges the channel dimension and patch size dimension together before modeling it with a Transformer.

There seems to be a misunderstanding regarding our work. Parameter sharing in PSformer is not merely a simple module parameter-sharing technique. As shown in Figure 2 of our paper, it is utilized in three critical parts within the Encoder structure:

Within the Two-Stage SegAtt Mechanism

• The PS_block module is used to construct the Q, K, and V matrices required for the attention mechanism.

• These steps primarily focus on applying attention to critical parts of the time series and extracting key features for prediction.

During the Feature Fusion Phase

• The PS_block integrates and mixes spatial-temporal features across different segments, enhancing the representation of global features and improving overall expressiveness.

In our work, we introduced this technique to the time-series forecasting domain and utilized it as part of PSformer, achieving competitive predictive performance.

SegAtt vs. Transformer Comparison. It appears the reviewer may be equating the PSformer Encoder with the Transformer Encoder, and SegAtt with the self-attention mechanism. However, there are fundamental differences:

Attention to Local Spatial-Temporal Segments

SegAtt is specifically designed to apply attention to local spatial-temporal segments, enhancing the model’s ability to capture localized spatial-temporal relationships.

Key Differences from Standard Self-Attention

• In standard Transformer self-attention, the Q, K, and V matrices are obtained by multiplying the input x with weights from separate Linear layers.

• In SegAtt, the Q, K, and V matrices are generated using a three-layer MLP structure with residual connections (PS_block).

Furthermore, the PS_block encapsulates all parameters within the Encoder. This allows PSformer to learn both local and global feature representations more effectively while reducing the total parameter count by sevenfold (including Q, K, and V generation from the two SegAtt modules and the final output fusion). To better understand these distinctions, we encourage the reviewer to compare our code with standard Transformer implementations.

W2: What is the purpose of including the PS Block in the Segment Attention? Why not just add the PS Block after the output of the Attention, which would eliminate the need for parameter sharing? Why can two SegAtt be viewed as one FFN layer?

A2.1: As mentioned in our response to W1, the PS Block plays a crucial role within Segment Attention. Additionally, introducing the PS Block into SegAtt enhances the capability of the attention mechanism by leveraging the non-linear transformations of the PS Block. Compared to linear mappings, this significantly improves the construction of the Q, K, and V matrices and enables the model to better capture complex relationships in the input data.

A2.2: Even if the PS Block is not used within the attention mechanism, the parameter count of the Encoder would still increase. This is because neural network is still required to independently construct the Q, K, and V matrices. Moreover, removing the PS Block from SegAtt would negate the additional benefits of parameter sharing, such as reduced model size, improved generalization and learn global representations, which are key advantages of our design.

A2.3: To help readers better understand the structure of the PSformer Encoder, we mentioned that two SegAtt mechanisms can be analogous to an FFN layer. Specifically:

Without residual connections, the two-stage Segment Attention process can be described as:

$\text{two SegAtt}(X) = \text{Attention}(\text{ReLU}(\text{Attention}(X))).$

In comparison, a standard FFN is often represented as:

$\text{FFN}(X) = \text{Linear}(\text{ReLU}(\text{Linear}(X))).$

Because in Section 3.3 of the Transformer[1] paper, they define $\text{FFN}(X) = max(0,xW_1+b_1)W_2+b_2$. So the two-stage Segment Attention process can be seen as replacing the Linear part with Attention for better understand. However, this might lead to the misunderstanding that SegAtt is no different from the Attention mechanism in Transformer as discussed in W1. We will further clarify this point in the next version of the paper. For a more detailed explanation, please refer to Section 3.2 of our paper.

To be continued...

Thank you for your inspiring questions, which are beneficial for clarifying some unclear presentations and improving our work. Additionally, we have uploaded the rebuttal version of the paper, which integrates the content added during the rebuttal period to better showcase the paper's innovations, improving clarity, and enhancing the overall quality. We hope our response has adequately addressed your concerns. We take this as a great opportunity to improve our work and shall be grateful for any additional feedback you could give to us.

Thanks to the author for their rebuttal. Although the author has addressed these questions, some issues remain unresolved. The main points are as follows:

• W1: I still believe this paper's novelty is insufficient. The PS block is used to construct the Q, K, and V matrices for segmented attention and to reduce the number of model parameters. Additionally, segmented attention simply merges the channel dimension and patch size dimension to model spatial-temporal features.

• W5: The author conducted experiments with PatchTST, PDF, and TimeLLM as baselines. From the results, the prediction performance of PSformer shows only a slight improvement compared to PDF and PatchTST.

Thank you for your timely feedback. We have summarized the concerns you remain as follows:

Regarding Innovation. We are the first to introduce the idea of parameter sharing into the time series domain and the first to adopt segment attention to simultaneously capture local spatio-temporal information. We developed a transformer-based model, PSformer, and validated its effectiveness in time series forecasting. We believe that simple yet effective innovation is valuable and elegant as long as it can be applied in broader scenarios. For example, PatchTST was simply about introducing the patch concept to the time series domain, even though patch methods had already been popular in the CV domain for years. Nevertheless, PatchTST remains an important work in the time series domain, with over a thousand citations. Another example would be the LoRA fine-tuning in LLM, which is also a simple but impactful application of existing low-rank approximation method. Therefore, we would like highlight the significance of the proposed parameter sharing design and segment attention in time series domain, as verified by extensive experiments.

Regarding Prediction Performance. We have validated the superiority of PSformer's prediction performance by comparing it with a sufficient variety of competitive benchmark models. Based on the current empirical results, PDF’s predictive performance is arguably among the best in mainstream works. However, if you delve into its code, you’ll find that its parameter settings vary significantly for each task. Even so, across five datasets, PSformer still achieves better prediction performance than PDF on four of them.

Regarding the "slight improvement" of PSformer compared to PDF and PatchTST. We have demonstrated PSformer's superior prediction performance compared to a wide range of competitive benchmark models. However, we also acknowledge that with the rapid progress in this field, it becomes harder to obtain significantly better results without more extra data or computation. It would be useful if an example from top conference or submission in ICLR that demonstrates a more clear improvement over PSformer could be suggested for comparison.

In summary. Based on the best of our knowledge, we believe the achieved improvement is non-trivial and we tried to clarify the concerns over innovation and model predictive performance. It would be grateful if more constructive feedback could be provided to better address the concerns.

Thank you for your comments. As discussed above, we have reached a consensus on most concerns. For the remaining two, W1 and W5, we provide further responses as follows.

Regarding W5. In addition to the previous explanation, we further provide pairwise comparisons of PSformer with PatchTST and PDF models, using MSE-D to quantify the reduction in loss achieved by PSformer compared to these models. Furthermore, we include comparisons on the Exchange dataset, which is frequently used in time series forecasting studies and exhibits non-stationary characteristics. The detailed comparison results are presented in the table below.

From the comparisons across tasks on these six datasets, we observe that the MSE-D difference is not only positive in most cases but also exceeds 0.005 in 15 tasks compared to PatchTST and in 12 tasks compared to PDF. It is important to note that our comparisons were conducted under unified noDL settings, which avoids inconsistencies in test set length caused by dropping the last batch of the test set. The DL issue can significantly reduce the MSE, especially for small-scale datasets with large batch size.

Regarding W1. we understand that different perspectives may lead to varying interpretations of innovations. While we have carefully worked to address this, we also remain open to constructive feedback for further clarification or improvement. We have strived to ensure our contributions are conveyed in a clear and concise manner for better understanding. We acknowledge that some aspects might still require further refinement. In the next version, we will incorporate your valuable suggestions to better present this work.

Thanks to the author for rebuttal. Your answers address most of my concerns. Considering the paper's novelty, writing quality, and performance of the model, I raise the score to 5.

We thank the reviewer for the efforts in reviewing our paper and the thoughtful comments. Below, we offer detailed responses to each of your concerns.

W2: SegAtt may underperform in cases of univariate time series or with little dependency between variables. We tested the performance of PSformer in single-sequence forecasting. Specifically, we saved the 8 variables from the Exchange dataset into 8 separate single-sequence files, each supplemented with an additional column filled with zeros to form single sequences, along with an unrelated variable. We compared PSformer with the baseline model PatchTST[1] (which is channel-independent and performs well on the Exchange dataset).

The experimental results are shown below. As can be observed, PSformer consistently outperforms across all univariate time series, demonstrating that the PSformer architecture is not only effective in capturing cross-channel information but also performs well in univariate time series or with little dependency between variables.

During the experiments, PatchTST encountered a NaN loss on the validation set for a prediction length of 720, so the corresponding loss value was not recorded. In PSformer forecasting, we followed its current best configuration by adjusting the RevIN lookback window to 16 (For more about this setting, please refer to our response to Reviewer 2(xcmx)'s W2).

Q1: Are there any specific settings where PSFormer performs particularly well? Do they work better on datasets with strong correlations between variables? Why the Electricity and Exchange datasets exhibited weaker performance compared to others?

Based on our experimental experience, there is no specific configuration that guarantees particularly well model performance. Specifically, the model's performance does vary with different datasets and hyperparameter settings. however, this influence is complex, and we have not identified an optimal configuration. The settings we provide represent a relatively general and robust set of hyperparameters across different datasets. Nevertheless, it is still possible to enhance the model's performance through hyperparameter tuning according to the characteristics of datasets.

PSformer indeed enhances prediction capability by capturing cross-channel correlations, as evidenced by the temporal relationships observed in the attention maps (further discussed in the README of our anonymous repository). However, we believe it is not possible to conclude, based on the current eight datasets, whether PSformer performs better on strongly correlated variables. This would require further experiments comparing datasets with strong and weak correlations.

The weaker performance on the Exchange dataset can be attributed to its non-stationary nature and random walk characteristics, which prevent RevIN[2] from obtaining stable mean and variance statistics. These statistics are sensitive to the choice of RevIN's lookback window. Further testing revealed that the model performs best when the lookback window for calculating RevIN's statistics is very small (length 16), achieving results superior to all selected baseline models. The specific experimental data are listed in the table below, and more related discussions can be found in our response to Reviewer 2(xcmx)'s W2. For the Electricity dataset, due to time constraints, we did not conduct detailed experiments. However, we believe that for non-stationary data, RevIN's normalization should be used with caution. Adjusting the lookback window length can help identify more stable statistical means and variances, thereby facilitating model training.

To be continued...

Q2:Comparing PSFormer’s performance with and without SAM, or with other optimization methods such as Adam or SGD.

Regarding SAM[3], as described in Appendix A.6 and B.1 of our paper, it modifies the typical gradient descent update to seek a flatter optimum, which helps improve generalization. Specifically, SAM achieves this by perturbing the weights in the direction of the steepest gradient ascent within a small neighborhood of the current weights, with the parameter \rho controlling the magnitude of the perturbation. Then, SAM calculates the gradient at this perturbed position and updates the weights accordingly. This process helps the model converge to solutions that are robust to small changes in the parameter space, making it less prone to overfitting.

SAM is nested on top of a base optimizer, such as Adam or SGD. In our experiments, we used Adam as the base optimizer (please check exp part source code of our anonymous repository）. We also compared different base optimizers (e.g., AdamW, SGD) and found their difference impact to be minimal. For the performance without SAM, please refer to Figure 3 in Appendix B.1 of the paper, where a \rho value of 0 indicates that no additional perturbation is applied during gradient updates. This causes SAM to degrade into the basic Adam optimizer. To minimize the potential influence of SAM, we compared PSformer with other models that also use the SAM optimizer (e.g., SAMformer[4], TSMixer[5]) and verified that PSformer achieved better performance. The specific experimental results are presented in Table 11.

Q3 and W3: Conducting more experiments on the effects of positional encoding on different time series types (e.g., seasonal vs. non-seasonal, stationary vs. non-stationary).

We added a set of comparative experiments using positional encoding to better illustrate its impact. Specifically, we tested the encoding performance under two different data transformation modes: pos emb (time series), where positional encoding is applied to the original time series before dimension transformation; and pos emb (segment), where positional encoding is applied to the transformed segments. The default case, No pos emb, refers to the absence of positional encoding.

To highlight the differences between seasonal vs. non-seasonal and stationary vs. non-stationary characteristics, we selected the ETTh1 and ETTh2 datasets (relatively seasonal and stable) as well as the Exchange dataset (relatively non-seasonal and non-stationary). The experimental results are shown below.

The degraded performance of pos emb (time series) might be due to the incompatibility of the positional encoding with dimension transformation, as the original temporal order is lost in the segment dimension, making it unsuitable for dot-product attention calculations. On the other hand, pos emb (segment) shows smaller changes compared to the No pos emb case, but the performance still deteriorates slightly. This suggests that the significance of positional encoding in the context of multivariate time series forecasting might need to be re-evaluated, as there are fundamental differences between NLP and time series data when applying attention mechanisms.

To be continued...

Q4 and W1: Could the authors detail their hyperparameter tuning process, or discuss recommended tuning approaches (e.g., grid search, random search, Bayesian optimization)?

We believe that such fine-tuning relies more on judgment and intuition. For example, when selecting the number of encoder layers, we consider factors such as the dataset size and the level of noise in the data. For smaller datasets or those with low signal-to-noise ratio conditions, it is preferable to use fewer encoder layers to avoid the risk of overfitting. For parameters like the number of segments, which are less intuitive to adjust, we initially followed the settings of PatchTST, dividing the time series into 32 segments. While we also tested other potential segmentations, we found little improvement, so we ultimately standardized the setting to 32 segments.

Additionally, We believe parameter tuning is essential in practical business scenarios. For instance, in quantitative investment, some hyperparameter configurations may significantly enhance overall model performance. Therefore, efficient hyperparameter search methods, such as Optuna[6] or Hyperopt[7], are highly recommended for use in real-world applications. However, defining a good search space is equally important, as it directly impacts the effectiveness and robustness of the parameter search process.

Q5: how parameter sharing in the PS Block affects the model’s temporal pattern capture? Compare learned representations across layers with and without parameter sharing, and illustrating its influence on time series modeling.

In PSformer, the PS Block is the essential component of the two-stage SegAtt mechanism. It generates Q, K, V matrices to assign higher attention weights to critical temporal features. Within the single-channel temporal attention submaps under parameter sharing, we observe the model's ability to effectively capture temporal patterns. Besides, in the fusion stage, the PS Block integrates the features extracted in the first two stages of SegAtt.

To better illustrate this phenomenon of parameter sharing and its impact on spatio-temporal pattern capturing, we have provided additional visualizations and discussions in the README of our anonymous repository. Specifically, we compared the attention maps with and without parameter sharing, decomposing them into single-channel attention and inter-channel attention submaps. Furthermore, we compared the two-stage attention matrices under the non-parameter-sharing setup to analyze cross-layer variations.

Difference between with Parameter Sharing and without Parameter Sharing:

• Numerical Range on Attention map: From the attention maps, we observe that attention matrices with parameter sharing generally have values within [-3, 3], whereas without parameter sharing, the range broadens significantly to [-30, 40]. This larger variation in attention weights without parameter sharing might accelerate convergence during training. On the other hand, parameter sharing tends to stabilize the learning process by limiting extreme variations, which can be beneficial for training stability, particularly on smaller datasets.

• Cross-Channel Relationships: With parameter sharing, the relationships between channels are simpler and more interpretable. Clear, gradual transitions in attention weights across grid cells can be observed, indicating a more structured representation of spatio-temporal dependencies. In contrast, without parameter sharing, while progressive changes are still evident, the temporal relationships become more complex and harder to align with specific temporal positions. This highlights the role of parameter sharing in creating more structured attention patterns.

• Layer-wise Feature Extraction: Comparing the two layers of attention maps without parameter sharing, the first layer appears to focus on capturing basic temporal patterns, while the second layer refines and combines these patterns into more complex features.

[1] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations.

[2] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J. H., & Choo, J. (2021). Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.

[3] Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. Sharpness-aware Minimization for Efficiently Improving Generalization. In International Conference on Learning Representations.

[4] Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., & Redko, I. SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention. In Forty-first International Conference on Machine Learning.

To be continued...

[5] Chen, S. A., Li, C. L., Arik, S. O., Yoder, N. C., & Pfister, T. TSMixer: An All-MLP Architecture for Time Series Forecast-ing. Transactions on Machine Learning Research.

[6] Akiba, T., Sano, S., Yanase, T., Ohta, T., & Koyama, M. (2019). Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International conference on knowledge discovery & data mining.

[7] Bergstra, J., Yamins, D., & Cox, D. (2013). Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In International conference on machine learning.

We appreciate the reviewer's efforts and comments on our paper. Below, we offer detailed responses to each of your concerns.

W1: Writing errors present in the article.

Thank you for pointing out the writing errors. We will address them in the revised version of the manuscript.

Q1 and W2: Comparisons of convergence rates with and without parameter sharing; Illustrate the effectiveness of parameter sharing in time series forecasting.

Comparisons of convergence rates with and without parameter sharing. We compare the impact of parameter sharing on the convergence rate using the ETTh1 and Exchange datasets, recording the total number of epochs and the MSE loss under the same settings. (For the Exchange dataset, we use the current best-performing setup, specifically with the lookback window length in RevIN set to 16. For more details, please refer to our response to Reviewer 2(xcmx)'s W2.)

The experimental results are shown in the table below, where the values represent epochs/MSE loss. We observe that the number of epochs required with or without parameter sharing on the ETTh1 dataset varies depending on the prediction length. However, for the Exchange dataset, the convergence rate is faster without parameter sharing.

Additionally, in terms of MSE loss, using parameter sharing leads to greater reductions in loss and also results in fewer parameters (as discussed in our response to Reviewer 3(xM2V)'s W2). Therefore, there exists a trade-off between convergence rates, loss reduction, and parameter efficiency.

ETTh1

Exchange rate

Analysis of how parameter sharing affects model capacity. In the current encoder with parameter sharing, the PS_block network performs parameter sharing in seven places, including six networks for constructing the Q, K, and V matrices in the two-stage SegAtt, as well as the network for merging the output feature information of the two-stage SegAtt. This results in the encoder's parameter count under the parameter-sharing mode being one-seventh of that in the non-parameter-sharing mode. The total number of parameters in PSformer is the sum of the encoder's parameters and the final linear mapping. This means that as the number of encoders increases and the number of hidden layer nodes in the PS_block increases, the savings in total parameter count from parameter sharing become more significant. We have provided the total model parameter changes for different numbers of encoders; please refer to the response to Reviewer 3 (xM2V) regarding W2.

Q2: Providing detailed explanation of how the segmentation process preserves local temporal information while allowing for cross-channel interactions.

Maybe there was a misunderstanding as said "a token represents a down-sampled sequence along a channel", so we want to clarify as follows: The only instance we use "token" to describe sequence is in Appendix A.5, where we explain the reason for eliminating positional encoding. In this context, we discuss why positional encoding is not applied to segments, and we use a token to represent a temporal patch of a specific channel within a segment. It is not a down-sampled sequence but rather a local, continuous temporal sequence.

In the "The PSformer Framework" section of the paper, we provide a detailed explanation of the segment definition and the dimensional processing of SegAtt. We believe your confusion might mainly lie in how the PSformer Encoder handles local temporal dimension information and cross-channel information. To address this, we can perhaps clarify your concerns from both mathematical and visualization perspectives.

To be continued...

From the mathematical perspective: First, for each time series of length $L$, we divide it evenly into $N$ patches of length $P$, since $L = P \times N$. For $M$ time series, the $i$-th patch from each of the $M$ series is combined into the $i$-th segment, where $i \in \{1, 2, \dots, N \}$. Thus, dot-product attention is applied within each segment, with the attention matrix $QK^T \in \mathbb{R}^{C \times C}$, where $C = M \times P$. This indicates that attention is applied to local spatial-temporal segments.

Additionally, the process of capturing global temporal information across all $N$ segments is further refined. Specifically, this is achieved through the weights of PS Block, $W^S \in \mathbb{R}^{N \times N}$, which operate on all $N$ segments. More precisely, $W^S$ constructs $Q$, $K$, and $V$ matrices via matrix multiplication with the input data $X$. This allows the model to assign weights to different segments, enabling the selection of important segments from the global temporal space and extracting their local spatial-temporal information, which is then fused into global information during the final fusion stage of the PS Block. In summary, the overall process encompasses both global (cross-segment PS Block in SegAtt and the fusion stage PS Block) and local (segment-dimension attention) mechanisms.

From the visualization perspective. In the anonymous code repository's README, we further visualize and discuss the attention matrix of the first SegAtt, which may help in understanding the capture of local temporal and spatial information. In Figures 1.1 and Figure 1.2 of the README, you can observe the changes in attention weights within SegAtt across both channel and temporal dimensions.

Additionally, we further decompose the local spatio-temporal attention matrix into single-channel attention maps (e.g. Figure 2.1) and inter-channel pairwise attention maps (e.g. Figure 4.1). By comparing these maps with the corresponding temporal segments, we identify the temporal positions associated with high attention weights. This discussion may provide a more intuitive understanding of the mechanism behind the capture of local spatio-temporal information.

Thank you for your valuable comments, from which we benefit a lot for improving the work. Additionally, we have uploaded a new version of the paper, which integrates the content added during the rebuttal period to better showcase the paper's innovations, improving clarity, and enhancing the overall quality. We would be delighted to hear your further feedback.

We appreciate the thoughtful and helpful comments from the reviewer. Below, we offer detailed responses to each of your concerns.

W1: Evaluate segmentation numbers across more datasets; Providing further guidance on practical approaches for selecting segmentation numbers.

We further conducted tests on the selection of segmentation numbers using the ETTh2 and Exchange Rate datasets. The results are shown in the table below. For prediction lengths of 192 and 336, a segment number of 8 further improves the forecasting performance (compared with default segment number of 32). This suggests that there can not be a perfect fixed value for the choice of the number of segments. The choice of the number of segments may be influenced by the characteristics of different datasets, prediction lengths, as well as the overall increase in the scale of model parameters.

Therefore, for datasets without clear seasonality and with non-stationary characteristics (e.g., financial asset time series, including exchange rates), it is preferable to choose a larger segmentation number. This allows each segment to capture smaller local spatio-temporal patterns, better addressing unstable variation modes. On the other hand, for datasets with significant seasonality or relatively stationary characteristics (e.g., electricity and traffic), a relatively smaller segmentation number often facilitates model training.

ETTh2

Exchange Rate

W2: Validating the parameter-saving capacity and examining its implications for pre-trained models.

For further validating the framework’s parameter-saving capacity, we compared the parameter count of the PSformer under Parameter Sharing and No Parameter Sharing scenarios, including the comparison for 1-layer and 3-layer Encoders, as well as for different layers same as GPT2 models (GPT2-small, GPT2-medium, GPT2-large, and GPT2-xl) at 12-layer, 24-layer, 36-layer, and 48-layer configurations. The results are reported in the table below. In addition, if the hidden layer dimension is expanded from 32 to 1024 (as in GPT2), or if multi-head attention is adopted, the total number of parameters will also increase significantly.

W3: Provide multiple random seeds results to strengthen the robustness.

Below are the variances in MSE across 5 datasets. We repeated the experiments with 5 different random seeds. The complete metrics variations will be updated in the revised version of the paper.

To be continued...

W4: Channel Independence versus Channel-Mixing strategies.

In the discussion of PatchTST A.7, they suggest that predicting unrelated time series relies on different attention patterns, while similar time series produce similar attention maps. However, this does not fully explain why cross-channel temporal information cannot be used to improve forecasting performance. In fact, in many traditional multivariate time series models, it is common for multiple independent variables to determine the temporal patterns of the dependent variable. Additionally, they suggest that in the channel-independent mode, each time series generates its own attention maps, whereas in the channel-mixed mode, all time series share the same attention patterns.

Regarding these issues, in the ReadMe of our anonymous code repository, we analyze attention maps in Appendix B.4 of the paper in terms of local temporal and spatial attention. We find that in our channel-mixed mode, the initial attention maps can be decomposed into components that capture single-variable time series information, similar to the channel-independent mode, and additional components that capture cross-channel spatio-temporal patterns.

For example, (Figure 1.1 in ReadMe.md), we observe distinct high-attention regions (e.g., the top-left corner) and high-attention areas between different time steps and variables (e.g., the non-diagonal symmetric grid section in the top-left corner). From this, we can see that the coordinate positions within the spatial attention submatrix (inside the grids) exhibit the same or gradually changing attention weights across different time steps (between the grids). This suggests that the model may capture temporal consistency between variables, local stationary features, and long-term dependencies across time steps. We further separate channel-mixed attention maps into attention maps for individual channels and cross-channel attention maps. More related visual analysis and discussion can be found in the anonymous repository.

Besides, We think the challenges associated with multi-channel techniques, such as the risk of overfitting during training and the increased computational cost of attention, should not hinder their use. Instead, these issues should be addressed through other methods, such as parameter sharing and sparse attention, and so on, to mitigate these concerns.

W5: Which components contribute to its computational advantages over other state-of-the- art models?

Regarding computational efficiency, there are several components in the framework that contribute to its advantages over other state-of-the-art models:

• Parameter Sharing Techniques: The use of parameter sharing reduces the overall number of parameters that need to be optimized, leading to improved computational efficiency without compromising performance.

• Segment Attention Mechanism: By segmenting the input sequence and applying attention at the segment level, the input dimension for the ps_block is reduced to the number of segments, significantly decreasing the parameter count and computational overhead.

• Robustness Without Dropout or Positional Encoding: The inherent robustness of the model eliminates the need for operations like dropout and positional encoding, which are commonly used in other models. This further reduces computational costs while maintaining performance.

Thank you for your valuable comments and acknowledgement of our efforts. We learned a lot from your feedback and we have also uploaded a new version of the paper, which integrates the content added during the rebuttal period to better showcase the paper's innovations, improving clarity, and enhancing the overall quality. We would be delighted to hear your further feedback.

We thank the reviewer for the valuable comments on our paper. Below, we offer detailed responses to each of your concerns.

W1: The accuracy improvement is marginal, especially in ablation studies, and variances in metrics should be reported.
For the performance improvement of the model, it is important to highlight that many of the baselines we selected are recent and highly competitive models, making accuracy improvements both challenging and meaningful. Regarding the relatively marginal improvements observed in the ablation studies, This is because we conducted experiments in four aspects within the ablation study:

• The first two aspects focus on analyzing the impact of the number of segments and encoders in PSformer. From Table 3 and Table 4, the observed changes in these areas are indeed minor (with an average MSE variation of 0.002 on the ETTh1 and ETTm1 datasets), demonstrating that PSformer is robust to these two critical hyperparameters.

• The latter two aspects involve ablation studies on the model's key innovations: parameter sharing and segment attention. From Table 5 and Table 6, it can be observed that the variances in metrics are highly significant. Specifically, parameter sharing achieved an average MSE reduction of 0.016 on ETTm1, while SegAtt achieved an average MSE reduction of 0.017 on ETTh1. These results demonstrate that the two core contributions of PSformer significantly improve its performance, making it more competitive compared to a wide range of baseline models.

Below are the variances in MSE across 5 datasets. We repeated the experiments with 5 different random seeds. The complete metrics variations will be updated in the revised version of the paper.

W2: Why PSformer not work well on exchange?

We conducted a relevant analysis based on the work of DLinear[1] and SAN[2] and believe that the reason for not work well on the exchange rate dataset is related to the characteristics of the exchange rate data and the application of RevIN[3]. Financial time-series data, including exchange rates, typically exhibit non-stationarity and random walk models. Therefore, when using RevIN to estimate relatively stable mean and variance statistics based on past windows, it becomes difficult, as the mean and variance values calculated by RevIN are highly sensitive to the length of the statistical look-back window. This sensitivity is a key reason for PSformer currently does not work well on exchange rate dataset.

To be continued...

To validate this hypothesis, we kept the overall experimental setup for the exchange rate data unchanged (i.e., with an input length of 512) while varying the look-back window length used by RevIN to calculate the mean and variance. The specific window lengths tested were [16, 64, 128, 256, 512], and we compared the final prediction MSE loss, as shown in the table below. From the results, we observed that by simply reducing the look-back window for RevIN's statistical calculations, the prediction loss improved significantly. When the window length was set to 16, the average loss for the exchange rate was 0.3575, which is a 16.5% reduction compared to the current loss. This interesting phenomenon validates our idea that selecting an appropriate look-back window for RevIN based on the characteristics of different datasets helps identify more stable statistical means and variances, thereby facilitating model training. A deeper analysis of this phenomenon is worth further investigation in future research.

[1] Zeng, A., Chen, M., Zhang, L., & Xu, Q. (2023). Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence.

[2] Liu, Z., Cheng, M., Li, Z., Huang, Z., Liu, Q., Xie, Y., & Chen, E. (2024). Adaptive normalization for non-stationary time series forecasting: A temporal slice perspective. Advances in Neural Information Processing Systems.

[3] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J. H., & Choo, J. (2021). Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.

Thank you for the support of our work. You comments have greatly helped to improve the quality of our work. By the way, we have uploaded the new version of manuscript, which integrates the content added during the rebuttal period to better showcase the paper's innovations, improving clarity, and enhancing the overall quality. We would be delighted to hear your further feedback.

We appreciate the reviewer's thoughtful comments on our paper. Below, we offer detailed responses to each of your concerns.

W1: The article has some minor errors, and couldn't find Table 27.

Thank you for pointing out this issue. This is actually a file compilation problem—Table 27 refers to Table 11 for the full long-term forecasting results. We apologize for the misunderstanding caused by this and will fix this issue in the next manuscript version.

W2: Lack research on important references, such as TimeMixer, FITS, CrossGNN, FourierGNN and MICN.

Thank you for your advice. In recent years, there have been many notable and excellent works in the time-series forecasting field, such as TimeMixer[1], FITS[2], and GNN-based methods like CrossGNN[3] and FourierGNN[4] and MICN[5]. When selecting baseline models for our work, we considered three main factors:

1. First, since our work draws inspiration from SAMformer[6] by applying SAM[7] optimization techniques to time-series training, we wanted to include SAMformer, which also uses SAM optimization, alongside TSMixer[8] as part of our baseline. This allows us to more clearly demonstrate the performance of our work under the same optimization method.

2. At the same time, we wanted to include as many recent works from various time-series models as possible to present a comprehensive evaluation, including iTransformer[9], SAMformer, MOMENT[10], GPT4TS[11], and ModernTCN[12].

3. Additionally, some important works should not be overlooked, such as PatchTST[13], FEDformer[14], and Autoformer[15]. RLinear[16] is also crucial as it demonstrates the forecasting performance of pure linear mappings.

Since papers in the time-series field typically use around 10 models as baselines, we selected the current baseline models to allow for a more comprehensive comparison of our work. However, this does not imply that other models are of lesser reference value. In fact, they are equally important and help readers gain a more complete understanding of the latest developments in the time-series forecasting field. Therefore, we will improve the inclusion of more important models and works in the revised version of our paper.

W3: The PSformer should compare with models mentioned in W2 and with TimesNet.

We have gathered and organized these excellent recent works for a more comprehensive comparison in the table below:

ETTh1

ETTh2

ETTm1

To be continued...

ETTm2

Weather

Electricity

Exchange Rate

Traffic

To be continued...

We have collected the main experimental results of the relevant models. We did not include FourierGNN, as it did not report performance on most of the benchmark datasets. Although there are differences in experimental setups across each work, which may affect the results and prevent a completely fair comparison, we have provided some key settings to help better understand the model performance.

For each model, the name is followed by two values in parentheses: the first value represents the look-back window mode (with "F" indicating fixed input length and "M" indicating a grid search was performed across different input lengths), and the second value indicates whether the test set dataloader drops the last batch of data (noDL: does not drop last; DL: drops last).

From the results, it is evident that:

• Compared to models with fixed windows, PSformer performed best on 7/8 of the prediction tasks. This further highlights PSformer's competitive performance in forecasting.

• Even when compared to non-fixed window models like FITS, PSformer performed best on 4/7 of the prediction tasks.

• Regarding the reason why the PSformer model did not achieve the best performance on the Exchange dataset, we provided a discussion in Reviewer 2 (xcmx)'s W2. Additionally, after reconstructing the RevIN[17] window, we achieved up to a 16.5% reduction in loss and achieved competitive performance compared with selected baseline models.

Besides, we suggest that the main purpose of comparing more models is not only to pursue better forecasting performance but also to provide readers with a more comprehensive understanding of the latest research advancements in this field. It is important to draw greater attention to the innovative techniques behind these models.

W4: Providing details about the experimental platform; Specify the parameter search space; Have the best model parameters been determined using the validation set?

Experimental Setup. All experiments were conducted on two servers, each equipped with an 80GB NVIDIA A100 GPU and 4 Intel Xeon Gold 5218 CPUs.

Parameter Search Space. As stated in the paper, PSformer has a parameter-efficient structure, with high robustness and a limited number of hyperparameters. The main hyperparameters of PSformer include: 1. the number of encoders, 2. the number of segments, and 3. the SAM hyperparameter rho. For the number of encoders, we primarily searched within 1-3 layers. For the number of segments, we maintain the same as the number of patches in PatchTST，we also analyzed values that divide the input length evenly (specifically: 2, 4, 8, 16, 32, 64, 128, 256), and ultimately set all prediction tasks to 32 to avoid performance improvements caused by complex hyperparameter tuning. For the SAM hyperparameter rho, we referred to SAMformer and performed a search across 11 parameter points evenly spaced in the range [0, 1] (please refer to PSformer Appendix B.1 and B.2 for details). The number of encoders and segments can be found in the ablation study in section 4.2. Additionally, for the learning rate, we mainly tested values of 1e-3 and 1e-4, and for the learning rate scheduler, we tested OneCycle[18] and MultiStepLR.

Best Model Parameters. Regarding whether the best model parameters were determined based solely on the validation set, We cannot guarantee that all our model hyperparameters were exclusively determined using the validation set (with no reference to the test set performance). This might be a common issue in the time-series domain. However, We would like to emphasize that our model hyperparameters are relatively less and robust, and we try to avoid excessive hyperparameter tuning, as shown in Table 7 in Appendix A.2.

To be continued...

[1] Wang, S., Wu, H., Shi, X., Hu, T., Luo, H., Ma, L., ... & ZHOU, J. (2024). TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[2] Xu, Z., Zeng, A., & Xu, Q. (2024). FITS: Modeling Time Series with Parameters. In The Twelfth International Conference on Learning Representations.

[3] Huang, Q., Shen, L., Zhang, R., Ding, S., Wang, B., Zhou, Z., & Wang, Y. CrossGNN: Confronting Noisy Multivariate Time Series Via Cross Interaction Refinement. In Thirty-seventh Conference on Neural Information Processing Systems.

[4] Yi, K., Zhang, Q., Fan, W., He, H., Hu, L., Wang, P., ... & Niu, Z. FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective. In Thirty-seventh Conference on Neural Information Processing Systems.

[5] Wang, H., Peng, J., Huang, F., Wang, J., Chen, J., & Xiao, Y. (2023). Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations.

[6] Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., & Redko, I. SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention. In Forty-first International Conference on Machine Learning.

[7] Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. Sharpness-aware Minimization for Efficiently Improving Generalization. In International Conference on Learning Representations.

[8] Chen, S. A., Li, C. L., Arik, S. O., Yoder, N. C., & Pfister, T. TSMixer: An All-MLP Architecture for Time Series Forecast-ing. Transactions on Machine Learning Research.

[9] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations.

[10] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. MOMENT: A Family of Open Time-series Foundation Models. In Forty-first International Conference on Machine Learning.

[11] Zhou, T., Niu, P., Sun, L., & Jin, R. (2023). One fits all: Power general time series analysis by pretrained lm. Advances in neural information processing systems.

[12] Luo, D., & Wang, X. (2024). Moderntcn: A modern pure convolution structure for general time series analysis. In The Twelfth International Conference on Learning Representations.

[13] Nie, Y., Nguyen, N. H., Sinthong, P., & Kalagnanam, J. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In The Eleventh International Conference on Learning Representations.

[14] Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., & Jin, R. (2022). Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning (pp. 27268-27286). PMLR.

[15] Wu, H., Xu, J., Wang, J., & Long, M. (2021). Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34, 22419-22430.

[16] Li, Z., Qi, S., Li, Y., & Xu, Z. (2023). Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721.

[17] Kim, T., Kim, J., Tae, Y., Park, C., Choi, J. H., & Choo, J. (2021). Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations.

[18] Smith, L. N., & Topin, N. (2019). Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications.

Thank you for your valuable comments, and we have tried our best to clarify the concerns on the paper. Additionally, we have uploaded the rebuttal version of the paper, which integrates the content added during the rebuttal period to better showcase the paper's innovations, improving clarity, and enhancing the overall quality.

With the discussion period nearing completion in less than two days, we would appreciate it if you could let us know if any aspects remain unclear. We truly appreciate this opportunity to improve our work and shall be most grateful for any feedback you could give us.

Thank you for your feedback and patience. We have summarized your concerns into the following four questions. For the convenience of reading, we restate the meanings of the related abbreviations as follows:

• DL: The test set dataloader in the model experiment is set with drop_last=True.
• noDL: The test set dataloader in the model experiment is set with drop_last=False.
• F: The input sequence length in the model experiment is fixed.
• M: The input sequence length in the model experiment is determined through grid search across different lengths.

Q1:Models with noDL and DL make it difficult to ensure the fairness of the experiments.

1. First of all, the experimental results of these baseline models used for comparison are taken from the results reported in their corresponding original papers, including TimeMixer, CrossGNN, MICN, TimeNet, and FITS. The settings for DL or noDL are derived from the official code repositories of these models, and I only present these settings in the table for better comparison. Similarly, the settings for F or M are also determined by the experimental configurations and official code of the respective papers. A common understanding is that the performance of models under the DL or M settings tends to be better than under the noDL or F settings.
2. For more discussion on the impact of noDL and DL settings, I recommend two resources to better understand how this configuration affects model performance:

• The most direct approach is to analyze line 32 (drop_last=False) in the data_factory.py file of the current version of the Time-Series-Library and verify its impact on prediction performance under different settings. Of course, directly reviewing the relevant code sections of each model can also help in understanding this configuration.
• Additionally, we appreciate and recommend the discussion in TFB [1], specifically Issue 3 in the introduction section, as well as the discussion in the README of the FITS[2] official code repository. These address the effect of the DL setting, which drops the data from the last batch of the test set. From Table 2 in TFB, it is shown that as the number of samples in the last dropped batch increases, the predictive loss caused by DL continuously decreases. This impact is related to factors such as data stationarity, batch size, and input length.
  A general opinion is that for non-stationary datasets, the DL setting is likely to have a more significant effect on model performance as the batch size increases. This is because a larger batch size may result in more samples being dropped in the last batch, and the dropped samples tend to be more challenging to predict due to the non-stationarity of the data. Consequently, this can lead to a noticeable impact on the overall MSE loss for small-scale datasets.

3. Unifying all baseline models under the noDL and F settings and testing the corresponding experimental results may yield outcomes different from those reported in the original papers. However, we have not done so previously for two main reasons:

• The limited time during the initial rebuttal phase setting and the overall time allocation for addressing all comments.
• We consider PSformer to be based on the noDL and F settings. From the perspective of hypothesis inference, If we agree that the DL issue leads to reduced predictive loss, and that the M setting also leads to reduced predictive loss compared to the fixed input length F setting, then if PSformer outperforms the baseline models in the DL or M settings, it would still outperform those baselines under the noDL and F settings.

[1] Qiu, X., Hu, J., Zhou, L., Wu, X., Du, J., Zhang, B., ... & Yang, B. (2024). TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. VLDB2024.

[2] Xu, Z., Zeng, A., & Xu, Q. FITS: Modeling Time Series with $10k$ Parameters. ICLR2024.

Q2:Do the baseline models selected in the original paper also suffer from DL issues or M issues?

We stated the baseline settings in Section A.2. The experimental results for ModernTCN come from our own experiments. The difference between our experiment and the official implementation of ModernTCN is that we standardized the look-back window to 512 and set drop_last=False for the test set in the dataloader. The results for other baselines were collected from the SAMformer[1], iTransformer[2], and MOMENT[3] papers. Since the DL issue gained broader recognition this year, these baseline models also suffer from DL issues. However, all baseline models use a fixed input length.

[1] Ilbert, R., Odonnat, A., Feofanov, V., Virmaux, A., Paolo, G., Palpanas, T., & Redko, I. SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention. ICLM2024.

[2] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In The Twelfth International Conference on Learning Representations. ICLR2024.

[3] Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., & Dubrawski, A. MOMENT: A Family of Open Time-series Foundation Models. ICML2024.

Q3：Fixed parameters across different tasks is challenging to guarantee that the optimal parameters.

Our original statement is: “For the number of segments, we maintain the same as the number of patches in PatchTST，we also analyzed values that divide the input length evenly (specifically: 2, 4, 8, 16, 32, 64, 128, 256), and ultimately set all prediction tasks to 32 to avoid performance improvements caused by complex hyperparameter tuning. ”

With this, we intend to convey two key points:

1. Although we analyzed the predictive performance of models with different segment lengths, we ultimately set the number of segments to 32 across different tasks.

2. This does not mean that we believe the same hyperparameters should be used for all tasks. Instead, the choice of hyperparameters depends on their specific characteristics and how they influence predictive performance.

Specifically, PatchTST is an important early work that introduced the patching technique to the time series forecasting domain. Following this, a series of subsequent works adopted this idea. We identified related models with complete running scripts for various datasets in their official code repositories, such as ModernTCN, GPT4TS, TimeXer[1], and TimeLLM. These works use patching and its derived methods for temporal dimension, aiming to enhance the extraction of local features in time series data.

We have summarized the patch parameters used in these models across different datasets and prediction tasks in the table below. All data is sourced from the respective official code repositories. The values in the table represent the patch length for each case. Since we set the segment number to 32 and perform non-overlapping segmentation along the 512, the length of each segment in the temporal dimension is also 16.

Q4：they cannot guarantee that all their model hyperparameters were exclusively determined using the validation set (with no reference to the test set performance).

1. We have compiled information from models presented at top conferences regarding whether they determine the best model parameters on the validation set, as well as their methods for hyperparameter tuning as follows:

• Regarding ModernTCN. There is no information about determining the best model parameters on the validation set. In the Model Parameters section, it defaults to using 1 block and stacks 3 blocks for larger datasets (ETTm1 and ETTm2). For small datasets, it recommends using a smaller FFN ratio to mitigate overfitting.

• Regarding PatchTST. There is no information about determining the best model parameters on the validation set. In the Model Parameters section, it defaults to three encoder layers with 16 heads and a latent space of size 128. For small datasets, the parameter count is reduced to prevent overfitting, and dropout=0.2 is used for all experiments.

• Regarding TimeLLM. There is no information about determining the best model parameters on the validation set. In Appendix B.4, they provide a table of model hyperparameters, which remain largely consistent across different long-term forecasting tasks.

• Regarding SAMformer. There is no information about determining the best model parameters on the validation set. In A.1, they use a one-layer transformer, a batch size of 32, set the model dimension to 16, and keep it consistent across all tasks.

• Regarding TimeMixer. There is no information about determining the best model parameters on the validation set. Table 7 shows their hyperparameter settings, and in Appendix D, they evaluate the number of layers, ultimately setting it to 2 to balance efficiency and performance.

• Regarding CrossGNN. There is no information about determining the best model parameters on the validation set. In A.7, they set the scale number to 5 and K to 10 for all datasets, with the channel dimension set to 8 for smaller datasets and 16 for larger datasets.

• Regarding MICN. There is no information about determining the best model parameters on the validation set. In A.2, they set the batch size to 32 and the hyperparameter i to {12, 16}.

• Regarding TimesNet. There is no information about determining the best model parameters on the validation set. Table 7 shows that they use the same hyperparameters for all long-term forecasting tasks.

• Regarding FITS. FITS is the only model that explicitly emphasizes performing parameter grid search on the validation set. In Section 4.2, they state that they choose the hyper-parameter based on the performance of the validation set. From their code scripts, these parameters likely include batch size, sequence length, random seed, and other hyperparameters related to the model. However, as they candidly acknowledged in their README, they initially did not address the DL bug, and their validation set configuration still has drop_last=True. This might have amplified the impact of the DL issue on the model during the parameter search, even though they conducted the parameter search on the validation set.

2. Our Hyperparameter Settings. Similar to most works presented at top conferences, as stated in Table 7 and B.12 of our paper, the number of encoders is set to 1 by default and adjusted to 3 for larger datasets. For the parameter $\rho$ in SAM, we adopt the same setting as SAMformer. In C.2 of their paper, they note that SAMformer has smooth behavior with $\rho$, and we have also verified this in B.1 of our paper. Regarding the choice of segment number, we follow the common practice of patch-based methods, setting the segment length in the time-series dimension to 16.

Q5: More discussion on hyperparameter search.

Thank you for your feedback, and we appreciate your rigor and dedication. We are pleased to hear that we have reached a consensus on the issue of noDL and DL, leaving the primary concern to be the application of hyperparameter search techniques in time series model and a fairer comparison with FITS. Our response to this concern can be divided into three main aspects:

• We provide experimental results comparing FITS using the same experimental settings of PSformer.

• We present the experimental results of PSformer following FITS in hyperparameter search on the validation set.

• Finally, we summarize and further discuss the pros and cons of hyperparameter search on the validation set.

1. Comparison of PSformer and FITS under the same noDL and F settings.

   Following the experimental settings described in Section A.2 of the paper, we report the results of FITS under seq_len=512 and noDL. Due to the small parameter size of FITS and its fast runtime, we evaluated its performance across 8 datasets. Given the multiple versions of runtime scripts in the official repository, we prioritized using the "Best" script where available; for datasets without a "Best" script, we selected other available scripts. As there were no scripts provided for the Exchange Rate dataset, we followed FITS's hyperparameter search methodology to identify the best experimental settings in terms of cut-off frequency, batch size, and training mode.

   The summarized experimental results are presented in the table below. From these results, we observe that the performance of FITS under the F setting is not significantly different from that under the M setting. The predictive performance degrades primarily on the ETTm2 dataset, suggesting that frequency-domain-based models may be less sensitive to input lengths in the time domain. Overall, under the experimental settings of Section A.2, PSformer outperforms FITS on 6 out of the 8 mainstream datasets.

2. Experimental comparison of PSformer with hyperparameter search on the validation set.

   To better demonstrate PSformer’s performance under hyperparameter search, we followed FITS’s hyperparameter search methodology from its experiments. Since the FITS model consists of only a single linear layer, its training speed is very fast. However, most Transformer-based models require more time for parameter search. To better reflect PSformer’s performance within a limited timeframe, we restricted the search to essential hyperparameters. The hyperparameter search space was based on W4 and extended to include six input lengths: [64, 96, 384, 512, 640, 768], ensuring non-overlapping segment divisibility.

   We conducted experiments on ETTh2， Exchange and ETTh1 datasets to capture the performance on distinct time series characteristics. To address the engineering complexity introduced by the high-dimensional hyperparameter space, we used Optuna, optimizing for validation set loss as the objective, instead of grid search. These settings allowed us to present experimental results promptly, as summarized in the table below.

   From the results, PSformer outperforms FITS in most prediction tasks, except for the experiment on ETTh1 and ETTh2 with pred_len=720. This may be due to FITS' ability to filter out minor frequency components, reduce noise interference, and better capture long-term patterns through transformations from the time domain to the frequency domain. Additionally, PSformer demonstrates superior predictive performance on datasets like Exchange, which lack clear seasonality and stationarity.

3. Summary and further discussion on the pros and cons of hyperparameter search on the validation set.

   Like two sides of a coin, based on our experimental experience, we summarize the advantages and disadvantages of hyperparameter search as follows:

   Advantages:

   • As discussed above, performing hyperparameter search on the validation set effectively prevents overfitting hyperparameters to the test set data.
   • Moreover, adjusting the model’s hyperparameters to an optimal state based on the characteristics of the validation set helps improve model performance in a targeted manner according to the dataset's specific traits.

   Disadvantages:

   • Regarding time series data. Due to the non-stationarity of time series data and the strict chronological order used in dataset splits, the three subsets often exhibit time-varying statistical characteristics. Even the validation set should minimize hyperparameter overfitting, and the definition of the search space requires caution.

   • Regarding model types. For FITS, as a frequency-domain-based model, the input length in the time domain may mainly act as a hyperparameter for converting information to the frequency domain. As a result, FITS can employ various training modes regarding input length, including converting long sequences formed by concatenating X and Y into frequency-domain information or further fine-tuning pre-trained parameters on Y. However, the impact of input length on time-domain models is likely more complex, requiring careful consideration when treating input sequence length as a hyperparameter.

   • Regarding engineering effort. One of FITS’s key advantages is its small parameter size, typically consisting of only a fully connected layer for frequency upsampling, making it highly efficient to train. In contrast, Transformer-based models, CNN-based models, or large pre-trained models may face substantial engineering challenges from hyperparameter search, especially on large datasets like traffic, significantly increasing research difficulty.

Thank you for your constructive comments and feedback on our work. We have made efforts to address each of your concerns, and we are pleased that we reached a consensus on most of them. We also acknowledge that a more complete comparison is needed. In response, we have further organized the experimental results. In the table below, we present the unified experimental results of these models under the F and noDL settings, where the results for MICN and TimesNet are from Leddam[1]. To better illustrate the comparison between PSformer and each model, we use MSED to represent the degree of MSE loss reduction of PSformer compared to the corresponding baseline model for each prediction task. Apart from FITS, PSformer outperforms most models in the majority of tasks, and only underperforms MICN on 4 prediction tasks and GrossGNN on 3 prediction tasks. When compared to FITS, PSformer performs better than 22 prediction tasks, with MSED greater than 0.005 in 16 tasks, and only 2 tasks where MSED is below -0.005.

Additionally, regarding the results of the hyperparameter search, we further conducted experiments on other datasets. We have organized the results for ETTh1 and combined them with the previous results in Q5. It can be observed that the results on ETTh1 are similar to those on ETTh2.

We hope our additional responses address your suggestion to include more methods for a fair comparison with PSformer. We are striving to present better experimental comparisons to strengthen your confidence in our work. Furthermore, We acknowledge that we have not achieved the best performance in all prediction tasks, so we will replace the SOTA claim with the statement that our work demonstrates competitive performance. Thank you again for your rigor and contributions. We believe that by incorporating your valuable feedback, we can further improve our work. We also hope that our work will be seen by a wider audience in the community and receive more support from you. Once again, thank you for your contribution to our work.

[1] Yu, G., Zou, J., Hu, X., Aviles-Rivero, A. I., Qin, J., & Wang, S. Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling. ICML2024