Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting

We would like to sincerely thank Reviewer LNPN for acknowledging our technical novelty and empirical contributions, as well as the comments regarding larger datasets and the related multi-scale baseline method. We have revised our paper accordingly.

Q1: Results on larger datasets.

A1: We seek larger datasets from two perspectives: data volume and the number of variables. We add two datasets, the Wind Power dataset and the PEMS07 dataset, to evaluate the performance of Pathformer on larger datasets. The Wind Power dataset comprises 7397147 timestamps, reaching a sample size in the millions, and the PEMS07 dataset includes 883 variables. If the reviewer may mention other larger datasets, we are also willing to test on them. Pathformer also demonstrates superior predictive performance on these larger datasets compared with some state-of-the-art methods such as PatchTST, DLinear, and Scaleformer. We add experiments on these larger datasets in $\underline{\text{Section A.6 of the revised supplementary}}$.

Q2: Compare with Scaleformer.

A2: We mentioned the method of Scaleformer in $\underline{\text{the related work of the original submission}}$. In the revision, we add Scaleformer results in $\underline{\text{Tabel 1 of the revised paper}}$ as an important baseline of multi-scale models. We also provide a more detailed comparison between Scaleformer and our model in $\underline{\text{the related work of the revised paper and Section A.5.3 of the revised supplementary}}$, as follows:

Scaleformer also utilizes the modeling of multi-scale features for time series forecasting. It differs from our proposed Pathformer in the following aspects:

• Scaleformer employs fixed sampling rates, while Pathformer has the capability for adaptive multi-scale modeling based on the differences in time series samples.
• Scaleformer obtains multi-scale features with different temporal resolutions through downsampling. In contrast, Pathformer not only considers time series features of different resolutions but also models from the perspective of temporal distance, taking into account global correlations and local details. This provides a more comprehensive approach to multi-scale modeling through both temporal resolutions and temporal distances.
• Scaleformer requires the allocation of a predictive model at different temporal resolutions, resulting in higher model complexity than Pathformer.

Dear Reviewer LNPN,

We would like to sincerely thank you for your time and efforts in reviewing our paper.

We have made an extensive effort to try to successfully address your concerns, by conducting experiments on larger datasets with more timestamps and variables, comparing our proposed Pathformer with Scaleformer, and making revisions to the paper and appendix accordingly.

We hope our response can effectively address your concerns, If you have any further concerns or questions, please do not hesitate to let us know, and we will respond timely.

All the best,

Authors

We would like to sincerely thank Reviewer Sqch for acknowledging our technical contributions, providing the recent advanced method to compare with, and the suggestion for a more complete benchmark. We have revised our paper accordingly.

Q1: Compare with Time-series Dense Encoder (TiDE).

A1: The results of TiDE presented in the original paper are the best ones selected from different input sequence lengths of (48,96,192,336,720). Considering the urgency and fairness, we conduct a comparison under the condition of a fixed input sequence length of 96. We show results of some datasets here, and the complete results of other datasets and methods are available in $\underline{\text{Table 1 of the revised paper}}$.

Q2: More datasets in the benchmark.

A2: We add the ILI and Traffic datasets in the benchmark commonly used by previous papers such as PatchTST. The complete results of other compared methods can be found in $\underline{\text{Table 1 of the revised paper}}$.

We also evaluate the Exchange_rate dataset, which is proposed in the benchmark of the Autoformer paper. Our proposed Pathformer also outperforms Autoformer on this dataset.

Dear Reviewer Sqch,

We would like to sincerely thank you for your time and efforts in reviewing our paper.

We have exerted significant effort to effectively address your concerns, by including experiments comparing the proposed Pathformer to TiDE, adding ILI, Traffic and Exchange datasets in the benchmark, and making revisions to the paper and appendix accordingly.

We hope our response can address your concerns. If you have any further concerns or questions, please do not hesitate to let us know, and we will respond timely.

All the best,

Authors

We also conduct experiments to compare the generalization capabilities of Pathformer and DLinear, where we train the model on one dataset and test the performance on other datasets. The specific experimental setting is the same with Section 4.2 of the paper.

The results of the generalization on other datasets:

In the generalization experiments, Pathformer still outperforms DLinear. This indicates that a relatively complex Transformer architecture may have better generalization capabilities than a simple linear model.

Q4: Compare with PatchTST

A4: We mention the source of inspiration of patching in $\underline{\text{Section 3 of our revised paper}}$. We also want to clarify that Pathformer extends from patch division to realize adaptive multi-scale modeling, which is a novel design. It is not a perturbation of PatchTST, as how the series are divided, and how the correlations are modeled are both designed differently. The main differences with PatchTST are as follows:

• Adaptive Multi-scale Modeling: PatchTST employs a fixed patch size for all data, hindering the grasp of critical patterns in different time series. We are the first to propose adaptive pathways that dynamically select varying patch sizes tailored to the dynamic features of individual samples, enabling adaptive multi-scale modeling.

• Partitioning with Multiple Patch Sizes: PatchTST employs a single patch size to partition time series, obtaining features with a singular resolution. Pathformer utilizes multiple different patch sizes for partitioning, which captures multi-scale features from various temporal resolutions.

• Global correlations between patches and local details in each patch: PatchTST performs attention between divided patches, overlooking the internal details in each patch. Pathformer not only considers the correlations between patches but also the detailed information within each patch. It introduces dual attention (inter-patch attention and intra-patch attention) to integrate global correlations and local details, capturing multi-scale features from various temporal distances.

We also add the above discussion in $\underline{\text{Section A.5.1 of the revised supplementary}}$.

Q5: Compare with NHITS

A5: NHITS also models multi-scale features for time series forecasting, and Pathformer differs from it in the following aspects:

• NHITS models time series features of different resolutions through multi-rate data sampling and hierarchical interpolation. Pathformer not only takes into account time series features of different resolutions but also approaches multi-scale modeling from the perspective of temporal distance. Simultaneously considering temporal resolutions and temporal distances enables a more comprehensive approach to multi-scale modeling.
• NHITS employs fixed sampling rates for multi-rate data sampling, lacking the ability to adaptively perform multi-scale modeling based on differences in time series samples. In contrast, Pathformer has the capability for adaptive multi-scale modeling.
• NHITS adopts a linear structure to build its model framework, whereas Pathformer enables multi-scale modeling in a Transformer architecture.

We add the above discussion in $\underline{\text{Section A.5.2 of the revised supplementary}}$. We also compare our performance with NHITS in $\underline{\text{Table 9 of the revised supplementary}}$

We would like to sincerely thank Reviewer 5cD6 for providing a detailed review and insightful comments regarding important basic baselines, more datasets, and more detailed comparisons with existing methods. We have revised our paper accordingly.

Q1: More discussion on some basic baselines.

A1: Thanks a lot for raising this valuable comment. We agree with the reviewer that we should also include some basic baselines as proposed in the comment, which may put our paper in a more correct context, give the readers a clearer view of the potentials and challenges of current Transformer methods, and make our contributions more convincing. We have made the following revisions according to your comment:

• We add these important basic baselines in $\underline{\text{Introduction or Related Work in the revised paper}}$, and rewrite these two parts accordingly.

• We compare the performance of our method with these baselines, such as DLinear, NLinear and NHITS, to make our empirical improvements more convincing. The detailed experimental results are in the following parts.

Q2: Include additional results on ILI and Traffic from PatchTST

A2: We add the ILI and Traffic datasets in $\underline{\text{Table 1 in the revised paper}}$. Here, we show some models with superior performance: PatchTST, DLinear, FEDformer, and Autoformer (bold indicates the best). Results of other compared methods are also included in the revised paper.

Q3: Include and compare with the results from Zeng et al.

A3: Some results presented in the paper of Zeng et al. (DLinear) are better than ours in Table 1, which is because that DLinear uses a larger input length (H=336) than ours (H=96). To ensure a fair comparison, we conduct separate experiments for input sequence lengths (H) of 96 and 336. Here, we show results of some models and datasets, with the complete results of other models and on other datasets available in the $\underline{\text{Table 1 in the revised paper and Table 9 in supplementary}}$.

The results for the input sequence length H=96:

The results for the input sequence length H=336:

The results above reveal that our proposed model Pathformer outperforms DLinear with both input sequence lengths of 96 and 336. Zeng et al. point out that previous Transformer cannot extract temporal relations well from longer input sequences, but our proposed Pathformer performs better with a longer input length, indicating that considering adaptive multi-scale modeling can be an effective way to enhance such a relation extraction ability of Transformers.

Dear Reviewer 5cD6,

We would like to express our sincere gratitude for your time and efforts in reviewing our paper.

We have made an extensive effort to try to successfully address your concerns. In our response:

• We provide more discussions with basic models, such as DLinear, NLinear, and rewrite the introduction and related work according to your suggestions.

• We add ILI and Traffic datasets from PatchTST.

• We compare our performance with these basic baselines on diverse datasets to make the results of the proposed pathformer more convincing.

• We also compare Pathformer with PatchTST and NHITS to show the novelty and effectiveness of Pathformer and make revisions to the paper and appendix accordingly.

We hope our response can address your concerns. If you have any further concerns or questions, please do not hesitate to inform us, and we will be more than happy to address them promptly.