Long-term Time Series Forecasting with Vision Transformer

  Thank you for your comment. We have carefully and seriously considered the questions and concerns you raised, and provided targeted explanations. We hope our response can change your perception of this work.

For the 1st concern of weaknesses

  The reviewer mentioned that Figures 1, 2, and 4 do not clearly convey insights. We do not quite understand. Do you have any clearer suggestions? For Tables 2, 3, and 4. Table 2 shows the results of univariate prediction, for which we only evaluated on the 4 ETT datasets. This is because in the field of time series prediction, univariate prediction is not the main challenge, and conventionally it is evaluated only on these 4 datasets or less. Anyway, we evaluated Swin4TS on the other four datasets as shown below (evaluated by MSE), and the results showed that Swin4TS still outperformed other baselines.

However, this comparison is unfair to other baselines because they have not conducted the same experiments in their original work, and they may require appropriate adjustments in hyperparameters for these additional datasets to achieve optimal performance. Table 3 is the ablation analysis for ETTm1 and ETTm2. We also conducted ablation experiments on Swin4TS/CI on four other datasets, and in most cases, the effectiveness was confirmed. Please understand that due to limited time, we were unable to complete all the ablation experiments.

It should be noted that presenting obvious ablation effects for all datasets is demanding. At least for time series prediction, it is customary to present ablation results only on some datasets. Table 4 is about the experiment on hierarchy design. We do not take the 4-stage experiment for the ILI dataset. This is because the input length of ILI is 108 which is unreasonable to design a 4-stage architecture, as we have mentioned in the main text. Overall, results of this work seem incomplete, primarily due to some customary practices rather than our work being incomplete. Hope the reviewer can consider again.

For the 2nd concern

   We would like to reiterate the motivation behind our work. Swin Transformer is essentially a Transformer that takes into account information at multiple scales (i.e., hierarchical design) and combines window-based attention to reduce complexity to linear. Although it was originally proposed to solve CV problems, it is essentially designed for handling sequential data. Time series is naturally sequential data, while image is partitioned into patch series. Therefore, from this perspective, Swin Transformer is even more suitable for handling time series.

   We want to emphasize that the core architecture of Swin Transformer is independent of CV tasks. It can be applied to a wide range of tasks for modeling sequential data. A good example of this is the Video Swin Transformer (VST, https://arxiv.org/abs/2106.13230), which actually deals with image-based time series. In fact, VST has not made specific designs for video data, but simply extends Swin Transformer from 2D to 3D, and has achieved SOTA performance on different tasks. If each frame of video data is reduced to a point or vector, then video data becomes time series. This means that VST already implies an attempt to process time series data with Swin Transformer. The main contribution of our work is showing the structural similarity of time series and image, making these two modalities be modeled within a unified framework.

(To be continued in Part 2/2)

(Following Part 1/2)

   Indeed, time series and images have their own characteristics. Or, as you mentioned, there is a gap between these two modalities. However, the characteristics of these two modalities can be integrated to their own underlying patterns. As long as Swin Transformer can learn these underlying patterns, there is no need to design too much specifically for their own features. As you know, ViTs do not have strong inductive bias like CNNs. But they still can get superior performance by training on large datasets. For example the shift invariance in image you mentioned. Due to that the partition of image in ViT destroys the space structure, ViT uses a positional embedding to learn the space structure. For time series analysis, at least for forecasting task, the shift invariance is not obvious. Thus the positional embedding and the relative bias in Swin Transformer do not facilitate the performance in practice. While for time series classification task, we expect to see the shift invariance may help the model performance.

   Anyway, thank you for pointing out this. Your concerns made us realize that we did not clearly articulate our motivation and contribution in the previous manuscript. In the Conclusion section of the revised one, we added a short discussion to comprehensively address our views. Please kindly consider again.

For the 3rd concern

   Regarding the performance incremental. The SOTA for time series prediction has reached a bottleneck. As far as we know, very recent works nearly all have limited improvements for PatchTST. Regarding the effectiveness of Swin4TS, we have additionally provided more experimental results to validate the feasibility and effectiveness of Swin4TS, such as TNT4TS and hierarchical analysis. We hope that these results will meet your requirements. Regarding the necessity of using backbones from other fields. As stated in our main text, the significance of our work lies in the recognition of the structural similarity between the time series and image modalities, and modeling them in a unified framework. Therefore, time series modeling can draw on advanced models from the image modality to solve its own problems, instead of designing various sophisticated models as before. This will promote the development of time series analysis. Finally, the reviewer suggested more helpful inductive bias to prove the rationality of Swin4TS. This is back to the 2nd concern mentioned above, which we have stated.

For the 4th concern

  Our wording may have been misleading. Our intention was to use a unified model to model time series and image patch series based on their structural similarities, rather than providing contributions for multi-modality task. In the revised manuscript, we expressed it in a different way.

For the 1st question

  Regarding the gap. When Swin Transformer is applied for image classification, it first obtains a representation of the image, and then uses this representation for classification by a linear layer. Similarly, Swin4TS obtains a representation of the input time series and then directly connects a linear layer to map the future time series. The main difference is just that the former is 2-dimensional and the latter is 1-dimensional.

   Regarding the representation granularity. Since existing time series datasets are far less complete than image datasets, large models may overfit. The size of Swin4TS is smaller than Swin4CV, which is mainly reflected in the smaller numbers of attention heads, layers, and stages and dimension of hidden vectors. However, this does not mean that the underlying pattern of time series is essentially low-level. The patterns of time series data, such as meteorological data, may be very complex, especially considering the interrelationships between multiple time series.

For the 2nd question

  Previous prediction models, such as Transformer, Autoformer, and Fedformer, can indeed be compatible with CI/CD. However, these models just fuse multiple channels with an embedding layer and do not explicitly consider the correlations between them, and thus the complexity of these models is not relevant to the number of channels. Swin4TS, on the other hand, can be compatible with both CI/CD strategies and can explicitly consider the correlations between multiple variables under CD strategy.

Thanks for your responses. The authors have addressed several questions with more detailed experiments and clarifications. And I would like to specify a few questions mentioned:

• About Figure 1: I think Figure 1 just shows the operation, it needs to show more correspondence. As the author clarified "times series also require division into patches to eliminate the randomness", how does the figure convey this?
• About Figure 2: It is hard to distinguish patches and windows by the line style. It can be more intuitive to use some color for blocks. It is better to explain the circle (time points localized in this or just sub-series in the period?) and avoid dense arrows.
• About Table 2. As far as I know, ETT is not a well-acknowledged univariate forecasting benchmark, which contains the covariates for predicting each other. Why not test them on a more widely accepted M4 benchmark as proposed in NBEATS[1]?

Considering the above responses, I am more convinced of the author's motivations. I think the contribution is enough if this work aims to show the structural similarity of time series and image and to make these two modalities benefit a unified framework (finding the same foundation model). However, it still needs more cases to support this (as the author newly included TNT), focus more on the conclusions instead of detailed structure, and provide some takeaway messages. If this article is to be written in this way, I think there are at least a few points that need further consideration:

• Time series forecasting needs not only locality. It has many properties (such as periodicity) to consider that images do not.
• The scale of the time series and image are also different. Time series do not have a fixed range like RGB domains.
• Not only does the attention model matter, but it is also an essential task to consider the other parts. The current projection head, which is a straightforward linear layer from the past features to the future prediction, can overfit on some of the small datasets of the paper since recent research find that a simple linear layer can especially work better on this [2].

Overall, I raised the score to 3. If the author can kindly take into consideration of the experiments and presentation of the paper, I will go further on this.

[1] N-BEATS: Neural basis expansion analysis for interpretable time series forecasting.

[2] Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping.

   First of all, thanks very much for your comments.

   Regarding the issue of dataset size, most existing baselines for time-series prediction use the 8 datasets mentioned in the text as benchmarks. We selected them in order to have a fair comparison with other baselines. However, as you mentioned, these datasets are relatively small and there are possible distributional shifts between the training and validation sets, which is a problem faced by all existing baseline models.

   In our collaboration project with a wind farm, we applied Swin4TS to predict wind power. The training dataset contained wind power data for 14 wind turbines, recorded every 15 minutes over a span of 5 years, comprising approximately 175k timesteps, far larger than the aforementioned 8 datasets. The goal was to predict wind power for the next 24 hours (96 points) and 72 hours (288 points), with an input size of 288. We compared three models, PatchTST, DLinear, and LightGBM. The prediction results are shown below (the results are in terms of normalized MSE).

   Swin4TS has performed well in predicting 96-point and 288-point. Unfortunately, due to commercial cooperation agreements, we cannot open this dataset. We believe that Swin4TS's superior architecture is the reason for its better performance rather than the accidental fit to specific datasets, and we expect it to perform well on other large datasets as well.

   Thanks for reminding us of the Scaleformer. Scaleformer and Swin4TS both use a hierarchy design, but their architectures are very different. Firstly, Scaleformer uses an external hierarchy design to obtain predictions at multiple scales, while Swin4TS uses an internal hierarchy design to concentrate information of different scales in the final representation and obtain the final prediction. Secondly, the prediction module in Scaleformer is totally independent, and Scaleformer is compatible with various transformer-based models. However, Swin4TS is a complete model, making it difficult to directly compare the performance with Scaleformer. Due to length limitations, we have added a short comment on Scaleformer in “Related Work” of the revised manuscript.

   Although you did not mention the contribution, we believe that the contributions of this work might be underestimated. We not only provide an effective time series prediction framework, but more importantly, it confirms the structural similarity of time series and image, which indicates the possibilities of modeling time series modality using architectures from image modality. We elaborated on this point in detail in our response to reviewer 9ZZq and SUuV. We also supplemented the design of a TNT4TS framework to support our view. We sincerely hope you can consider our work again.

   Thank you very much for your recognition of our work.

   Regarding the issue of the confidence interval for MSE and MAE that you mentioned, our main experiment setup uses a fixed random number, which is conventionally set to the number of current year. Therefore, MSE and MAE are the results of a single experiment, and there is no need to calculate the confidence interval. Of course, we have studied the impact of random seed (i.e. different initialization) on the model's performance, and the relevant results are in Appendix C.1. Since the results are presented in histogram form, there is no calculation of the confidence interval, please understand.

   For the selection of hyperparameters. We make similar datasets adopt the same hyperparameters, such as the 4 ETT datasets. Furthermore, different datasets may require different hierarchical designs, and the setting of hyper-parameters should ensure that the input length can be divisible during the downsampling process. In addition, other hyperparameters will reference the settings of other models, such as learning rate or early stop. Different datasets, due to their own properties, usually have different parameter sets. For example, in Traffic and Weather, the former’s periodicity is more obvious than the latter, and the interaction between multiple variables is also different.

   For the novelty. We borrowed the idea of Swin Transformer for modeling time series, and indeed, the novelty is not particularly strong if only considering this. However, our contribution is not only proposing an effective method for modeling time series. More importantly, we consider the structural similarity between image (partitioned into patch series) and time series, which indicates the possibilities of modeling time series modality using architectures from image modality. We additionally supplemented a TNT4TS framework to support our view. The reviewer can refer the detailed response to Reviewer 9ZZq and SUuV, and we hope the reviewer can reconsider the contribution of our work.

   Thanks very much for your review and suggestions. We next reply to your questions and concerns point to point.

For the 1st concern about weaknesses

  In fact, we have compared three non-transformer methods: DLinear (MLP-based), TimesNet (CNN-based), and MICN. Please see the description in the “Compare baseline” section.

For the 2nd concern

  Indeed, under the CD strategy, multiple time series can not be treated as an image, as the order of the time series can vary. In our code implementation, the order of the time series is shuffled in each training batch, and we found that this trick yielded better results than fixed initial order. We provided additional clarification on this point in the main text, and relevant experiments presented in the Appendix C.3.

For the 3rd concern

   The final layer of Swin4TS/CD is directly mapped to MT (M is the number of series, and T is the prediction length) after learning a representation using Swin Transformer. As you might know, this is not a good choice because for the 720-prediction task on the Traffic dataset, the size of final layer would be 862x720. This is also why the prediction performance on Traffic (M=862) and Electricity (M=321) is relatively poor under the CD strategy. A reasonable way to address this would be to design a U-net architecture and gradually obtain MT through upsampling on the representation. We added a part in Appendix D to address the effectiveness of this modified framework. However, to ensure the algorithm’s simplicity and consistency with the CI strategy, we still chose the simpler one as mentioned above. We did not want to complicate the main architecture because based on the motivation of this work, we hoped to demonstrate the structural similarity between the two modalities of time series and image and that they can be modeled using the same framework.

For the 4th concern

   Our results show that prediction performance varies non-monotonically with input length. In fact, PatchTST has also reported similar phenomena, as shown in Table 9. For example, for the 720-prediction task on ETTh2, 336-input leads to 0.379-MSE, 720-input leads to 0.394-MSE. Autoformer and FEDformer exhibit similar phenomena, as shown in Figure 2 of the PatchTST‘s paper. One possible explanation is that the input data contains both underlying patterns about the time series as well as noise. Once the input length is sufficient to capture the underlying patterns, longer input will introduce more noise and harm prediction performance.

For the 5th concern

   In our work, we gave the analysis of the hierarchy design in two parts. The first part is the Ablation Study in section 4.2 Results, where the hierarchy design was ablated in the manner just suggested by the reviewer, as described in the main text. The second part is the varying hierarchical design in section 4.3 Effect of Hyperparameter (we moved this part in Appendix), which is not ablation of the hierarchy design, but rather an examination of the impact of different hierarchy architectures on the model. Please note the distinction.

For the 6th concern

   Thanks for your suggestion. We have attempted to integrate dynamic covariates (the timestamp of the samples) into the input via embedding. Surprisingly, there was a significant improvement in predictive performance on the Traffic dataset! However, the effect was not observed on other datasets, which may be due to the Traffic dataset’s natural periodicity. We have incorporated this trick into the code, and added some descriptions of it in the Appendix C.6, and update the prediction results in Table 1.

For the 1st question

   We have addressed this in our previous response. We carefully considered your suggested approach and found that it is similar to our U-net solution. This design and relevant experimental results are supplemented in the Appendix D.

For the 2nd question

   We have answered this above.

For the 3rd question

   We appreciate your suggestions. In the revised manuscript, we cited these two works. However, it‘s hard for us to directly compare these two baselines in our main results. This is because the input size L is set to 720 for TiDE, while our Swin4TS sets L=512. It is unfair to compare with it directly. In N-HiTS, the input size is varied among datasets, which makes the comparison even more difficult. We hope the reviewer can understand this. In addition, after the use of dynamic covariates as you suggested, the prediction performance of Traffic of Swin4TS is comparable with TiDE.

For the 4th question

   We have addressed this above.

For your suggestion

   In addition, we analyze the information extracted by the hierarchical design at different scales. Please see Figure 6 and relevant descriptions.

   In the end, we hope the reviewer can reconsider the contribution of our work. We not just propose an effective model to handle time series forecasting problem. More importantly, we consider the structural similarity between image (partition into patch series) and time series, which makes these two modalities can be modeled in a unified architecture. Further, time series modeling can benefit from the development of ViTs, please see the extensive discussion with Reviewer SUuV. We added some disscusions at the end of the revised manuscript, and additionlly designed a TNT4TS model to support our view, and showed relevant experimental results in Appendix E.