ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

Thank you for your in-depth reviews and constructive suggestions. We appreciate the opportunity to clarify the distinctions between our approach and MOIRAI, as well as highlight the key contributions of our paper.

While there are similarities between our model and MOIRAI due to shared influences from image/video generation Transformer frameworks such as DiT [1] and SORA [2], there are significant differences in our objectives and approaches.

MOIRAI focuses on universal forecasting, emphasizing pre-training and transferability across datasets. It introduces any-variate attention and probabilistic predictions to enhance these capabilities. However, these features do not directly address the challenge of robust forecasting across arbitrary horizons. Moreover, while MOIRAI incorporates techniques from recent LLM architectures, such as vanilla RoPE, RMSNorm, and SwiGLU, these were applied without specific adaptations for time series. As shown in the global response (Figure 3), using vanilla RoPE directly in varied-horizon forecasting still presents significant challenges.

Our work specifically targets robust varied-horizon forecasting with three key designs:

(i) structured self-attention masks for consistent outputs across different forecasting horizons,

(ii) tunable rotary position embeddings for adapting to varying periodic patterns, and

(iii) multi-scale patch assembly to balance patch settings preferred by different horizons.

The full ablation study in the global response clearly demonstrates that each of these designs plays a crucial role in enabling robust varied-horizon forecasting:

(i) Structured attention masks ensure reliable inference on horizons that differ from the training horizon (Figures 1 and 2).

(ii) Tunable RoPE is crucial for enhancing the model’s extrapolation capability (Figure 3).

(iii) Multi-scale patch assembly reduces sensitivity to patch size hyper-parameters across different horizons (Figure 4).

While our model also supports any context length, this feature was not highlighted in the manuscript as it is less central to our key technical contributions.

In summary, ElasTST and MOIRAI address different challenges in time series forecasting, each offering solutions to distinct problems. In the future, our methods could potentially be integrated within a broader framework to enhance both generality and robustness in time series forecasting.

---

Improve the presentation of this paper

1. The statement, "we find that on the challenging datasets such as Weather and Electricity, dataset-specific tuning still offers advantages" (L238-L239) is a little eyebrow raising, in the sense that it is stating the obvious. I believe most researchers expect in-distribution predictions to outperform zero-shot predictions. Such statements should be rephrased.
2. Effects of tuning rotary embeddings in figure 4 seem very marginal, error margins should be provided for such cases.

Thank you for your valuable suggestions.

We agree that some statements could be better phrased. Specifically, regarding the statement in L238-L239, our intention was to emphasize that while the zero-shot capability of foundation models is impressive, additional dataset-specific tuning may still be necessary for challenging datasets to achieve optimal forecasting performance. We will revise these statements in the manuscript to convey our findings more clearly and accurately.

In the experimental section, we will incorporate key results from the global response into the main text and add error margins to Figure 4 to improve result representation.

We appreciate your feedback and will refine the paper accordingly to ensure a clearer and more thorough presentation in the revised manuscript.

[1] Peebles, W., & Xie, S. (2023). Scalable diffusion models with transformers. ICCV.

[2] Tim Brooks et. al., (2024). Video generation models as world simulators.

Dear reviewer,

The author-reviewer discussion ends soon. If you need additional clarifications from the authors, please respond to the authors asap. Thank you very much.

Best,

AC

Thanks for your in-depth reviews, insightful questions, and constructive suggestions. We hope the following responses can help to address all your questions and concerns.

Response to Weakness 1

A comparison with methods such as no positional embedding, vanilla positional encoding, trainable positional embedding matrices, and original rotary positional embeddings, would clarify the benefits of different approaches. Since tunable RoPE appears easily transferable, demonstrating its performance enhancement by applying it to previous models like PatchTST and Autoformer would provide more intuition.

Thank you for highlighting this gap in our experiments. We have updated Figure 3 in the global response to include the performance of ElasTST with various positional embeddings. The results demonstrate that Tunable RoPE significantly enhances the model’s extrapolation capabilities. While other positional embeddings perform well on seen horizons, they struggle when the forecasting horizon extends beyond the training range. Although the original RoPE performs well in NLP tasks, it falls short in time series forecasting. Dynamically tuning RoPE parameters based on the dataset’s periodic patterns proves highly beneficial, especially for unseen horizons, with the benefits becoming more pronounced as the inference horizon lengthens.

We appreciate your suggestion to apply Tunable RoPE to previous models like PatchTST and Autoformer. However, PatchTST requires per-horizon training and deployment, making it difficult to showcase the benefits of Tunable RoPE since it cannot handle longer inference horizons once trained for a specific horizon. Essentially, our architecture—after removing the tunable RoPE and multi-patch designs—can be viewed as a version of PatchTST that adapts to varied horizons. As for Autoformer, it replaces the self-attention mechanism with an auto-correlation mechanism that inherently captures sequential information, making positional embeddings unnecessary.

Your feedback has been invaluable in refining our work, and we will update the manuscript to include these discussions.

Response to Weakness 2

Further analysis like how much adding a patch size can increase computational costs or what patch size combination are more reasonable to choose under resource constraints would be beneficial.

ElasTST uses a shared Transformer backbone to process all patch sizes, which can be done either in parallel or sequentially, depending on whether shorter computation times or lower memory usage are prioritized. We chose sequential processing, where each patch size is handled individually, keeping the memory consumption comparable to baselines like PatchTST (see Table 1 in global response).

In this approach, memory usage is primarily influenced by the smallest patch size, not the number of patch sizes used (see Table 3 in global response). For example, the multi-patch setting in the paper (sizes {8,16,32}) requires the same maximum memory as using a single patch size of 8. Under resource constraints, adjusting the minimum patch size helps balance model performance and memory usage. We provide further analysis on the performance of different patch size combinations in the response to Question 3.

Response to Question 1

Given the significant increase in computational cost from longer token sequences, would independently inputting each patch size into the transformer layers, instead of all at once, reduce memory usage while maintaining model performance?

Yes, as mentioned in our response to Weakness 2, independently inputting each patch size into the transformer layers can indeed reduce memory usage without degrading model performance.

Response to Question 2

With multiple patch sizes used in the study, how are the RoPE applied? Are they applied independently to tokens from each patch size, or collectively to all tokens from different patch sizes?

RoPE is applied independently to each patch size, and tokens from different patch sizes do not interact within the Transformer layer.

Response to Question 3

Ablation studies indicate that incorporating smaller patch sizes seems to benefit model performance substantially. Would using a patch size of 1 worsen the model performance? Would adding "1_" to the existing best patch size setting "8_16_32" improve performance?

Thank you for your questions. We would like to clarify that smaller patch sizes do not always improve model performance. In Appendix C.3 of the submitted manuscript, we discuss how different patch size combinations impact performance across various training horizons. For instance, with a training horizon of 720, a smaller patch size can sometimes result in poorer performance.

In Figure 4 of the global response, we included the case with a patch size of 1 for a more comprehensive comparison. The results indicate that using a patch size of 1, or adding it to a multi-patch combination, does not consistently lead to better or worse performance. The optimal patch size varies by dataset and forecasting horizon. Overall, the 8_16_32 combination provides a good balance between performance and memory consumption for varied-horizon forecasting.

We sincerely appreciate your suggestions and feedback. We will revise the paper to clarify the workings of Tunable RoPE and multi-patch assembly, and we will include a more comprehensive comparative and computational analysis to provide a holistic view of the proposed model.

Thanks for your in-depth reviews, insightful questions, and constructive suggestions. We hope the following responses can help to address all your questions and concerns.

Response to Weakness 1

We will expand our literature review to provide a more comprehensive discussion of recent time series foundational models. We briefly summarize these models in the table below. Most of these efforts employ standard architecture designs, position encodings, and patching approaches. However, they often lack in-depth investigation into the challenges of generating robust forecasts across varied horizons. Our work directly addresses this gap by enhancing model design for varied-horizon robustness. Detailed discussions will be incorporated into the revision.

Response to Weakness 2

We acknowledge the importance of including a more comprehensive baseline. Due to the word limitation, we have provided the averaged performance of these baselines across all datasets in the table below for reference. The revised manuscript will include detailed results and analysis to provide a more thorough evaluation.

Response to Weakness 3

Thank you for raising this important point. ElasTST introduces minimal additional computation compared to other non-autoregressive Transformer-based models, as shown in Table 1 in the global response. However, we acknowledge the need for further scalability improvements. To address this, we are exploring solutions such as dimensionality reduction, quantization, and memory-efficient attention mechanisms to reduce computational overhead for large datasets.

Response to Weakness 4

Thank you for your observation regarding the evaluation metrics. As noted in Appendix B.3, some existing models apply z-score standardization during data preprocessing and compute metrics before reversing the standardization, which can complicate direct comparisons when different scaler are used. We chose NMAE (Normalized Deviation in the M4 and M5 literature) as our primary evaluation metric because it is scale-insensitive and widely accepted in recent studies [1].

Additionally, we have also calculated MAE and MSE both before and after de-standardization. These results will be included in the Appendix in future revisions.

[1] Oreshkin, B. N., Carpov, D., Chapados, N., & Bengio, Y. (2020). N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. ICLR.

Response to Weakness 5

While our method improves varied-horizon forecasting, as shown in Figure 3, the training horizon is a tunable hyper-parameter, and the optimal value varies across datasets. An inappropriate choice, like 96, can significantly degrade performance. Besides, the selected horizon of 336 used in the main results is not always optimal, suggesting that tailoring the training horizon for each dataset can further enhance the model’s effectiveness.

Response to Question 1

In Table 2 of the global response, we compare the memory usage of ElasTST with different positional encodings. The results show that RoPE adds a negligible number of parameters compared to vanilla absolute position encoding. Although applying the rotation matrix introduces slightly higher memory usage, it remains acceptable.

Response to Question 2

Thank you for your insightful observation. To understand the performance similarities, we analyzed the trend and seasonality of these datasets [1]. We find that the similarities between ElasTST and PatchTST could be attributed to varying levels of seasonality. Specifically, ETTm1 and ETTm2 have minor seasonality, accompanying similar performance, while ETTh1 and ETTh2 exhibit stronger seasonal patterns, where ElasTST shows clear advantages.

Without the additional designs for robust varied-horizon forecasting, ElasTST essentially functions as a variable-length version of PatchTST. Therefore, the differences in performance could reflect the datasets’ sensitivity to our specific architecture designs.

[1] Wang, X., Smith, K. A., & Hyndman, R. J. (2006). Characteristic-based clustering for time series data. Data Mining and Knowledge Discovery, 13(3), 335–364.

Response to Question 3

In Figure 2 of the global response, we compare the performance gains of each model design across different points within the forecasting window. The benefits of structured attention masks are consistent throughout the entire horizon, while the advantages of tunable RoPE and multi-scale patch assembly become more pronounced when dealing with unseen horizons. Notably, tunable RoPE significantly enhances the model’s extrapolation capability. Due to the word limitation, we will include a more detailed analysis in the revised version.