TimelyGPT: Recurrent Convolutional Transformer for Long Time-series Representation

Q5: Hyperparameter sensitivity analysis could provide more insights into optimal configurations.

R5: Our optimization of model parameters is guided by the scaling law discussed in Section 4.1. As illustrated in Fig. 2b, the largest TimelyGPT with 18M parameters continues to improve as data size grows. Therefore, our choice of the overall number of parameters such as number of layers and embedding size is mainly limited to our computational resource. For certain hyperparameters, like the decay rate, we follow the specifications from the RetNet paper. Additionally, we have conducted an analysis of convolution operators, the results of which are shown in Table 12.

Q6: Evaluating true few-shot generalization with limited fine-tuning data could better demonstrate transfer learning benefits.

R6: In the revised version, the Section 5 (Conclusion) discussed that the pre-trained TimelyGPT exhibits limited generalization ability across different types of biosignals. For example, model pre-trained on ECG (SleepEDF dataset) does not generalize well on PPG signal. We acknowledge this limitation in Section 5 and identify it as an avenue for future exploration.

Q7: The focus is on a decoder-only architecture. Encoder-decoder and encoder-only architectures could also be analyzed.

R7: Existing time-series transformer mainly focus on on encoder-only (like PatchTST) and encoder-decoder (like Informer, Autoformer, Fedformer) architectures, which have been investigated in experiments. Our paper focuses on decoder-only transformer since extrapolable embedding is constrained to unidirectional attention [1]. Attempts to adapt to bidirectional attention have shown reduced extrapolation performance [1]. Our future plans involve developing a novel design to accommodate extrapolable embeddings to bidirectional attention and encoder architecture.

[ref 1] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

Q8: Societal impacts regarding potential misuse of forecasting need to be considered.

R8: We revise manuscript by incorporating societal impacts in Section. 5 (Conclusion). The misuse of pre-trained model in forecasting can lead to serious privacy risks, such as unauthorized prediction of patient health status and breaches of privacy.

Q9: While this paper explores an intriguing aspect of time-series analysis—specifically, modeling irregular time-series—it is not without its shortcomings, particularly in the design of experiments meant to objectively situate this work within its field. Please refer to the 'weaknesses' section for a detailed list of concerns raised by the reviewer. The reviewer would be pleased to revise their score upwards provided these issues are adequately addressed.

R9: In the revised version, we provide a discussion of key shortcomings in Section 5 (Conclusion). It covers domain adaptation, the constraint of xPOS to unidirectional attention, and social concerns regarding privacy leak.

Thanks for your careful review and constructive suggestions. Please find our response to your questions and concerns.

Q1: Analysis of the computational overhead and scaling behavior compared to other transformers could be expanded.

R1: Our study primarily examines computational complexities, emphasizing TimelyGPT's linear training complexity for long sequence length. As established in Section 2.5, the quadratic term becomes dominant when $n > 6hd$ (where $n$ is sequence length, $h$ is head number, $d$ is embedding size per head). Leveraging chunk-wise retention mechanism, TimelyGPT segments each long sequence into multiple small chunks, where $n < 6hd$ in each chunk. Therefore the time complexity is mainly on the embedding size, i.e., the second term in $O(12n h^2 d^2 + 2n^2hd)\equiv O(12n h^2 d^2)$, which is linear to the sequence length. Detailed analysis is provided in Section 2.5 and Appendix B.3.

In practice, we analyzed the scaling law on the Sleep-EDF dataset (i.e., MAE as a function of dataset size; Fig. 2), which is the biggest time-series dataset publicly available. We plan to expand this analysis to larger datasets in further studies.

Q2: Only one large-scale time series dataset (Sleep-EDF) was used for pre-training experiments. More diverse pre-training data could help.

R2: Since Sleep-EDF is the biggest time-series dataset publicly available, we are limited in finding datasets of comparable size. Nonetheless, we also explore pre-training on other datasets like PTB-XL. This dataset has been examined in other pre-training studies (CRT and TF-C), offering a basis for comparative analysis for follow-up works.

Q3: For the classification tasks, comparison to a broader range of datasets and architectures like RNNs would be useful. For the forecasting, what's the performance for the ETT, weather and electricity dataset used in the PatchTST paper.

R3: To address the concern about a broader range of architectures, our revised manuscript now includes comparisons with LSTM (same layer and dim as TimelyGPT) and two novel approaches (TS2Vec and TimesNet) in classification and regression tasks. The table presented below shows TimelyGPT's superior performance, particularly over LSTM. This effectiveness can be attributed to TimelyGPT's recurrent attention mechanism, effectively combining the strengths of the sequential modeling capabilities of RNNs and the scalability of transformers.

In Section 2.1, we point out that commonly used forecasting datasets are small data (up to 69K). As demonstrated in Section 4.1, time-series transformers follow scaling law and underperform on small data. Consequently, we focus on large-scale pre-training (up to 1.2B) instead.

Q4: Theoretical analysis of frameworks to relate the different components to time series properties is limited.

R4: We consider the following time-series properties: (1) trend patterns; (2) sequential nature; (3) multi-scale signals (i.e., patterns revealed at sub-sequence of different lengths); (4) global and local features.

Our TimelyGPT framework addresses the above four properties with the three key components, namely xPOS, retention, and convolution. For xPOS, we examine its potential for time-series extrapolation, building on its known capability to model long-term dependencies in text data [1]. We found that xPOS captures long-term trend patterns in time-series, facilitating extrapolation over long timesteps. Additionally, RNNs are recognized for their effectiveness in sequential data modeling, aligning well with the inherent sequential nature of time-series data [2]. Convolution modules are capable of extracting both local features [3] and **multi-scale features ** [4] from time-series data, representing various characteristics of time-series. The integration of attention mechanisms and convolution modules effectively capture both global and local interactions in time-series data [5].

[ref 1] A Length-Extrapolatable Transformer

[ref 2] Efficiently modeling long sequences with structured state spaces

[ref 3] Convolutional Networks for Images, Speech, and Time Series

[ref 4] Omni-scale CNNs: a simple and effective kernel size configuration for time series classification

[ref 5] Lite Transformer with Long-Short Range Attention

Thank you for spending your time and effort to review our paper. We greatly appreciate your valuable suggestions and have made revisions accordingly.

We understand that you initially had reservations about our work, and we would like to ensure that all your concerns have been adequately addressed. If you believe that your concerns have been resolved, we would be grateful if you could consider revising your rating. We highly value your feedback and sincerely appreciate your consideration in this matter.

We sincerely appreciate your thorough review and valuable feedback. We provide our responses to your insightful questions and concerns.

W1: Most of the empirical insights (section 2) for using Transformers for time series presented by this paper are kind of repeating the findings of previous works [1][2].

[1] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are transformers effective for time series forecasting?, 2022.

[2] Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation, 2022.

R1: Our research diverges from the conclusion of paper [1]. Rather than attributing transformer underperformance solely to model design [1], we identify overfitting as a critical issue in Section 2.1 and Section 4.1. Additionally, paper [1] believes that inherent flaws in attention and position embedding lead to underperformance. In our study, we investigate effective designs of attention and position embedding for time-series modeling in Section 2.2 and Section 2.3, respectively. Inspired by the findings of paper [2], we propose the extrapolation technique in time-series transformers for the first time.

[ref 1] Are transformers effective for time series forecasting?

[ref 2] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

W2: In the ablation studies, this paper only presents the results of classification and the results for forecasting tasks are not provided.

R2: In the revised version, we provide a forecasting-specific ablation studies in Table.9. The table reveals that vanilla GPT-2 lacks extrapolation ability; while GPT-2 with RoPE achieves limited extrapolation over shorter distances. Section 4.6 offers visualized ablation results, indicating xPOS effectively captures long-term trends and achieves superior extrapolation performance.

W3: The model architecture is a little bit incremental. xPOS embeddings and retention module are simple extensions of previous works.

R3: Although xPOS and extrapolation are applied in NLP domain, our paper is the first work to thoroughly investigate extrapolable embedding in time-series forecasting. Note that NLP and time-series domain have fundamental difference, such as trend and periodic patterns in time-series. We explore the underlying mechanism that enables xPOS's extrapolation in time-series domain, which is not studied in previous works. Additionally, TimelyGPT is the first work to introduce RNN-based transformer in time-series domain. We expect a research shifts toward unexplored areas like extrapolable embedding and RNN-based transformers in time-series domains.

Q1: How do you calculate the dataset size?

R1: The revised manuscript explains the dataset size in terms of timesteps clearly.

Q2: How does TimelyGPT perform if pre-training is removed?

R2: The revised version includes a quantitative ablation study on pre-training in Table. 11. The following table shows that TimelyGPT with pre-training obtains a notable increase of 3.89% in biosignal classification (SleepEDF), while the improvement (0.83%) is less evident in irregularly-sampled time series (PopHR).

Q3: For ultra-long-term forecasting (section 4.2),

3.a. How many parameters do baselines have?

R3.a: For ultra-long-term forecasting, both TimelyGPT and all transformer baselines were configured with 18 million parameters. This number is in line with the scaling law outlined in Section 4.1. Model parameters and architectures for other tasks are provided in Appendix C.5.

3.b. Are the baselines pre-trained? How do you pre-train baselines?

R3.b: For fair comparison, all transformer baselines (PatchTST, Informer, Autoformer, Fedformer, TST, CRT) are pre-trained.

The pre-training methods of TST and CRT are detailed in the Introduction section. PatchTST employs TST's masking pre-training approach, with 40% of the patches masked. Other transformers also use a similar masking strategy, masking 40% of timesteps following TST's methodology. The details of pre-training and fine-tuning can be found in Appendix. C.4.

We are grateful for your constructive comments. Below we address each question and concern.

Q1: Authors emphasise the focus on pre-training. However, I could not find results ablating the benefits of pre-training and whether TimelyGPT is particularly well suited for it. Some experimental settings are odd where models are pre-trained on different datasets for classification vs regression. Task specific pre-training deviates from the foundational model paradigm, why was that done?

R1: The revised version includes a quantitative ablation study on pre-training in Table. 11. The following table shows that TimelyGPT with pre-training obtains a notable increase of 3.89% in biosignal classification (SleepEDF), while the improvement (0.83%) is less evident in irregularly-sampled time series (PopHR).

Regarding the question about ``task specific pre-training", we pre-trained TimelyGPT on the same SleepEDF dataset for both forecasting (Section 4.2) and classification (Section 4.3) downstream tasks. However, the pre-trained model exhibits limited generalization ability across different types of biosignals. For example, model pre-trained on ECG (SleepEDF dataset) does not generalize well on PPG signal. We acknowledge this limitation in Section 5 and identify it as an avenue for future exploration.

Q2: Figure 3.a results look almost too good to be true, 720 vs 6K prediction window size is a huge difference and one would expect at least some performance decay. At what prediction window size do you start seeing performance decay and what in TimelyGPT makes it so well suited for such long range prediction?

R2: The ultra-long-term forecasting in Fig 3 shows our most exciting results. The inference length of 8K (2K loop-up window as prompt plus 6K forecasting window) is the maximum length without observing performance decay. This aligns with the performance of xPOS-based transformer in NLP domain [1]. In the revised version, our visualization results (Section 4.6) explores the underlying mechanism behind xPOS's extrapolation, which is driven by its modeling of trend patterns in time-series.

For comparison, existing time-series transformers (Informer, Autoformer, Fedformer, PatchTST) with absolute position embedding do not exhibit extrapolation ability [2]. In visualization results (Section 4.6), these transformer baselines without extrapolation capabilities generate forests flat trajectories that largely deviate from the groundtruth, which aligns with our analysis in Section 2.4 and a recent study [3].

[ref 1] A Length-Extrapolatable Transformer

[ref 2] Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation

[ref 3] Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping.

Q3: Some components like the temporal convolution module have very specific sets of operators. How sensitive are the results to these choices? I think the paper would also benefit from clearly highlighting novel components vs novel application of previous work.

R3: In the revised version, Table 12 compares various convolutional operator configurations on the downstream classification for SleepEDF dataset. Our Temporal Convolution module outperforms two commonly used operators designed for modeling global-local interaction in transformers.

We agree that while the proposed architecture of our TimelyGPT is novel, there is a need to explicitly highlight the level of our contribution, be it at the level of the sum of those parts or at the level of the parts themselves. Following your suggestions, we provide an explicit distinction between novel applications (i.e., applying xPOS to time-series data and retention on continuous time-series data) and novel components (i.e., extension of retention for irregular time-series and the temporal convolution module).

[ref 1] Conformer: Convolution-augmented Transformer for Speech Recognition

[ref 2] Lite Transformer with Long-Short Range Attention

Thank you for spending your time and effort to review our paper. We greatly appreciate your valuable suggestions and have made revisions accordingly.

We are encouraged by your positive assessment for our work, and we would like to ensure that all your concerns have been adequately addressed. If you believe that your concerns have been resolved, we would be grateful if you could consider revising your rating. We highly value your feedback and sincerely appreciate your consideration in this matter.

We appreciate the reviewer’s comments. We update our paper and address your concerns here:

Q1: As mentioned in Section 1, it is suggested that ``challenges observed in small-scale time series might stem from overfitting limited data rather than inherent flaws in transformers". This statement is a bit confusing as the title suggests that the model can handle long time series. However, it seems that the goal is to address the overfitting to limited data, which implies that only short-time series are available. This appears to be contradictory. Please clarify what problems in time series applications are mainly addressed here。

R1: The focus of our research is on leveraging large-scale datasets (1.2B timesteps) for transformer pre-training, enabling long-sequence time-series representation (4K timesteps). While large time-series datasets have been available, previous time-series transformers have not effectively utilized them, often focusing on modeling small data (like ETT with 69K timesteps) and short-sequence (96 timesteps).

The rationale for our research is twofold. First, in Section 2.1, we point out that the existing transformer-based frameworks are overparameterized for small datasets. Secondly, in Section 4.1, we show that these transformer-based models follow the scaling law and do not perform well when the data are small. To address these limitations, we pre-train TimelyGPT on the large Sleep-EDF dataset, addressing the overfitting in small data scenarios. With large datasets, TimelyGPT can learn long-sequence representation in time-series domain for the first time.

To improve the clarity, we revised that sentence as follows: We argue that the seemingly inadequacy of the current transformer-based models in modeling time-series data is due to their inability to model long and big time-series data. Once these challenges are resolved, we would observe a typical scaling law comparable to the one observed in the NLP applications.

Q2: Table 1 can be further improved. For example, it is unclear what is meant by `data size.' Is it the number of time points or the number of windows? And what are the numbers of the proposed model?

R2: In the revised Table 1, we now clearly specify ``Dataset size (Timesteps)" (which is analogous to the ``number of tokens" in NLP applications) to clarify data size.

Detailed information about TimelyGPT model parameters and architectures is provided in Appendix. C.5. The following table provides a summary:

Q3: The writing could be further improved. The formulas in section 3.2 are difficult to follow, and it would be helpful to introduce each formula one by one in a logical manner.

R3: Following your recommendation, we have revised Section 3.2 by presenting equation one by one with improved descriptions. Recognizing that Retention is key contribution of RetNet paper, the main text only includes essential equations and descriptions for its understanding. Comprehensive descriptions are in Appendix B.4.

Q4: The proposed TimelyGPT refers to the Timely Generative Pre-trained Transformer. However, details of the pretraining and fine-tuning processes are missing. Specifically, what datasets and how many epochs were used for pretraining? What fine-tuning strategies were used for downstream tasks?

R4: In the revised version, we provide details about pre-training and fine-tuning in Appendix. C.4. In the pre-training, TimelyGPT performs next-token prediction using cross-entropy loss. Given a sequence starts with a [SOS] token, each token's output embedding is used to predict next token using right shift. Following PatchTST's finding that end-to-end fine-tuning yields best results [1], we adopt it on the pre-trained TimelyGPT. The pre-training and fine-tuning of TimelyGPT involve 20 and 5 epochs, respectively, with early stopping.

We already provided the details of datasets in Appendix C.2.

[ref 1] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

Q5: There are many related works (e.g., TS2Vec, TimesNet) on time series self-supervised representation learning, but there is a lack of systematic investigation, discussion, and comparison.

R5: In the revised version, we discuss TS2Vec and TimesNet in section of Related Works and evaluate on various downstream tasks. The following tables shows that TimelyGPT surpasses both TS2Vec and TimesNet among these tasks.

Q6: Regarding the prediction results in Figure 3, it should be noted that the dataset is not commonly used for forecasting. Additionally, a too-long prediction horizon may not make sense as real-world prediction scenarios are usually dynamic and unpredictable.

R6: In Section 2.1, we point out that commonly used forecasting datasets are small data (up to 69K). As demonstrated in Section 4.1, time-series transformers follow scaling law and underperform on small data. Consequently, we focus on large-scale pre-training (up to 1.2B) instead.

Ultra-long-term forecasting is crucial for healthcare applications such as patient health monitoring. Biosignals like Sleep-EDF are sampled at a high rate of 100 HZ. Forecasting 6000 timesteps helps doctors effectively monitor patients' health status in the next 1 minute. Short-term forecasting models fall short in real-world healthcare contexts.

Q7: It is recommended to summarize the prediction results in a table rather than an overly simplified figure. This makes it easier for other works to follow.

R7: In the revised version, we summarize the forecasting results in Table 9.

Q8: Visualization results are encouraged to be included.

R8: In the revised version, we provide visualization result in Section 4.6. Specifically, we show TimelyGPT effectively extrapolates to long-sequences. PatchTST and GPT-2 with absolute embedding rely on linear mapping for forecasting, which aligns with our analysis in Section 2.4 and a recent study [1]. Our visualization results indicate benefits of integrating extrapolable embeddings into time-series transformers for future research.

[ref 1] Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping

Q9: The use of a relative position embedding is not a new idea and it has been studied in different communities, e.g., transformer variants and time series applications. For example,

[ref] STTRE: A Spatio-Temporal Transformer with Relative Embeddings for Multivariate Time Series Forecasting

[ref] Improve transformer models with better relative position embeddings

R9: In the revised version, we have referenced STTRE in Section 2.3. However, the STTRE [1] was available online on 09/30/2023. According to the ICLR Reviewer Guide, there's no requirement to compare with papers published "on or after May 28, 2023". Moreover, STTRE is spatio-temporal transformer and another paper [2] addresses NLP task only. Although relative position embeddings are studied in various domains, TimelyGPT is the first work to thoroughly investigates its advantages in time-series applications. TimelyGPT is also the first work to explore the underlying mechanism of extrapolable embedding in time-series modeling. It opens new avenues for future research to transition from absolute to relative position embeddings in time-series domain.

[ref 1] STTRE: A Spatio-Temporal Transformer with Relative Embeddings for Multivariate Time Series Forecasting

[ref 2] Improve transformer models with better relative position embeddings

Q10: If the proposed recurrent attention can handle irregular time series well, it would be beneficial to compare it with popular irregular time series methods.

[ref] Multi-time attention networks for irregularly sampled time series

R10: In the revised version, we compared TimelyGPT with the recommended baseline (mTAND) [1], along with three widely-recognized baselines (GRU-D [2], SeFT [3], RAINDROP [4]) for irreguarly-sampled time-series. The full results can be found in Table 10.

The following table shows that TimelyGPT exhibits superior performance (average AUPRC of 57.6%), surpassing the SOTA algorithm mTAND (56.4%). It also exceeds other baselines, including GRU-D (55.8%), SeFT (48.7%) and RAINDROP (54.6%). TimelyGPT secures the top rank (1.50) among baseline models.

[ref 1] Multi-time attention networks for irregularly sampled time series.

[ref 2] Recurrent neural networks for multivariate time series with missing values

[ref 3] Set Functions for Time Series

[ref 4] Graph-Guided Network for Irregularly Sampled Multivariate Time Series

Q11: The source code is incomplete.

R11: We already provided the source code of TimelyGPT for reproduction purpose. The organized code will be available on Github after acceptance.

Thank you for spending your time and effort to review our paper. We greatly appreciate your valuable suggestions and have made revisions accordingly.

We understand that you initially had reservations about our work, and we would like to ensure that all your concerns have been adequately addressed. If you believe that your concerns have been resolved, we would be grateful if you could consider revising your rating. We highly value your feedback and sincerely appreciate your consideration in this matter.

I appreciate the authors' detailed responses and I'm satisfied with the supplementary experiments on irregular time series. However, Their answers just partially addressed my concerns. I have decided to upgrade my rating to 5.

There are still some aspects of the authors' responses that are not satisfactory. Specifically:

• The writing regarding the formulation of the methods still needs further improvement. For instance, there is a lack of shape information for many tensor variables.
• The discussion on self-supervised representation learning is insufficient. In the revised version, the modification "Additionally, self-supervised representation learning techniques in time series, such as TS2Vec and TimesNet, offer valuable representation learning capabilities for forecasting tasks (Yue et al., 2022; Wu et al., 2023)" appears to be oversimplified.
• Including more commonly used datasets, such as electricity and traffic, may help demonstrate the model's performance on general datasets. However, this suggestion applies only if the authors have not explicitly limited the scope of their research to healthcare contexts in the title.
• Table 6 is lacking information on the training, validation, and test sets. Besides, it would be better to add links to these datasets if they are publicly accessible.
• The prediction results in Figure 9 are remarkably good. The time series trends seem to be very challenging. I would like to see the reasons why the model can model such difficult trends. It would be beneficial to visualize the historical data as well to see if the model relies on similar historical knowledge to accurately predict such challenging trends.
• Regarding predictive transformers like Autoformer, it is unclear how they can be applied to classification tasks.
• Another crucial issue is the out-of-memory error often encountered when using Autoformer for long sequence processing. How was this addressed in your long sequence prediction experiments?
• Transformers often face high complexity and struggle to handle long historical sequences, thereby limiting their performance. In NLP, there are various improved variants of attention mechanisms that can be applied to very long sequential data. I did not see any specific comparisons of these variants in the context of long-range prediction tasks.

Due to time constraints, I would like to have a brief discussion with the authors regarding a specific question:

• There is a perspective here that overparameterized deep learning networks do not overfit [1]. What is your viewpoint on this?

[1] Zhou Z H. Why over-parameterization of deep neural networks does not overfit?. Science China Information Sciences, 2021, 64: 1-3.

Thanks for your constructive feedback and insightful suggestion. Due to constrained time, we would like to address your main concerns as below.

Q1: The writing regarding the formulation of the methods still needs further improvement. For instance, there is a lack of shape information for many tensor variables.

R1: This suggestion provides valuable assistance in improving the readability for our work. In response to your suggestion, we have revised our paper and clearly stated the shape of every variable in Section 3.1 and Section 3.2.

Q2: The discussion on self-supervised representation learning is insufficient. In the revised version, the modification "Additionally, self-supervised representation learning techniques in time series, such as TS2Vec and TimesNet, offer valuable representation learning capabilities for forecasting tasks (Yue et al., 2022; Wu et al., 2023)" appears to be oversimplified.

R2: In the revised version, we include an analysis for the challenge faced by TimesNet. Given TimesNet's dependence on frequency-domain information and Fourier transform, the masking in time-series leads to the distribution shift [1], adversely impacting the performance.

[ref 1] Self-Supervised Time Series Representation Learning via Cross Reconstruction Transformer

Q3: Including more commonly used datasets, such as electricity and traffic, may help demonstrate the model's performance on general datasets. However, this suggestion applies only if the authors have not explicitly limited the scope of their research to healthcare contexts in the title.

R3: In our research, we mainly focus on time-series in healthcare contexts. We highlight at the beginning of our paper: ``Time-series data mining holds significant importance in healthcare, given its potential to trace patient health trajectories and predict medical outcomes (Ma et al., 2023b; Eldele et al., 2021; Fawaz et al., 2019). In the field of healthcare, there are two primary categories: continuous and irregularly-sampled time-series data".

Q4: Table 6 is lacking information on the training, validation, and test sets. Besides, it would be better to add links to these datasets if they are publicly accessible.

R4: Table 6 in Appendix C.1 (Dataset Description) organizes the information regarding datasets, where all biosignal datasets we cited in Appendix C.1 are publicly available. We have included the pre-processing details in Appendix C.2 (Data Pre-processing). We summarize the train/validation/test sets as follows:

Q5: The prediction results in Figure 9 are remarkably good. The time series trends seem to be very challenging. I would like to see the reasons why the model can model such difficult trends. It would be beneficial to visualize the historical data as well to see if the model relies on similar historical knowledge to accurately predict such challenging trends.

R5: The effectiveness of trend modeling in Fig.8 is largely attributed to the xPOS embedding in our TimelyGPT framework. This position embedding utilizes exponential decay $\gamma$ to attenuate the influence of distant tokens, aiding in capturing long-term dependency and extrapolation ability in NLP applications [1]. In the time-series analysis, we found that the exponential decay can effectively capture long-term trend patterns (Fig. 9), which aligns with Fig. 2 in xPOS paper [1].

In the revised version, we include visualization results of a 8K-timestep history, a 2k look-up window, and a 6K forecasting window for a sleep stage transition (Fig. 10). For the temperature signal, we observe distinct historical patterns that differ from the forecasting window, highlighting the temperature changes over 16K timesteps (160 seconds). Leverage pre-training power, Section 4.6 found that ``the small bump in the prompt right before the 1000-th timestep is a typical indicator for temperature drop". In contrast, the periodic EEG signal shows consistent periodic patterns throughout the 16K timesteps.

[ref 1] A Length-Extrapolatable Transformer

Q6: Regarding predictive transformers like Autoformer, it is unclear how they can be applied to classification tasks.

R6: Transformers designed for forecasting (like Autoformer, Fedformer, and PatchTST) typically do not incorporate classification tasks directly in implementation. Nonetheless, models like TST and CRT perform classification tasks by using a linear layer with averaging last layer's embeddings.

Q7: Another crucial issue is the out-of-memory error often encountered when using Autoformer for long sequence processing. How was this addressed in your long sequence prediction experiments?

R7: To address memory limitations often occurred in the training of transformers, we conducted our experiments on a A-100 GPU with 40GB memory. Additionally, we leveraged multiple strategies to optimize memory usage in our submitted code: (1) gradient checkpointing; (2) gradient accumulation; (3) PyTorch's garbage collection.

Q8: Transformers often face high complexity and struggle to handle long historical sequences, thereby limiting their performance. In NLP, there are various improved variants of attention mechanisms that can be applied to very long sequential data. I did not see any specific comparisons of these variants in the context of long-range prediction tasks.

R8: While comparing various long-context attention modules would be valuable, the Retnet paper has already conducted such comparisons, including RWKV, H3, Heyna, and Linear Transformer. Applying all these models specifically to time-series data would be an extensive work beyond our current scope.

Q9: Due to time constraints, I would like to have a brief discussion with the authors regarding a specific question:

There is a perspective here that overparameterized deep learning networks do not overfit [1]. What is your viewpoint on this?

[1] Zhou Z H. Why over-parameterization of deep neural networks does not overfit?. Science China Information Sciences, 2021, 64: 1-3.

R9: This paper [1] provides a valuable perspective on overparameterization in DNN, specifically in the context of conventional machine learning theory. By viewing a MLP as a combination of large-size feature space transformation and small-size classifier construction, it is possible for the number of parameters to surpass the sample size without overfitting.

However, in the context of Transformer, it typically follows the Neural Scaling Law [2]. The study [2] observed that ``For a smaller fixed D (dataset size), performance stops improving as $N$ (model size) increases and the model begins to overfit". It shows that both model parameters and dataset size need to be scaled up simultaneously to improve model performance. Our finding in Fig. 2, along with studies in other domains [3], corroborate this scaling law. Thus, while the paper [1] provides valuable insights, it may not fully align with the scaling behaviors exhibited by Transformer models.

[ref 1] Why over-parameterization of deep neural networks does not overfit?

[ref 2] Scaling Laws for Neural Language Models

[ref 3] Scaling Vision Transformers