AutoTimes: Autoregressive Time Series Forecasters via Large Language Models

Response to Reviewer k2uw

Many thanks to Reviewer k2uw for providing a valuable review.

Q1: Reclarify the contributions of the proposed method.

The reviewer mentioned that "the paper exploits the autoregressive property of the LLMs". We agree with this argument but also would like to highlight detailed contributions of our work:

• We delve for the first time into the autoregression in LLM4TS methods. It addresses the raising concerns about the effectiveness of prevalent non-autoregressive LLM4TS methods (refer to $\underline{\text{Q1 of Reviewer 1PWH}}$).
• We proposed the one-for-all benchmark that breaks the status quo of respective training on specific lookback-forecast length (appreciated by $\underline{\text{Reviewer EAH7}}$), which is an essential step toward foundation models.
• Our method achieves state-of-the-art forecasting performance and requires significantly fewer tunable parameters among advanced LLM4TS methods. The method's efficiency is acknowledged by all other reviewers.
• We present the concept of in-context forecasting for the first time, where time series prompts and prompt engineering are closely researched (refer to $\underline{\text{Q3 of Reviewer acJd}}$).
• Beyond adopting LLMs on end-to-end forecasting, we facilitate the full capabilities of LLMs for time series, such as iterative multistep generation, zero-shot generalization, and scaling behavior.

Q2: How much improvement is brought by autoregressively generating time series?

To address your concern, we provide a comprehensive ablation study: AutoTimes (FlattenHead) replaces the original segment-wise projection layer by (Flatten + linear head) of PatchTST, the prevalent module in non-autoregressive models. Here are the results:

As shown in the above tables, the performance of non-autoregressively generation is consistently worse than original AutoTimes. In addition to empirical results, we'd like to provide an analysis of the two approaches:

• One model fits all lengths: While most deep forecasters have to be trained and applied to specific length settings, autoregressive models are more feasible in variable-length scenarios. We provide the comparison as follows:

• Fewer parameters for training: Supposing $L$ is the lookback length, $F$ is the forecast length, $S$ is the segment (token) length and $N$ is the segment (token) number. We count the parameters for embedding and projection:

• Non-autoregression: time series segment ($S$) -> representation ($D$) -> flattened ($N\times D$)-> future time series ($F$)

• Autoregression: time series segment ($S$) -> representation ($D$) -> next time series segment ($S$)

In non-autoregressive models, all the tokens of the lookback series are flattened and projected to future time series, which leads to costly parameters of $ND\times F$. Instead, the projection of AutoTimes is independently conducted on segments, leading to fewer introduced parameters $D\times S$.

• Consistent with the utilization of LLMs: the main claim of our paper is that non-autoregressive LLM4TS leads to inconsistencies, where inherently GPT-style models are fine-tuned in the BERT-style. Instead, we suppose the token transition of LLMs is general-purpose and find it transferable as the transition of time series segments. Consequently, the powerful generation ability of LLMs can be naturally inherited.

Q3: Concern about the scaling behaviors of LLM-based forecasters.

Thank you for your feedback regarding the conclusions drawn from Table 5. It is recommended to see the scaling behavior in $\underline{\text{Figure 4 of the main text}}$, where the scaling behavior is not only observed in OPT-x models but also revealed among GPT-2, OPT, and LLaMA.

We further eliminate the influence of parameter count in trainable layers. As shown in the following table, a larger LLaMA-7B with fewer trainable parameters can still achieve better performance compared to OPT (1.3B~6.7B), demonstrating the performance stems from the scaling of LLMs, not simply the parameter of trainable layers.

Dear Reviewer k2uw,

Thanks again for your valuable and constructive review. Would you please let us know if your concerns about autoregression effectiveness and scaling behavior are resolved? Since the Author-Reviewer discussion period is ending in two days, we eagerly await your response.

Till now, we find that your rating is still "reject". In a very worried mood, we respectfully remind you that we have comprehensively evaluated the improvement of autoregression and validated the scaling effect on different LLMs categories, which should help you better assess our work. Also, we have clarified the contributions of the method and paradigm renovation, and how they differ from previous approaches.

We do hope you can consider our rebuttal and kindly let us know if our rebuttal has addressed your concern.

Sincere thanks for your dedication! We are looking forward to your reply.

Authors

Dear Reviewer k2uw,

Thank you again for your valuable and constructive review.

We kindly remind you that there is only a half day left before the Author-Reviewer discussion period ends. We find that your rating is still "reject", so we are eager to receive your response to our rebuttal. We respectfully ask if you have any further concerns. Please let us know if you have other questions about our paper. We will be more than happy to engage in more discussions and improvements to the paper.

Thank you sincerely for your dedication! We eagerly await your reply.

Authors

Response to Reviewer EAH7

Many thanks to Reviewer EAH7 for providing a thorough insightful review and recognizing our contributions.

Q1: Suggestion to improve the presentation of the paper.

Thanks for your valuable feedback regarding the color scheme and the mentioned typo. We will use a more subdued scheme and fix the typo in the revision.

Q2: Address inter-dependencies in real-world time series data.

We acknowledge that real-world multivariate time series often exhibit complex inter-dependencies that can significantly impact analysis. In terms of that, AutoTimes adopts Channel Independence like previous methods and further uses the timestamp as the position embedding to implicitly align different variates.

As you insightfully point out, it is necessary to explore the complex inter-dependencies, which is a hot topic in current deep time series models. It is also an essential problem for LLM4TS methods since the gap between natural language (1-D discrete token sequence) and time series (multi-dimensional continuous sequence) poses increasing challenges for LLMs to explicitly utilize the relationship between sequences.

We will explore several potential approaches: integrating textual descriptions of variates and employing adaptors for variate correlating. Your suggestion will guide us in refining our methodology.

Q3: Claim of missing data for foundation models and explore the scalability on larger datasets.

Thanks for your mentioned works, we are excited to see recent works advancing the development of datasets and pre-trained large models in the field of time series. We will cite them in the related work section and polish the claim.

Based on the mentioned works, we also provide an evaluation of AutoTimes on larger datasets (Time-Series Pile[1]) to address your concern about the scalability of the trainable layers:

As we delve into layer scalability, the essence of designing the embedding scheme to address heterogeneous time series is highlighted, which provides good guidance for our future research.

Q4: The effectiveness of fine-tuning and comparison with more LLM4TS methods.

We appreciate your concerns about the effectiveness of our fine-tuning approach, especially in light of several works that use LLMs for time series without fine-tuning. We intend to leverage the general token transition of LLMs while tailoring it to the specific characteristics of the dataset, which is achieved by freezing the LLM backbone and training new embedding layers only.

We provided detailed code and scripts in the $\underline{\text{supplementary material}}$ to ensure all the results are reproducible. Further, we compare the mentioned TEST[2], and AutoTimes achieves better performance on the majority of datasets.

Q5: How does AutoTimes handle missing data or irregular time series intervals?

Thank you for your insightful question. At this stage, AutoTimes does not specifically address missing data or irregular intervals. It is the same with current works focusing on regular forecasting scenarios where time series are complete and consistently sampled.

We acknowledge that handling missing values and irregular intervals is a critical aspect of time series analysis, and we will add this as a limitation and conduct evaluations on well-acknowledged datasets in the future.

According to your suggestions, we will also consider moving the limitations section to a more prominent position within the main body of the paper to ensure that readers can easily access and engage with this critical information.

Q6: How does AutoTime embed and feed time series and texts?

The claim: "our proposed method regards time series itself as the instructive prompt..." refers to the following:

• As depicted in $\underline{\text{Figure 1(b)}}$, previous LLM4TS methods feed (language tokens | lookback time series) to handle multimodal input.
• As depicted in $\underline{\text{Figure 3}}$, AutoTimes feed (time series prompt | lookback time series) to enable in-context forecasting, where time series is self-prompted.
• As shown in $\underline{\text{Equation 5}}$, AutoTimes uses textual timestamps as position embeddings and adds them with the corresponding series embedding of each token.

Therefore, AutoTimes presents prompting time series in two directions. Horizontally, AutoTimes append the time series prompt at the front of the lookback series and regards it as the task demonstration. Vertically (series embedding + timestamp embedding), merging token-wise embeddings can use timestamps in natural language and aligns multiple variates.

[1] Goswami et al. Moment: A Family of Open Time-Series Foundation Models.

[2] Sun et al. TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Series.

Response to Reviewer 1PWH

Many thanks to Reviewer 1PWH for providing a detailed and insightful review.

Q1: Whether AutoTimes truly leverages the capabilities of the pre-trained LLMs.

We noticed that the recent work[1] has raised questions about non-autoregressive LLM4TS methods. It is also the main claim of our paper that the inconsistent model structure and generative approach will cause insufficient utilization of LLMs for forecasting. We thoroughly conduct all types of ablations[1] (Random Init is the suggested ablation by the reviewer):

The above results highlight that the autoregression way of AutoTimes can truly utilize the LLM. The core difference: instead of regarding LLMs as representation extractors in a BERT-style, we figure out the mechanism of LLM4TS that the general-purpose token transition is transferable among time series and natural language, such that the generation ability of LLMs can be fully revitalized.

Q2: The description of multimodal in Table 1 seems to be overselling.

Thanks for your suggestion. In the initial version, we demonstrate that AutoTimes can take advantage of textual timestamps, which are the most accessible ones in real-world applications. Considering the scope of multimodal models, we will remove the point in Table 1 unless being evaluated on well-acknowledged multimodal datasets.

Q3: The use cases for in-context forecasting and how in-context forecasting gets the promotion.

The value of the proposed in-context forecasting is to extend the input context of time series forecasting beyond a continuous lookback window. As the reviewer mentioned, $\underline{\text{Table 16}}$ shows that extending the lookback window (P.2) or using trivial prompts (P.3) respectively excel at different subsets but the overall difference is small.

Since the essence of prompts is to incorporate useful domain-specific knowledge, here is one use case of in-context forecasting: Considering predicting the weather of one day, one approach is to extend the lookback length from days to weekends. However, it can also introduce noisy information since non-stationary meteorological conditions can change with seasons. Another practical way is to consider how the weather changes on the same day in the last year (or years). Although the input is not continuous, the input context becomes more relevant based on prior knowledge about the periodicity (yearly). Therefore, in-context forecasting makes the prior knowledge incorporatable and gets the promotion.

We also provide an exploration of prompt engineering in $\underline{\text{Q3 of Reviewer acJd}}$, in which the usage of discontinuous lookback windows can indeed outperform continuous lookback windows at well-acknowledged datasets.

Q4: Ablations about the timestamp embedding.

As per your suggestion, we compare the ways of embedding timestamps in AutoTimes. Here are the results:

Results show that using timestamp embeddings from LLMs achieves better performance, which indicates better alignment with learned series embeddings in AutoTimes.

Q5: How does the method overcome error accumulation?

It is true that there is no specific design for it in AutoTimes. Actually, the performance degradation during rolling forecast not only comes from the gap between the ground truth and prediction (error accumulation) but also comes from the dropping of lookback time series (lookback cut-off).

To be more precise, AutoTimes for one-for-all scenarios mainly copes with the second issue: predicting the next token of each position to keep our LLM-based forecaster feasible on prolonged inputs, while non-autoregressive models have a fixed input length. We will rephrase the relevant statements in our paper.

Q6: Report the number of learnable parameters in Table 5 and confirm the scaling behavior of LLMs.

Thanks for your scientific rigor. We will include the following results in our revision, where a larger LLaMA-7B with fewer trainable parameters can still achieve better performance compared to OPT (1.3B~6.7B).

[1] Tan et al. Are Language Models Actually Useful for Time Series Forecasting?

[2] Woo et al. Unified Training of Universal Time Series Forecasting Transformers.

Response to Reviewer acJd

Many thanks to Reviewer acJd for providing a detailed review and recognizing our contributions.

Q1: More baseline models for evaluation.

We acknowledge the importance of the recent works and will certainly incorporate the suggested references in our revision. As per your suggestion, we include the mentioned models based on the official code to enlarge our time series forecasting baselines. Here are the results:

As shown above, MOIRIA and Chronos follow the paradigm of pre-training -> zero-shot forecasting (- indicates that the test set is included in the pre-training dataset and thus not reported). MOMENT follows pre-training -> fine-tuning on each dataset and length. AutoTimes does not involve pre-training on time series since it adopts the pre-trained LLM and fine-tunes it on each dataset, using one model for all prediction lengths.

In terms of performance, AutoTimes consistently achieves the best. Still, we also appreciate the zero-shot forecasting ability of natively trained large time series models, which provide an out-of-the-box experience that is free from training/tuning the models.

Q2: More benchmark datasets for evaluation.

We appreciate your suggested works that have contributed valuable data resources. Thus, we conduct evaluations on several datasets from [1], which come from various domains and applications.

The above results show that AutoTimes still outperforms the state-of-the-art deep models, which are good enhancements for the robustness of experiments. We will include these in the revision and make more complete evaluations of the lotsa dataset [1].

Q3: Systematic evaluations on the proposed in-context forecasting.

Thanks a lot for your scientific rigor. We adopt M3 and M4 datasets, which are consistent with the zero-shot experiment of One-Fits-All paper, to present the promotion of our in-context paradigm. As per your request, we extend the evaluation to widely recognized datasets. Details of the experiment are as follows:

By using a trained model checkpoint on a source domain (Traffic), we conduct forecasting without gradient update on target ETT datasets. We evaluate the Pred-96 performance on the last variate (OT).

• For the zero-shot scenario, the input is Length-288 lookback series.
• For in-context forecasting, the input is (Length-384 series prompt + Length-288 lookback series). Considering the dataset periodicity, the prompt is uniformly selected as the Ahead-24 series of the original lookback series.
• To eliminate the performance boost that comes from extending the input length, we also provide the results of Length-672 lookback series in the zero-shot scenario.

Moreover, we further delve into the effect of different strategies to select time series prompts:

• Ahead-Period: The prompt is uniformly selected as the Ahead-24 series of the original lookback series where 24 is one of the periods of ETT.
• Ahead-Random: The prompt is randomly selected as the previous series of the original lookback series.
• Fixed Prompt: The prompt is fixed as the first piece of the series in the variate-OT.
• Other Variate: The prompt is uniformly selected as the Ahead-24 series, but comes from other variate of ETT.

The above results demonstrate the effectiveness of using suitable time series prompts and highlight the influence of prompt engineering. Using in-period time series prompts can even outperform extending the lookback window. We also provide a detailed explanation in $\underline{\text{Q3 of Reviewer 1PWH}}$. Thus, the in-context forecasting paradigm is meaningful for the real-world application.

Q4: Whether we aim to forecast covariates or not?

In Section 3, we use the timestamps as the covariate to improve forecasting, but we do not predict the covariate. As shown in $\underline{\text{Equation 1 of the main text}}$, we are concerned about the multivariate scenario, where each time series needs to be predicted.

[1] Woo et al. Unified Training of Universal Time Series Forecasting Transformers.

Thank you for your feedback, but we respectively disagree that the discussion would correspond to almost a major revision of the paper:

1. The original baselines compared in our paper are the most up-to-date and advanced deep-learning approaches.

• For LLM4TS methods, we compared AutoTimes with the state-of-the-art models: TimeLLM (ICLR 2024, cite 157), and FPT (NeurIPS 2023 Spotlight, cite 147).

• For deep time series forecasters, we compared with the most prevalent ones, covering various architectures with state-of-the-art performance: iTransformer (ICLR 2024 Spotlight, cite 189), DLinear (AAAI Oral, cite 978), PatchTST (ICLR 2023, cite 615), TimesNet (ICLR 2023, cite 466).

2. Our method is evaluated on diverse tasks and extensively analyzed in the original submission.

• Our evaluations include long-term time series forecasting, short-term time series forecasting, zero-shot time series forecasting, and in-context time series forecasting, covering 10 datasets beyond those in previous LLM4TS methods.
• Our analysis covers method generality, scaling behavior of LLMs, method efficiency, and hyperparameter sensitivity, which are hardly explored in previous LLM4TS approaches.
• Our ablations cover almost all components in the proposed method: textual timestamp embedding (Appendix D.5), the LLM backbone ( LoRA Adaptation in Section 4.4), and autoregressive forecasting (Figure 7 and Table 9).

Even if it is indispensable to include the suggested evaluations (datasets of LoTSA, comparison with concurrent Large Time Series Models, and exploration of our proposed in-context forecasting) that have not been included in any previous works in the LLM4TS direction, these experiments can be easily added by lines in existing tables and incorporate them in the Appendix, such that the work do not need a major revision.

We believe that such adjustments can enhance the integrity of the paper without too much impact on the overall structure. We hope you will reconsider this point.

Thank you for your understanding and support.