Time-LLM: Time Series Forecasting by Reprogramming Large Language Models

We sincerely thank the reviewer for their valuable suggestions to further improve our paper. We are appreciative of the reviewer's recognition of our contributions as innovative and significant in illuminating future research across a wide range of applications. Below, we address the primary concerns raised by the reviewer.

W1. While this paper is generally well-written, there are areas that could benefit from further refinement. For instance, Fig. 5 lacks clarity and might benefit from being displayed at a larger scale to improve visibility. In patch reprogramming (Sec. 3.1), it would be beneficial to illustrate how the linear projection aligns the hidden dimensions in Fig. 2. Within the Prompt-as-Prefix discussion, reference is made to the inclusion of statistics within the prompt. Elaborating on the specific content and the calculation methods employed would enhance clarity.

• We agree with the reviewer's suggestion to highlight Fig. 5 in a larger size. However, due to space constraints in the main text, scaling up Fig. 5 poses challenges as it may compromise the presentation of other results and discussions.
• In response to the suggestion of depicting the transformation from $\mathbf{Z}^{(i)}$ to $\mathbf{O}^{(i)}$ in Fig. 2, we have slightly revised this figure in the part of "Patch Reprogram".
• To construct the statistical prompting block shown in Fig. 4, we compute the following representative statistics: (1) the minimum, maximum, and median values of the given time series; (2) the overall trend of the time series, determined by calculating the sum of differences between consecutive time steps. A sum greater than 0 indicates an upward trend, while a lesser sum denotes a downward trend; (3) the top-5 lags of the time series, identified by computing the autocorrelation using fast Fourier transformation and selecting the five lags with the highest correlation values. Please refer to our code for more details, which has been made available to AC and all reviewers. We have also revised the paper accordingly and included this technical detail in the appendix.

W2. The paper's foundational concept hinges on the idea of reprogramming. However, it omits references to several pivotal reprogramming studies from recent years (as listed below). Integrating these pertinent studies in the introductory or related works section could provide a more robust understanding of the concept.

Thank you for the suggestion. To enhance comprehension of model reprogramming and more related works, we have revised and expanded the discussion on cross-modality adaptation in Appendix A.

W3. There are some writing issues. For instance, there should be uniformity in the references' formatting, with a preference for citing formally published works from conferences or journals over preprints.

Thank you for the suggestion. We have thoroughly reviewed and updated the reference list, including both the format and venues of the references. Additionally, considering the rapid growth in LLMs and related studies, many references in our work are recent preprints from 2023. These references are crucial in offering the necessary background knowledge for a better understanding of the proposed method.

Q1. Could you elucidate the mechanism behind "Output Projection" (Sec. 3.1), particularly the aspects related to flattening and the linear projection? A formulaic representation would greatly aid in understanding.

Upon packing and feedforwarding the prompt and patch embeddings $\mathbf{O}^{i}$ through the frozen LLM, we discard the prefixal part and obtain the output representations, denoted as $\tilde{\mathbf{O}}^{i}$. Subsequently, we follow PatchTST [1] and flatten $\tilde{\mathbf{O}}^{i}$ into a 1D tensor with the length $P \times D$, which is then linear projected as $\hat{\mathbf{Y}}^{i} \in \mathbb{R}^H$. Due to space constraints in the main text, we have revised the paper accordingly and included this technical detail in Appendix B.1.

Q2. In the "Prompt-as-Prefix" paragraph, there is a reference to computing the trend and lag in relation to time series, with several potential implementations hinted at. Could the authors detail the methodology used to determine the trend and lag information within the prompts?

See our response above to W1.

We hope our responses provided above can adequately address the reviewer's concerns and questions.

We extend our sincere thanks to the reviewer for their constructive feedback on our paper. We are deeply grateful for the recognition of our contributions and the significant potential of this work. Below, we have addressed the main concern raised by the reviewer.

W1. The authors chose a limited set of time series data to benchmark this model on. Many of the models they compare against have been benchmarked against other data sets: weather, traffic, electricity consumption, and the spread of influenza. a) If this work is meant to show that the model is state-of-the-art across many potential time series tasks then comparisons across other datasets (mentioned above) would be much more convincing. This work only uses ETT and M4. b) If this work is meant to benchmark this model's capabilities on a single data set then please make that clear. The state-of-the-art claims should only be made with regard to the data sets the model was benchmarked against.

• Our primary objective in this work is extending the capabilities of large language models to practical time series forecasting within a model reprogramming space. We also mean to evidence that Time-LLM consistently outperforms or matches the state-of-the-art performance in mainstream forecasting tasks, especially in few-shot and zero-shot scenarios. We believe this falls into the category (a) mentioned by the reviewer.

• We have performed additional experiments on four datasets: (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). Our detailed results can be found in the main text (i.e., Tables 1,3, and 4) and appendices (i.e., Tables 10,11,14, and 15) of the revised paper. For the reviewer's convenience, a very brief summary is provided below, highlighting the overall performance on a set of most representative methods on additionaly added four datasets.

  • A concise summary of additional long-term forecasting results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  Method	Time-LLM		GPT4TS		DLinear		PatchTST		TimesNet		FEDformer
  Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
  Weather	0.225	0.257	0.237	0.270	0.248	0.300	0.225	$\underline{0.264}$	0.259	0.287	0.309	0.360
  Electricity	0.158	0.252	0.167	0.263	0.166	0.263	$\underline{0.161}$	0.252	0.192	0.295	0.214	0.327
  Traffic	0.388	$\underline{0.264}$	0.414	0.294	0.433	0.295	$\underline{0.390}$	0.263	0.620	0.336	0.610	0.376
  ILI	1.435	$\underline{0.801}$	1.925	0.903	2.169	1.041	$\underline{1.443}$	0.797	2.139	0.931	2.847	1.144

We express our sincere gratitude to the reviewer for providing valuable feedback on our paper. We are deeply appreciative of the acknowledgment of the contributions and significance of our work. To our best effort, we have addressed the concerns and questions raised by the reviewer below.

W1.1. Inclusion of diverse datasets such as Weather, Electricity, and Traffic, commonly featured in literature, would offer a more holistic assessment.

We have performed additional experiments on four datasets: (1) Weather, (2) Electricity, (3) Traffic, and (4) Influenza-like Illness (ILI). Our detailed results can be found in the main text (i.e., Tables 1,3, and 4) and appendices (i.e., Tables 10,11,14, and 15) of the revised paper. For the reviewer's convenience, a very brief summary is provided below, highlighting the overall performance on a set of most representative methods on additionaly added four datasets.

• A concise summary of additional long-term forecasting results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  Method	Time-LLM		GPT4TS		DLinear		PatchTST		TimesNet		FEDformer
  Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
  Weather	0.225	0.257	0.237	0.270	0.248	0.300	0.225	$\underline{0.264}$	0.259	0.287	0.309	0.360
  Electricity	0.158	0.252	0.167	0.263	0.166	0.263	$\underline{0.161}$	0.252	0.192	0.295	0.214	0.327
  Traffic	0.388	$\underline{0.264}$	0.414	0.294	0.433	0.295	$\underline{0.390}$	0.263	0.620	0.336	0.610	0.376
  ILI	1.435	$\underline{0.801}$	1.925	0.903	2.169	1.041	$\underline{1.443}$	0.797	2.139	0.931	2.847	1.144

• A concise summary of additional 10% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  Method	Time-LLM		GPT4TS		DLinear		PatchTST		TimesNet		FEDformer
  Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
  Weather	0.234	0.273	$\underline{0.238}$	$\underline{0.275}$	0.241	0.283	0.242	0.279	0.279	0.301	0.284	0.324
  Electricity	0.175	$\underline{0.270}$	$\underline{0.176}$	0.269	0.180	0.280	0.180	0.273	0.323	0.392	0.346	0.427
  Traffic	0.429	$\underline{0.306}$	0.440	0.310	0.447	0.313	$\underline{0.430}$	0.305	0.951	0.535	0.663	0.425

• A concise summary of additional 5% few-shot learning results in average on Weather, Electricity, Traffic, and Influenza-like Illness (ILI):

  Method	Time-LLM		GPT4TS		DLinear		PatchTST		TimesNet		FEDformer
  Metric	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE	MSE	MAE
  Weather	0.260	0.309	$\underline{0.263}$	0.301	0.263	0.308	0.269	0.303	0.298	0.318	0.309	0.353
  Electricity	$\underline{0.179}$	0.268	0.178	$\underline{0.273}$	0.176	0.275	0.181	0.277	0.402	0.453	0.266	0.353
  Traffic	$\underline{0.423}$	$\underline{0.298}$	0.434	0.305	0.450	0.317	0.418	0.296	0.867	0.493	0.676	0.423

We express our gratitude to the reviewer for providing constructive feedback on our paper, and we greatly appreciate the acknowledgement of our contributions. We have addressed the specific concerns raised by the reviewer as detailed below.

W1.1. Statistical methods, such as AutoARIMA, AutoTHETA, and AutoETS, were not compared with.

We have included AutoARIMA, AutoTheta, and AutoETS as baseline methods. The detailed results can be found in Table 11 of the revised paper. For the reviewer's convenience, we provide a summary of the relevant (averaged) results below:

W1.2. N-BEATS and N-HITS were only compared during short-horizon forecasting.

We have compared to N-BEATS and N-HITS in long-term forecasting tasks, following the reviewer's suggestion. The detailed results are in Table 11 of the revised paper. We have provided a summary of the relevant (averaged) results below:

W1.3. Recent papers on using LLMs for time-series forecasting were not compared against for, e.g. LLM4TS and PromptCast. I am aware that the paper "LLMs are zero-shot forecasters" only got recently published, but it would improve the experiments if the authors were able to compare with it.

This is a good suggestion. Following the reviewer's recommendation, we have also included a comparison with LLMTime, even though this is completely optional according to the author guide this year. Our considerations are twofold: (1) LLM4TS is not yet open-sourced, and (2) LLMTime has shown better performance compared to PromptCast and operates directly on time series data. The additional results we have incorporated can be found in Tables 5 and 16. For ease of reference for the reviewer, we have summarized the relevent (averged) results below:

Note that to ensure a fair comparison, we have aligned the backbone LLM used in LLMTime with that of our method in terms of model size, specifically employing Llama2-7B. It should also be noted that LLMTime does not explicitly involve a transfer step but directly inferences on target datasets, resulting in some entries sharing same results. In a nutshell, Time-LLM demonstrates a substantial improvement over 75% and 53% in terms of the averged MSE and MAE.

We thank the reviewer for offering the valuable feedback. We have addressed each of the concerns raised by the reviewer as outlined below.

W1. It is unconvincing to directly transfer the knowledge of natural language in LLMs to time series tasks. Note that 1) text and time series are distinct data modalities, and 2) the pre-trained LLMs are not pre-trained with text-time-series pairs.

• While LLMs are typically pre-trained on extensive text corpora, they have been proven to be effective in pattern recognition and reasoning over complex sequences of tokens in approximating flexible distributions over numbers [1,3]. This capability can be well extended to time series data, as demonstrated in this work and in recent concurrent and follow-up studies [2-6]. As an emerging and promising field of study, a recent survey [7] also offers a comprehensive overview of research that utilizes LLMs not only for time series but also for more intricate spatio-temporal data mining. This further underscores the proficiency of LLMs in understanding and reasoning on temporal data.
• In addition to the evidence presented above, LLMs are known to encapsulate rich information and rules about the physical world through natural language modality, leading us to hypothesize that some inherent knowledge may also be relevant to understanding time series. Notable examples, such as those in references [2] and [3], have achieved a certain level of precision in zero-shot forecasting through direct prompting. Our contention is that to fully activate the LLMs' ability in time series understanding and reasoning, it is crucial to effectively align the modalities of time series and natural language, as discussed in the third paragraph of the introduction. This concept is further elaborated in the fourth paragraph of the introduction in two aspects: (1) the central idea is to reprogram the input time series into text prototype representations that are more naturally suited to language models’ capabilities; (2) to further augment the model’s reasoning about time series concepts, we introduce Prompt-as-Prefix (PaP), an innovative approach that enriches the input time series with additional context and provides task instructions in the natural language modality.
• Our established and newly added experimental results clearly demonstrate the fact that off-the-shelf LLMs are can be very effective in general time series forecasting, whether compared to SOTA time series models or the best available (open-sourced) related models like GPT4TS and LLMTime [3]. Although LLMs are often considered black-boxes, our analysis in Fig. 5 illustrates how the proposed LLM reprogramming framework bridges the gap between natural language and time series.

[1] Mirchandani, S., Xia, F., Florence, P., Driess, D., Arenas, M. G., Rao, K., ... & Zeng, A. (2023, August). Large Language Models as General Pattern Machines. In 7th Annual Conference on Robot Learning.

[2] Xue, H., & Salim, F. D. (2022). Prompt-Based Time Series Forecasting: A New Task and Dataset. arXiv preprint arXiv:2210.08964.

[3] Gruver, N., Finzi, M. A., Qiu, S., & Wilson, A. G. (2023, November). Large Language Models Are Zero-Shot Time Series Forecasters. In Thirty-seventh Conference on Neural Information Processing Systems.

[4] Chang, C., Peng, W. C., & Chen, T. F. (2023, August). Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469.

[5] Spathis, D., & Kawsar, F. (2023). The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models. arXiv preprint arXiv:2309.06236.

[6] Cao, D., Jia, F., Arik, S. O., Pfister, T., Zheng, Y., Ye, W., & Liu, Y. (2023, October). TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting. arXiv preprint arXiv:2310.04948.

[7] Jin, M., Wen, Q., Liang, Y., Zhang, C., Xue, S., Wang, X., ... & Xiong, H. (2023, October). Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196.

Thanks to the authors for explaining the critical errors in the code. However, these explanations are not convincing. There are several contradictory points in these explanations that make it difficult for me to believe that the current version of the experimental results is accurate and reliable.

Firstly, there is a clear contradiction regarding the explanation that long-term, short-term, and few-shot forecasting — we implemented validation loss for early stopping, aligning with methodologies in related literature. This is because the short-term prediction experiments are conducted using the M3 and M4 datasets, and according to the description in Table 8 of Appendix B.2 in the submission, M3 and M4 datasets do not have validation sets. Therefore, it is impossible for the short-term predictions to be based on validation loss. In the data_loader.py file, specifically in the Dataset_M4 code, it can be observed that the M4 data indeed does not include a validation set (note that the exp_short_term_forecasting.py file also uses test loss to determine the best model, whereas the GPT4TS paper uses training loss for short-term forecasting). Based on this, the author's explanation lacks persuasiveness.

Secondly, the description regarding using test loss for zero-shot forecasting is also problematic. Please refer to the code of GPT4TS (https://github.com/DAMO-DI-ML/NeurIPS2023-One-Fits-All/blob/main/Zero-shot_Learning/main_test.py). In their zero-shot experiments, they indeed use training loss for early-stopping, not test loss.

Furthermore, in the provided README.md file, the author presents some training process information for the ETTm2 task. In these results, the author reports an MSE of 0.161 (and MAE of 0.253). This result is based on selecting the best test loss. If early stopping were based on validation loss, the MSE result should be 0.1674806. Unfortunately, the aforementioned incorrect results of 0.161 and 0.253 have been reported in Table 10. This deviation is significant and makes it difficult to determine the effectiveness of the proposed model.

Training process provided by the authors in README.md

Start training: long_term_forecast_ETTm2_512_96_TimeLLM_ETTm2_ftM_sl512_ll48_pl96_dm16_nh8_el2_dl1_df32_fc3_ebtimeF_dtTrue_Exp_0

• Epoch: 1 | Train Loss: 0.2048776 Vali Loss: 0.1279695 Test Loss: 0.1799820
• Epoch: 2 | Train Loss: 0.1946422 Vali Loss: 0.1226610 Test Loss: 0.1738900
• Epoch: 2 | Train Loss: 0.1946422 Vali Loss: 0.1226610 Test Loss: 0.1738900
• Epoch: 4 | Train Loss: 0.2154310 Vali Loss: 0.1196599 Test Loss: 0.1697755
• Epoch: 5 | Train Loss: 0.2022108 Vali Loss: 0.1181143 Test Loss: 0.1652272
• Epoch: 6 | Train Loss: 0.1936386 Vali Loss: 0.1249270 Test Loss: 0.1682286
• Epoch: 7 | Train Loss: 0.1956175 Vali Loss: 0.1149597 Test Loss: 0.1674806 > based on best validation loss
• Epoch: 8 | Train Loss: 0.1864047 Vali Loss: 0.1160124 Test Loss: 0.1614796 > based on best test loss
• Epoch: 9 | Train Loss: 0.1804296 Vali Loss: 0.1165261 Test Loss: 0.1651560
• Epoch: 10 | Train Loss: 0.1817255 Vali Loss: 0.1188157 Test Loss: 0.1667696

testing: long_term_forecast_ETTm2_512_96_TimeLLM_ETTm2_ftM_sl512_ll48_pl96_dm16_nh8_el2_dl1_df32_fc3_ebtimeF_dtTrue_Exp_0 mse:0.16147961839199066, mae:0.25324239444732666

Based on the rigor of research, I strongly encourage the authors to reconsider the validity of the model in subsequent versions and invest more time to provide sufficient training details and correct experimental results to verify the validity of the model.

I appreciate the authors' efforts to add a lot of supplementary experiments during the rebuttal period. However, due to bugs in the code and inconsistencies in the explanations provided, I cannot believe that the current version of the experimental results is accurate and reliable.