Are Language Models Actually Useful for Time Series Forecasting?

Thanks for your thoughtful and positive review! We've responded to each of your points below.

W1: Can our work extend to other uses of LLMs for time series? We agree this is an exciting, natural direction for our work. While it’s beyond the scope of one paper, we hope our work inspires the community to do exactly this, ultimately taking a closer look at such LLM-based methods broadly.

W2: Can our work extend to irregularly-sampled time series? Thanks for noting our focus on evenly-spaced time series. We completely agree that irregularly-sampled time series are of critical interest and we hope to consider them in future works. For this current work, we believe our hypotheses are best tested by following the prior works’ experimental setups as closely as possible and as LLMs are used for more data types, we hope they are subjected to careful examination!

W3 and Question: Could we provide deeper discussions of LLMs’ weaknesses and give theoretical insights? This is a great idea and diving deeper into why the LLMs underperform is extremely interesting. But there are many open questions about this type of interpretability and many challenges of proving negative results. While we probe at this “why” question in Sections 4.3-4.6, we hope our work inspires others to ask similar questions and join in developing the theory behind when/why/if LLMs can benefit time series forecasting (and other tasks). As it stands, we believe our work successfully presents tension between the promise of big, multi-modal models and a need to deeply understand the sources of practical performance.

Per your direct question, developing theoretical explanations should be done with care over longer periods of time than a rebuttal. But in the spirit of brainstorming, we provide two speculative lines of attack for future works:

1. There may be a lack of key, transferrable properties of time series in LLM pre-training data. For example, forecasting often depends on learning periodic trends in time series, which are not a key signal present in natural language.
2. Long, forecastable sequences of numbers may not be in the pretraining data, driving a need to update an LLM’s weights beyond the point of leveraging its language abilities.

Thank you for the positive and encouraging feedback and for your endorsement of our work! We've addressed your questions below.

W1: Why does RQ3 only consider LLaTa?

Thanks for suggesting we extend RQ3 to include all methods. We completely agree this would strengthen this RQ and our choice was due to computational constraints—The “woPre+FT” ablation requires training the LLM from scratch. Training Time-LLM’s 7B base LLM so many times is beyond our immediate resources. But we will take your suggestion and see if we can find a way to run this for the final version. We also omitted OneFitsAll from this experiment because OneFitsAll and LLaTA have similar performance, use the same base LLM, and our aim in this RQ is just on the impact of the LLM’s pretraining. But we will take your suggestion and run this experiment. We strongly believe this finding doesn't hinge on this experiment, but we will include at least one more method (hopefully two).

For OneFitsAll’s random initialization experiment, we’d like to note that their results were reported in a few-shot setting, unlike ours. Running our experiment again in a few-shot setting is an interesting future experiment, though.

W2: How does PAttn compare to PatchTST?

This is a great question! We agree that they are similar, and we’d like to note that PAttn is designed only to explore the role encoders play in LLM-based forecasters’ performance. As shown in our extended comparisons (see our uploaded pdf), PAttn and PatchTST actually do perform similarly, likely due to their architectural similarity. The key difference is PAttn lacks a position embedding and feedforward layer in the transformer.

Thank you for your focused and actionable review. We are glad you agree our work provides interesting and insightful observations! As detailed point-by-point below, we’ve addressed your remaining concerns by running your suggested experiments.

W1: Including state-of-the-art forecasting models

Thank you for this suggestion! We’d like to clarify that each LLM-based method claims to be state-of-the-art, reporting they outperform non-LLM methods like PatchTST and iTransformer:

• “TIME-LLM is a powerful time series learner that outperforms state-of-the-art, specialized forecasting models” (taken from Time-LLM’s abstract).
• “[LLaTA] establishes state-of-the-art performance for both long-term and short-term forecasting tasks” (taken from LLaTA’s abstract; note that “LLaTA” was renamed to “CALF, which we will update in our final version).
• “pre-trained models on natural language or images can lead to a comparable or state-of-the-art performance in all main time series analysis tasks” (taken from OneFitsAll’s abstract).

So the merit of our work doesn’t depend on this comparison. Our reproduced MSE and MAE values are also nearly-identical to those of each method’s original experimental results, so we have no reason to believe this comparison won’t also hold. We also agree with you that comparing to non-LLM methods is largely out-of-scope of our contributions.

But to address your suggestion directly, and in case future readers have a similar question, we have added extended versions of all comparison tables to the Appendix (see pdf uploaded with our rebuttal). These tables include results from the methods compared in the iTransformer paper. As expected, our main findings are unchanged: The LLM-based methods are slightly better than the non-LLM methods, but our ablations indicate this isn’t due to the LLM.

W2: Including more details on LLaTA’s source of performance

Thanks for this suggestion, we believe you are describing RQ6 where we show that a simple patching method is surprisingly performant (compared to all methods and also other new and simple baselines). While RQ6 focuses on encoders and particularly on the more-popular patching and decomposition, there are still some insights about LLaTA’s encoder. In Section 4.6, we will clarify that “LTrsf” is LLaTA’s encoder without cross-modal attention.

Q1: Why does RQ4 focus on the ETTh1 and Illness datasets?

Thank you for suggesting we add more datasets to RQ4—We chose ETTh1 and Illness randomly due to compute constraints leading up to the submission.

Per your suggestion, we ran this experiment, so now RQ4 includes all 8 datasets. We've previewed our findings below, but could only include "Sf-all" and "Ex-half" due to OpenReview's space constraints. All results will be added to the Appendix. We also note that answering RQ4 doesn't depend on the number of datasets because if LLMs’ strong sequence modeling really drove their forecasting performance, shuffling time series should always drop their performance more than their ablations’.

Table: Subset of results from RQ4 experiments on six more datasets. Due to OpenReview's character constraints, we only show "Sf-all" and "Ex-half" results for Time-LLM—These experiments are already completed for all methods. These new results agree with those from our original paper.

Q2: How is LTrsf implemented and how does it connect to iTransformer?

Thank you for noting the similarity between LTrsf and iTransformer. The key difference is that "LTrsf" leaves each channel independent, ignoring multivariate relationships. LTrsf is implemented as LLaTA's encoder with cross-modal attention removed (so it's only the transformer encoder) followed by one linear layer. We will mention this in RQ6 and clarify that they outperform one another in different cases, so their strengths appear orthogonal.

Q3 : Can we compare PAttn and PatchTST?

Thanks for suggesting we add PatchTST's results to those of PAttn. We have added this comparison to the pdf uploaded with our rebuttal. We’d like to note that comparing non-LLM forecasting methods is unrelated to our work. Still, we will add this result, per your suggestion, to establish how PAttn compares to others. Please also note that we use PAttn to probe the behavior of LLM-based forecasting methods, not to outperform non-LLM methods. However, it is competitive in many cases.

Thank you for the encouraging feedback and the very positive review! We've fixed the typo and responded to your query about Table 5 below.

Why does "woPre+woFT" perform well in Table 5?

This is a great observation! We were initially surprised by this, too. But we’d like to clarify that “woPre+woFT” describes only the LLM in the model, where the LLM’s parameters are randomized (woPre) and then frozen and left untrained (woFT). The rest of the model is still finetuned. So the fact that its metrics are similar to the other variants actually backs up our main claim: If the rest of the architecture can achieve similar performance even with a random, frozen LLM injected into the middle, the LLM is likely not driving the forecasting performance.