Lightweight Online Adaption for Time Series Foundation Model Forecasts

Dear reviewer, thank you for your detailed review and constructive comments. We are happy that you thoroughly assessed our claims and found that they held up. We provide answers to your question below.

1. Adding details to ablations section

Thank you for raising this issue. We have updated the paper to expand the ablation section, including adding experimental data presented in a table, which we hope will address your concerns.

2. Ablation shown in Figure 8.

We agree that our rationale behind the decision to remove high-frequency components would be further enhanced by including more datasets to Figure 8. We have updated Figure 8 to include the Traffic and Solar datasets, and we find that the same conclusions hold.

3. Definition of online feedback

The online feedback we discuss in the paper is given by the dynamically arriving data points from the time series. Each data point gives feedback on the accuracy of the previous forecasts which aimed to predict the value of that data point (and others). This feedback is then used to update the AdapTS-Weighter and AdapTS-Forecaster. Additionally, we do not assume the data points are corrupted in any way other than that modelled by the time series itself (i.e. they are reliable). We tried to explain this on the fourth line of the second paragraph of the introduction but understand from your comment that this need to be more clear, especially in the "Rolling Window Forecasting" subsection which formally describes the setting we look at. Therefore, we have updated the paper to fix this issue.

5. Prompt and rapid fine-tuning techniques

We agree with you that there have been many methods proposed in vision and NLP to perform rapid finetuning of FMs and will add discussion of this to our related work section. There is however one large difference between vision and NLP and time series, which is that in time series linear models are still competitive. We exploit this fact in our construction of AdapTS in that the AdapTS-Forecaster is a lightweight linear model and that we do not require the adding of parameters or the backpropagating through the FM which prompt tuning and typical adaptor based methods require. This makes AdapTS computationally efficient and agnostic to the FM used. Additionally, looking at work on rapid finetuning of FMs, AdapTS would roughly fall into the adaptor based fine-tuning paradigm. As pointed out by the other reviewers there has been a work on using adaptor based continual finetuning in time series, i.e. TAFAS. We have now compared to TAFAS, with the results shown in the table in our comment to reviewer 6Y6Z. Our findings are that AdapTS outperforms TAFAS in all cases looked at.

6. Use of AdapTS in other TSA tasks

Thank you for suggesting these potential avenues to increase the scope of this work. While we focus on time series forecasting in this work and AdapTS is built for forecasting, it certainly is interesting to think whether it could be extended to perform continual adaption for other TSA tasks. To construct such a method for one of these tasks you would need some lightweight and online updatable method to replace the AdapTS-Forecaster. For example, for imputation you can use FITS. Then for generative tasks you can still use the AdapTS-Weighter. However, for predictive tasks you would need to use a slight modification whereby, as in the Hedge algorithm, you select a class/action using the weights as the probabilities instead of doing a weighted average. We see this direction as future work and want to focus this paper on time series forecasting but, given your comment, have added mention that AdapTS could be extended to these other tasks into the conclusions section.

---

We would like to thank you again for your thoughtful review and hope that we have satisfactorily answered your remaining questions about the work.

Thank you for your helpful comments and questions! We provide answers to your questions below and hope we have satisfactorily answered them, in particular by performing several additional experiments.

1. Why fixed FMs?

The main reason we keep the FM fixed is due to the computational expense of updating it. This is demonstrated by the fact AdapTS is 2506x faster than online finetuning TTM; the fastest FM to finetune. Additionally, online updating of FMs and neural networks more generally is not straightforward, coming with numerous complications as demonstrated by work in continual learning (CL). Furthermore, current CL solutions for these problems like FSNet are often complex and architecture specific. In contrast, one of the goals of AdapTS is to be simple and FM agnostic. Your comment illustrates that this point needed to be clarified in the paper which has been edited accordingly.

To justify using a frozen FM instead of learning it online, we run an experiment where we finetune TTM alongside AdapTS. The results of this experiment (TTM-Fine+AdapTS) are given in the table for question 2. and show that it performs generally worse than AdapTS with TTM frozen (results are given relative to TTM+AdapTS). This alongside the vast compute benefits demonstrate why keeping the FM fixed is beneficial.

2. Comparisons with OneNet, FSNet and TAFAS

Thank you for pointing out the missing comparisons to OneNet, FSNet and TAFAS, which we agree should be compared to AdapTS. We have run these experiments as well and we provide the results for these baseline methods on the ETT datasets in the table below. Additionally, we provide results for OneNet where one of its forecasters is replaced with TTM (OneNet-TTM). We find that in all cases TTM+AdapTS performs better. Importantly, we perform much better than OneNet-TTM, our closest comparator. Importantly, AdapTS is also more computationally efficient than the other methods. We hope now to have better contextualised the effectiveness of AdapTS relative to these more compute-intensive adaptation methods.

Regarding the novelty of AdapTS compared to OneNet, the main contribution of AdapTS over OneNet is its superior computationally efficiency stemming from our ability to avoid performing online gradient optimization. The table indicates that this approach also provides a performance gain.

3. Higher frequency datasets and AdapTS performance when using a more advanced forecaster (or FITS)

We have run experiments using TTM on 3 per-second datasets: SWind, SSolar[5] and SCloud[6]. The results are given in the table below. As in the rest of our experiments we find that using AdapTS improves performance of FM forecasts. This shows that on high frequency datasets where computationally-fast adaptation is necessary, AdapTS still improves performance.

We also looked at using a more advanced forecaster, FSNet, and FITS instead of the AdapTS-forecaster. The results are provided in the table given in response to reviewer 26Ko. We find that using FSNet or FITS instead of the AdapTS-Forecaster results in reduced performance. We hope the results for FITS answers your 5th question. The reason we believe FSNet performs poorly compared to the AdapTS-Forecaster is that the online updating of complex models like FSNet is to a large degree an unsolved problem as shown by results in continual learning. In contrast, learning the AdapTS-forecaster online is relatively easy, resulting in better online performance.

4. Why does linearly combining AdapTS-Forecaster and FM forecasts improve performance?

AdapTS corresponds to ensembling an FM (trained across numerous diverse time series) and a lightweight online forecaster (fit on recent time series data). The difference in training data means the forecasters are less correlated which is known to provide a performance benefit. Also, the weighter is able to optimally tune the ensemble weight online based on past performance leading to better performance than the individual models.

[5] Monash Time Series Forecasting Archive

[6] How does it function? characterizing long-term trends in production serverless workloads

Dear Reviewer, thank you for your comments and constructive feedback. We are especially happy you found that "the contributions of the paper are novel and present ideas on how to ensemble predictions in an online fashion". We answer the comments you had below and are thankful that you pointed out issues in notation and framing which we have fixed.

Ensembling: You identify that the proposed method is actually an ensembling method - encouraging us to reframe with the terminology of ensembling or exponential weights.

We have edited the paper to incorporate this feedback and to frame the method in ensembling terminology. Additionally, to fully take your feedback into account we have renamed our method to ELF, standing for Ensembled with online Linear Forecaster. e.g. TTM+AdapTS becomes TTM+ELF. To reduce confusion in the rebuttal we have resorted to using the name AdapTS.

Details and Formal Notation: You point out that the paper is lacking details and formal notation in places. In particular, the AdapTS-Forecaster description could be made clearer.

Thank you for pointing this out; we feel that addressing these points has improved the quality of the paper. We have now updated the paper to make the explanation of the AdapTS-Forecaster clearer, clarifying the notation and providing a more precise explanation of model fitting. We have added dimensionality information where missing, as well. Additionally, an algorithmic description of fitting the AdapTS-Forecaster is now given in the Appendix.

Baselines: You identify a lack of comparison with relevant baselines, both for the AdapTS-Forecaster and AdapTS-Weighter

Thank you for this comment, it was mirrored in the feedback of other reviewers as well and has now been addressed. We have compared the AdapTS-Forecaster with FSNet, FITS and a linear model trained by online gradient descent (OGD) and we have compared the AdapTS-Weighter with Hedge as recommended. The results for TTM on the ETT datasets are provided in the table below. Results are given relative to our approach so that values above 1 correspond to worse performance. In each case results are worse than using AdapTS. These results will be extended to all datasets and FMs.

Results Presentation: You propose that it would be better to normalize the metrics to a baseline, then aggregate over the different datasets and settings.

Thank you for suggesting this way to incorporate model results into the main text. We have updated the paper to normalized MASE (w.r.t. the naive seasonal forecaster) and once all our additional experiments have run we will aggregate in the way you propose to ensure that as many of them can be presented in the main text as possible.

I want to verify that the loss used in exponential weighting is from previous time steps, i.e. no data leakage?

Yes, the true values used to train both the weighter and the forecaster at some time step t are only those already seen by that time step (i.e. 0 to t).

"Time series foundation models could be replaced with standard deep learning forecasting models, and the paper still reads exactly the same."

You are correct that this method could in principle be applied to standard deep-learning forecasters. We decided to focus on FMs due to the particular difficulties and computational cost of finetuning these approaches online.

Alteration Summary

Thank you again for your feedback. Below is a summary of improvements made based upon your feedback. We look forward to hearing your thoughts.

1. Compared Forecaster and Weighter against Baselines
2. Addressed issues with notation and terminology
3. Reframed our approach with the terminology of ensembling
4. Updated to Normalized MASE

Dear reviewer, thank you for your kind words about our work and constructive comments/questions. We are particularly happy that you found that AdapTS gives "significant performance improvement with low computational cost" and is "applicable to any FM". We have included additional baseline comparisons against test-time adaptation methods and provide answers to your questions below.

Summary of comparisons against test-time adaptation and gradient-based methods

In response to your comments, we have now run experiments with several additional baselines: TAFAS (TTM-TAFAS); AdapTS-forecaster to predict residuals of the FM forecasts (TTM+Residual-Adjustment); a gradient-based online-ensembling approach for time series forecasting (OneNet-TTM) and fine-tuning the FM (TTM-Fine). The experiments are shown in the table below for TTM, the best performing FM in our experiments, on the ETT datasets. We give the results relative to the MASE performance of TTM+AdapTS, for example a score of 1.043 means that TAFAS performs 4.3% worse than AdapTS.

We find that, in all cases looked at, AdapTS performs better than the baselines and discuss each experiment in more detail in the sections below. We hope these experiments satisfy your comments on comparing to test-time adaptation and gradient-based methods and we believe those additions significantly strengthen the paper. We further note that we are currently running experiments across all the datasets and FMs and will add them to the paper once they complete.

1. Why is $w_r$ a scalar, rather than being designed as a vector or a tensor?

If we understand correctly, when you mention $w_r$ you are referring to the weight used to merge the forecasts of the FM and AdapTS forecaster (in Eq 2.)? If so, making $w_t$ a vector would mean that there would be a different weight applied per time-step (using the element wise product between $w_r$ and $y_{t,FM}$). In the development stage of this work we did look at this, but found that using only per channel weights yielded better performance, due to overfitting. Therefore, we did not discuss it in the paper, however to clarify the text and to address your comment, we will add appropriate mentions of it in the paper. As for making the weights a tensor, we do not know exactly what you mean by this, could you please clarify?

2. Comparison to test-time adaptation method TAFAS

We agree that this work would be greatly strengthened by comparisons with the test-time adaption baseline TAFAS. As mentioned before, results of these experiments and other baselines are given in the table at the start of our comment. We find that in all cases looked at AdapTS performs better than TAFAS. We also note that TAFAS requires backpropagating through the FM, making it computationally slower than AdapTS. This demonstrates the advantage of AdapTS over TAFAS.

3. Why didn't the gradient-based approach work in experiments? In addition, gradient updates are not just about fine-tuning; for example, adding a small neural network to fit the residuals.

Gradient-based fine-tuning does not work well for the online learning of neural networks due to the problems studied in the continual learning literature. These problems mainly manifest as catastrophic forgetting whereby updating on new data the model forgets large amounts of information about old data. Hence, the gradient-based fine-tuning of TTM did not work for online updating (as shown in the table above). We note here that by using a linear model as the AdapTS-forecaster we can sidestep all the problems of continual learning which only occur for more complex models, while being significantly faster.

Additionally, we have run an experiment in which we use the AdapTS-forecaster to predict residuals of the FM's forecasts and use this to modify the forecast. We present the results for this method in the table at the start of our comment, and find that it performs worse than AdapTS in all cases looked at.

---

We thank the reviewer again for your constructive feedback. We hope given our answers and especially the added experiments comparing to TAFAS, fine-tuning, residual predictions and OneNet, we have satisfactorily addressed your concerns and believe that your comments have made our submission stronger.