In-Context Fine-Tuning for Time-Series Foundation Models

We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further comments or concerns. If our response sufficiently addresses your concerns, we hope that you will consider updating your score accordingly.

Given that the cost of evaluating performance should be reasonable, it would be worth to explore a grid of options for the number of in-series and out-series example and plot a heatmap of the performance. For this, I believe it is a must to consider a set of datasets that are not used for the final selection (or at least to restrict the collections and be explicit about this) to not risk overfitting.

Thank you for this feedback. Based on your suggestion, we constructed a validation dataset from the training portion of a subset of the Monash datasets (specifically: weather, traffic, australian electricity, ercot, ETTm, and ETTh). We chose these datasets because they contained many training examples long enough to construct up to 20 in-series examples. We measured the validation MASE error of TimesFM-ICF with the number of in-series examples varying from 0-20, and the total number of in-context examples (including randomly selected examples) varying from 1-50. The resulting heatmap is attached in Figure 1 in this link: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. The configuration with smallest validation MASE was 11 in-series examples and 34 total examples. The geometric mean MASE ratio (averaged over 5 runs with different random examples selected) was 0.780+/.003 (so within a standard error of the MASE value we report in Figure 5).

Given that your results depend on a random sampling, analysing the noise coming from this decision would be important, it could be done with the analysis I discussed above by reporting confidence interval on mean performance estimates. In addition, reporting the confidence interval on the aggregated results on Fig 5 and others across datasets would be good to convey the uncertainty on the scores.

Please see updated OOD tables in https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf which now include confidence intervals for TimesFM-ICF. Each entry in the tables is an average over 5 runs of random in-context example selection (using our original in-series configuration). Note that our Figures already include confidence intervals.

instead of a separator, is there a reason why you did not consider a simple positional encoding? (eg a categorical embedding mapping the time-series index concatenated to your input, of-course using a fix index for the input time-series)

Using a time series level positional encoding is definitely a valid option. One advantage of our proposal is that the pretrained model weights could potentially generalize to more than 50 in-context examples thanks to our current choice of not using any form of positional encoding. This is an open question we will empirically explore in the future.

why are you making a causal mask in your architecture for the conditioning out-series? I would have imagined a mask that allows only the current in-series to look at other out-series but not to let an out-series looks at previous ones as it is less relevant for a out-series to look at the previous ones (also reinforce the random choice of the time-series order when sampling)

As we train the model in decoder-only manner, during training time there is no notion of a single “in-series”: for the forward pass on a training context (see Section 5.1. Context Generation), every series (example) is its own “in-series” and all its preceding series are its “out-series”, therefore we have to apply a full context level causal attention so that decoder-only works. At inference time we hence choose to keep the attention between out-series, which is for consistency with model training. This is identical to how language models commonly handle in-context examples (few-shot prompts).

It is a good point that our design breaks the symmetry among out-series. Empirically we tried to minimize its effect by training with randomizing the order of in-context examples when there is no causal leakage (see Section 5.1. Context Generation, grouping, “Dataset level:”). It’s also an interesting question that what happens if we remove this attention between out-series at inference time - we will do it as a future study.

Fig 9: how are you computed the confidence interval? have you considered ensembling the predictions obtained when sampling different history?

In Fig 9, the uncertainty of the reported metric comes from the random selection of in-context examples, and the confidence interval is computed based on 10 runs of the same setup with different random seeds.

It’s definitely possible to ensemble the predictions based on repeated sampling of different in-context examples. The practitioner can make this choice at the cost of increased latency.

We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further questions. If our response addresses your concerns, we hope you consider raising your score accordingly.

Section 6.3: Moirai

Thank you for clarifying this point. Our interpretation from the Moirai paper in Section 3.2.2 Task Distribution was that the training datasets were augmented by “constructing multivariate time series from sub-datasets with univariate time series, by randomly concatenating them”. Based on this, it seemed that Moirai could support past related examples as multivariate features. We do not want to misrepresent the capabilities of the Moirai model in our paper, so we will contact the Moirai authors for clarifications, and correct this as the reviewer suggests.

Section 6.4.2: TimesFM (base)

We apologize for the lack of clarity - we indeed pretrained TimesFM-LH with a longer history length of 2048 (in a manner similar to the latest version of the TimesFM repo (https://huggingface.co/google/timesfm-2.0-500m-pytorch). We didn't just use TimesFM (base) directly with a 2K context length at inference time to get the TimesFM-LH results. We will make this more clear in Section 6.4.2

ICF appears to be highly inefficient in terms of inference time [...]

Since the total context length for TimesFM-ICF is 50 times larger than TimesFM, the 50 times inference speed slowdown is not unreasonable. However, TimesFM-ICF is still a zero-shot model, the addition continual pretraining phase for training TimesFM-ICF is a one-time cost that is independent of the target dataset – there is no cost to be paid to adapt the model for a new domain or dataset (unlike fine-tuning costs, which has to be paid to adapt TimesFM for every new dataset). More importantly, for practitioners, the ability to use the ICF model out of the box, and avoid having to build a fine-tuning pipeline to customize the model for their use-case is a very significant cost and resource advantage that we think can more than offset the 50x inference speed disadvantage (note that even after this 50x inference slowdown, TimesFM ICF can still perform inference for a single example in 43 ms on average - totalling 25 minutes on approximately 140k testing time series on TPUv5e with 4 cores - which is often sufficient for most practical forecasting use-cases)

Suggestions [Sec 6.2]

Thank you for the suggestions. We will update the final version of our paper accordingly.

other TSFMs beyond TimesFM?

We believe that our methodology is generalizable to other TSFMs. This is an interesting direction for future research.

maximum history length limited to 512?

The maximum history length constraint in TimesFM comes because of the pretraining configuration - the model has been pretrained on examples up to the maximum history length and might not generalize well to lengths beyond what it has been pretrained on. Increasing the maximum history length during pretraining will result in longer training times and compute resources

maximum of 50 examples in context?

The choice of 50 examples for TimesFM-ICF comes from its continual pretraining setup where the model is trained with up to 25K (512*50) context window. Increasing the context window in continual pretraining would have caused longer training time and compute resources, which we would not have been able to afford. At inference time the model may generalize to beyond 50 examples - there is no mechanism within the model casting a limit of 50 indeed. For simplicity and conciseness of the paper, we also use 50 examples in our empirical study for consistency with the continual pretraining setup. Verifying the generalization is a separate, empirically heavy task that we plan to study in the future.

I believe Moirai is trained to accommodate a maximum of 128 variables, not 50

Sorry for the wording issue. We did not intend to imply that it accommodates at most 50, just that it accommodates 50. We will clarify our wording in the final version.

What is the fundamental difference between in-context fine-tuning and using exogenous variables?

We view in-context fine-tuning and exogenous variables as two separate concepts: The former supplements the main forecasting history with time series generated from the same or a similar distribution, in which sense it is in-context “fine-tuning” to fit this distribution. On the other hand, exogenous variables can be of any distribution, and their usefulness mainly comes from their correlation with the main forecast task. In this regard, in-context fine-tuning is a more strict practice.

Why the length of in-context examples can be different?

At both training and inference time we apply paddings to short in-context examples to bring them to length L+h while at the same time apply proper attention masks. Please see Section 5.1., Context Generation.

We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further comments or concerns.

Only the subset of zero-shot benchmark is utilized. While this is a good idea to preserve the "zero-shot" setting, it would be advisable to additionally report results for the full benchmark and report which datasets are non zero-shot as this would make comparison to other papers/models easier.

Thank you for the suggestion. We have added evaluations on the missing four datasets (m4 quarterly, m4 yearly, weather, and traffic) in Table 1 (MASE) and Table 2 (WQL) here: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. In these tables, we additionally report the geometric means of the scores over the 23 (zero-shot) datasets originally reported in our paper (row “Geometric Mean (ZS)”) and over all 27 datasets (“Geometric Mean (All)”).

As the evaluation includes randomness of the in-context samples, the authors should provide the variation over the individual datasets of the benchmark similar to Figure 9 for the "overall result".

We have also added confidence intervals to Tables 1 and 2 in the link above (https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf). Note that all of our figures already include error bars. It was not clear enough in Figure 5 and we will fix it.

Context length increases linearly with an increasing amount of context samples, therefore, computation complexity for the transformer increases quadratically

We agree with the reviewer’s comment here. This quadratical attention computational complexity does not directly translate to a quadratical inference time likely due to (1) the latency of the feedforward layers and (2) optimization of transformer implementation on modern accelerators.

Why did you decide to call the benchmark OOD benchmark? Typically, OOD is more often referred to situations which one has to detect (OOD detection) as the models do not perform reliable on it. In my understanding, the idea of pre-trained models is that the models should generalize over time series data, including the evaluated zero-shot data. Hence, in most related literature, this evaluation is more often referred to as "zero-shot" benchmark.

Thanks for pointing this out. We have abused the notion of OOD as out of the pretraining data distribution. We did not call it “zero-shot benchmark” because such a name was mentioned in [1] and we wanted to be clear that our benchmark is its subset that’s zero-shot for TimesFM-ICF. We will clarify and update the name to zero-shot benchmark.

[1] Ansari, Abdul Fatir, et al. "Chronos: Learning the Language of Time Series." Transactions on Machine Learning Research.

We thank the reviewer for their helpful comments and suggestions. We address the main points below. Please let us know if you have any further comments or concerns. If our response sufficiently addresses your concerns, we hope that you will consider raising your score accordingly.

I would suggest the authors to update the bibliography entries.

Thank you for pointing this out. We will update the bibliography accordingly in the final version of our paper.

why in the proposed paper the base model is trained on even more datasets (LOTSA datasets) than it was in the original TimesFM paper? I wonder if the results would still hold if the same pretraining methodology was followed as in the original TimesFM paper.

While our study focuses on the improvement introduced by the in-context fine-tuning continued training over a base model, we intend to start from an up-to-date base model. Therefore we use the same datasets that are specified in the latest version (v2) of the TimesFM huggingface repo (https://huggingface.co/google/timesfm-2.0-500m-pytorch), which does include LOTSA.

In the original TimesFM paper one of the mentioned limitations is that covariates can not be included in the pretrained model. I wonder if the proposed approach in this paper can accomodate for this. Perhaps in-context-samples can be taken from covariates that evolve on time (dynamic features). This would be a clear-cut for practitioners to adopt this model in the industry, as covariates often come as de-facto request.

As our model is trained only with in-context examples coming from the same or a similar time-series that shares the same / a similar distribution as the target time series, we expect that providing other covariates as in-context examples likely would not yield good performance out-of-the-box. Adapting our model to allow for additional covariates is an interesting direction for future work.

I wonder if the provided model can generate probabilistic forecasts and what the corresponding evaluations in terms of mean weighted quantile loss would be.

Thank you for your suggestion. We have evaluated the wQL from our proposed model along with some of the other baseline methods, and the results show that our model performs significantly better on wQL compared to the baselines. The full results are uploaded separately in Table 2 here: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf, and we will add these results to the next version of the paper.

Missing visualization.

Thanks for pointing out our lack of clarification here. In the current paper, Fig. 7 is a limited visual example of how the ICF model behaves differently from a base model. We’ve created additional visualization on the australian_electricity dataset (which has 5 time series) to demonstrate how the forecast changes with a large number of in-context examples. Refer to Figure 2 in the following link: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. In this figure, we plot the predictions of TimesFM-ICF operating in three modes: 0 in-context examples, 20 (random) in-context examples, and 50 in-context examples (5 of which are within-series examples). These three configurations have increasingly better MASE scores on this dataset (with MASE values ~1, ~.9, and ~.8, respectively). The predictions visually appear to improve with the MASE values.

Report on 27 datasets instead of 23

We have added evaluations on the missing four datasets (m4 quarterly, m4 yearly, weather, and traffic) in Table 1 (MASE) and Table 2 (WQL) here: https://anonymous.4open.science/r/icml25-DB6C/icml25.pdf. In these tables, we additionally report the geometric means of the scores over the 23 (zero-shot) datasets originally reported in our paper (row “Geometric Mean (ZS)”) and over all 27 datasets (“Geometric Mean (All)”). The results of Figure 5 continue to hold when adding the additional 4 datasets (which recall are not zero-shot for our model).