Thank you for your insightful review. Please note that we updated the paper shortly after the rebuttal period commenced, including the following: an improved model that uses 437 basis functions that is trained on 25 datasets (an additional 15 training datasets that we sourced and collected after submission), improved performance across all tasks, and improved figures. Please see the comment summarising these and other initial changes. That said, we will be updating the paper before the end of the rebuttal period, accounting for any of your concerns and suggestions.

Regarding the DAM as a foundational model

The process of collecting, curating, and training on many time series datasets is time intensive and challenging. While there are many very short time series datasets readily available, these are less useful for training when the task is long-term forecasting, hence we do not include short datasets. More data means that we can train for longer without overfitting, and typically generalise better on test set performance and in the zero-shot setting. However, longer training is also costly and time-intensive. As we mentioned in the conclusion: future work will entail scaling up the DAM both in terms of model size and training corpus. As it is, the DAM is the only forecasting model that is designed for training across many time series datasets. We had to rethink the way time series were ingested and predicted in order to design the DAM; and that is our primary contribution. The generality, flexibility, high-performance, and utility of the DAM is what makes it a foundational model.

The nature of time series data

There are several challenges to overcome for training a time series foundational model, namely:

1. Collection and curation of time series data.
2. The non-iid nature of time series datasets (i.e., temporally close data points are highly dependent).
3. Owing to this non-iid property, balancing amongst data points, variables, and datasets is non-trivial and remains an open question. For example, an extremely long time series dataset that is highly repetitive (i.e., a daily repeating signal) is less 'useful' for training than a shorter time series that contains many diverse patterns that are 'useful' to learn on. Thus, the optimal training regime is complex and not yet known. Perhaps this is the performance bottleneck.
4. Scaling the model up results in longer training runs.

That said, the DAM is a foundational model because it can be trained across many datasets by design. Hopefully the inclusion of the additional 15 time series datasets, and consequent improved performance, is good evidence for this. We are open to any analysis suggestions you may have, however. Please let us know if there are any additional experiments you would like to see.

Many appendices

As you rightly mentioned, addressing a foundational model for time series forecasting is very challenging. Hence, we required to include longer appendices. . Another reason for the large number of appendices is that we endeavoured to be as open and clear as possible with this work. For example, including the PyTorch code of the model directly in an Appendix makes it far easier for other researchers to reuse and replicate our work. Most of the appendices are not strictly necessary, but rather a consequence of us trying open our research, since we believe that the DAM is an actual important contribution to the field of time series forecasting because of what it enables.

In the current update of the paper we improved the results discussion such that it relies less on the appendices.

Peak prediction

One of the contributions of this paper is that we compare 14 SoTA baseline methods (with 8 of in the appendix, owing to space constraints and their relatively lower performance compared to other baselines) across 10 benchmark datasets. It is rare in this field to see comparisons across this many datasets. We intentionally did not include weak or outdated methods (e.g., LSTMs) because of how strong the DAM's performance is, and we wanted to posit it amongst the absolute current SoTA. This is one reason for the DAM to show weaker peak time predictions in comparison to some of the selected methods, as we are only comparing against the best models in the SoTA. The updated model (see above) has improved performance on 'peak' data.

The DAM is univariate

Please see the separate comment entitled 'A note on inference and the univariate nature of the DAM', where we explain precisely how inference is accomplished, and how the DAM is a fully univariate model. To predict a given variable the DAM only consumes context from that variable. We will clarify this by adding a section in an incoming update. Future work will involve extending the DAM by adding a generalised multivariate component.

Flexible inference cost

The DAM is far smaller than common LLM models. What we are trying to show in this section is that the DAM is flexible, and that a single model can incur low or high computational cost, depending on user choice. The ability to balance performance and inference cost, without having to train additional models, can be extremely useful in practical scenarios. For example, in cloud resource allocation, 200ms might be too high a cost. Although we agree that 200ms is already inexpensive, this might not hold for much larger versions of the DAM (future work).

Good day.

Please note that we have updated the paper as per your commentary and requests. There is a summary entitled "Rebuttal Summary - After addressing reviewers' comments and questions".

Of particular note for your consideration:

1. Appendix N - a small scale test of the DAM's scaling with model size (for a perspective of 'scaling laws'). We put this experiment together based our discussions with you. We find that the DAM does indeed scale well with increased model size, but also that training longer and across all 25 training datasets evidences a strong benefit.
2. A reworked discussion on the the limitations of the DAM, explaining how a future version could take cross-variable (i.e., multivariate) information into account. This might be the current limiting factor to the DAM, but there are many avenues requiring exploration in the future.
3. We have included an ablation study of all the components of the DAM (now Section 5.2). This indicates which components of the DAM are important, and therefore should be focussed on for future work. We hope that this gives a better insight into what the 'bottlenecks' are for improving performance.

Please let us know if you have any more questions or requests that we can address before the end of the rebuttal period.

Thank you again for your review.

Thank you for your comment. Your perspective suggests that you think that the DAM takes a significant step toward being a foundation model, but that we have not yet provided sufficient evidence to convince you that it is already foundational. Would it suffice if we used the term "towards" when referring to the DAM in the context of a foundation model?

That said, we propose to change the following parts of the paper:

1. The title, from "DAM: A Foundation Model for Forecasting" to "DAM: Towards A Foundation Model for Time Series Forecasting". This also better captures the scope of the work. Please note: changing the title and abstract will not reflect in this openreview setting, but rather only on the paper itself. This will need to be accounted for for the camera ready version, and we will do that.
2. In the introduction, from "We present the deep data-dependant approximate analytical model (DAM) as a foundation model for universal forecasting" to "We present the deep data-dependant approximate analytical model (DAM) as a significant step toward a foundation model for universal forecasting".
3. Also in the introduction, from "To the best of our knowledge, the DAM is the first foundation model for universal time series forecasting" to "To the best of our knowledge, the DAM is the first model for universal time series forecasting that can be trained simultaneously across many diverse datasets, of different resolutions, and for various horizons, such that it generalises well both within and outwith the training set".
4. In Section 4.2, from "This is strong evidence that the DAM is a foundation model." to "The DAM generalises well outside of its training set".
5. In the conclusion, from "We presented the DAM – a foundation model for universal forecasting" to "We presented the DAM as a significant step toward a foundation model for universal forecasting.".

Will that be sufficient for you to raise your score? If so, we would very much appreciate if you did that before the end of the rebuttal period (a few hours from now). That said, please also consider all of the significant updates we have made to the paper when considering what to change your score to. Specifically, as a step "toward" a foundation model for time series forecasting, we believe that our work on the DAM overcomes many significant practical hurdles in the time series domain (e.g., cross-resolution training, variable horizons, interpretability, etc.), and, as such, deserves serious consideration.

We will update the paper as soon as the changes are made and after we receive your confirmation.

Thank you for your insightful review. Please note that the current updated version of the paper has a number of improvements, including using more basis functions (437 vs 219), better results, improved figures, and more training datasets (an additional 15).

Regarding zero-shot experiments

Based on these reviews we have identified that the main weakness of our paper is the limited analysis on the zero-shot setting. You are correct that the main advantage of a foundational model for forecasting is its applicability to unseen datasets. There is another very important advantage, namely that training on many datasets simultaneously improves performance on those datasets, when compared to training on each individually. We postulate that this is owing to a combination of factors, including but not limited to:

1. Mitigating the early overfitting that is common when training time series models. Many SoTA methods train for 1-10 epochs, which is only 3125 iterations using a minibatch size of 32 on a training dataset of length 10000. This is 336x shorter than the training of the DAM. We did not witness overfitting when training. See the section below entitled 'Computational cost' for a discussion on the importance of this.
2. Learning from patterns in one dataset being useful for generalising to another dataset. For example, weather data from Germany will likely contain patterns that are useful to learn from when generalising to weather data from the US.

That said, we agree that more demonstrations on the zero-shot setting would be pertinent. To this end, we are in the process of analysing zero-shot performance across the following 8 datasets (6 new): Illness, wiki-weekdays, Monash (including ‘temperature rain’, London smart meters, weather, web traffic), UCI 'individual household electric power consumption' (https://archive.ics.uci.edu/dataset/235/individual+household+electric+power+consumption), and the Azure 2019 function trace.

How to assess zero-shot transfer?

Another reviewer (wavr) suggested that when comparing the DAM to other methods in the zero-shot setting, those comparison methods should be trained on said datasets (i.e., not zero-shot transfer). Given how poorly PatchTST and DLinear transferred to new datasets (Table 2), it is understandable to have a closer comparison. However, the issue is that the DAM is the only method we know of that is designed as a universal forecaster that is dataset agnostic: there are no fully fair comparison methods. We are happy to do what is suggested (even though this advantages other methods quite strongly), but would like to know your opinion regarding how to perform this analysis? What would you, ideally, like to in such an experiment? We are open to discussion and suggestions during this rebuttal period, and will endeavour to make ready an updated paper, with more zero-shot analysis, by the end of the rebuttal period.

Computational cost

As mentioned above and in the paper itself, the DAM takes much longer to train. At first this seems to be a disadvantage, but we argue that it signifies a positive shift toward a better match between deep models and time series. Consider other fields, such as NLP or vision: training runs are typically far longer than several thousand iterations. This is simply because the datasets available are far larger than time series datasets. That is one of the primary motivating factors for our development of the DAM; we had to rethink the way time series were processed and predicted in order to design a modelling approach that could extend to any time series dataset. Only then could we actually apply this model to a collection of datasets and move toward training runs comparable to other fields.

Perhaps a pertinent comparison to make is that of training cost of the DAM against the cumulative training cost of other methods. Assume an average training run of 3125 iterations (see above) per dataset. Assume also that we would have to train baseline methods on each horizon-dataset combination. For $25+8=33$ datasets (considering the 25 datasets we used for training and additional 8 for zero-shot transfer) we would need to train $4*33=132$ variants of a baseline model. This is close to the scale of training we used for the DAM, but with the important caveat that baseline methods cannot be trained for longer owing to overfitting, while the DAM benefits from longer training runs by design. Further, these 132 variants do not necessarily transfer well to other datasets.

Good day.

Please note that we have updated the paper as per your commentary and requests. There is a summary entitled "Rebuttal Summary - After addressing reviewers' comments and questions".

Of particular note for your consideration:

1. A considerable extension to the held-out experiments, including an additional 6 datasets (now 8 in total) tested in zero-shot mode, and compared against 3 SoTA methods in zero-shot mode and when trained on the target datasets. We also included performance of the DAM when fine-tuned on target datasets for comprehensive comparisons. The DAM beats SoTA methods at zero-shot transfer across 14/16 metrics and even outperforms SoTA methods trained on the target datasets in some cases.
2. Improved interpretability demonstrations, including a video version submitted as supplementary material.

Please let us know if you have any more questions or requests that we can address before the end of the rebuttal period.

Thank you again for your review.

Thank you for your informative review. Since the submission deadline we have worked to improve the paper and have updated it accordingly. A summary of the changes is provided in another comment. Pertinent to your review, and in summary, please note that the new version of the paper uses a model with more basis coefficients (437 vs 219) for improved performance across all datasets, making it a stronger competitor as a foundational model; has improved figures and visualisations to aid in clarity of understanding; and includes an additional 15 training datasets from various sources. These additional training datasets cover an additional 2 resolutions (30 minutes and 1 minute), and additional domains, namely air quality, stocks, sunspot index, and cloud resources.

We have broken the rest of our response into two parts (this and another), covering the main weakness you mentioned (this) and other weaknesses and questions (in the next comment).

Addressing the main weakness

Our model now uses 25 datasets for training, with 8 held out (including 6 we will add to the paper by the end of the rebuttal deadline). We note that while we could have trained on much larger datasets, many available time-series data have very low signal-to-noise ratios. In our work, we have carefully selected our data to increase the signal in the training and evaluations. In addition, we have used open datasets to improve reproducibility. To the best of our knowledge, we are among very few papers that select datasets from very different domains.

That said, we claim the title of a foundational model because of the novel design of the DAM, not simply because it was trained on many datasets from diverse domains. We believe that the term 'foundational model' does not necessarily only apply to models trained on massive datasets, at huge cost, but that it is the model itself that is foundational by design if it can be used across disciplines. Otherwise, that claim will necessarily be reserved for big companies with high resources. This is particularly true for time series forecasting because designing a foundational model in this field requires more than just scaling up an existing model to more data. Instead, it required a rethinking of the way data was sampled and ingested to yield a flexible, efficient, and global context (i.e., the HSR and TV-tokens), and a non-fixed horizon forecast (i.e., the basis composition) for multiple use-cases.

Dataset-horizon combinations as individual tasks

In some sense, each individual dataset-horizon combination (in Tables 1 and 2) is a separate 'task' when considering how these are traditionally solved by training variants of a single model design. Please consider that the DAM is currently the only SoTA forecasting model that is, by design, agnostic to resolution, missing data, forecast horizon, and data-domain. That is the core novelty of this paper and we believe it is hence a foundational model. Please note that the updated version of the paper includes an additional 15 datasets for training and posits the DAM far more clearly. One of our requirements for using a dataset is that it should be sufficiently long. We are open to suggestions for sourcing additional datasets (aside from the 25 we use for training and the new datasets we used for the held out tests).

Regarding weaknesses

1. We have replaced all visualisations with new and improved digital versions.
2. Regarding inference, we have made an additional comment, titled ‘ A note on inference and the univariate nature of the DAM’ to address this. The DAM is always univariate, inference does involve fitting $\mathbf{\theta}_0$ to the context data. This is because $\mathbf{\theta}_0$ is not part of the model itself, but is instead an input to the model. We will ensure that there is an ‘inference’ section in an upcoming updated version of the paper.
3. Regarding very-long-term forecasting: the notion of ‘long’ is dependent on the resolution of the data. For example, 6000 time steps is roughly 41 days (as in Figure 5) for weather, but is 250 days for an hourly dataset and 16 years for a daily dataset. The practical significance for very-long-term forecasting is therefore reliant on dataset resolution. Testing the same sets of horizons amongst datasets of different resolution is for comparison against existing literature. We chose to demonstrate the very-long-term forecasting for weather because (1) longer term forecasting (e.g., 1008 steps=1 week) is useful for weather, and (2) the high resolution of the weather dataset (10 mins) requires longer term forecasting for practical use. We will clarify this in an incoming update to the paper. Regarding an experiment that uses the 'correct' horizon (+- 6000 steps in this case): it is practically very challenging to train conventional models with this horizon because of the way that they forecast (via a vector the length of the horizon). The DAM, on the other hand, does not require assessing all points between `now' and the distant future, making it elegant for the very-long term case. That all said, we can train PatchTST and DLinear using a very-long horizon of 6000 steps and update the paper by the end of the rebuttal period, should you require that. Please let us know if you still think this is necessary.
4. Thank you for pointing this weakness out; it is a pertinent deduction. We will ensure that this analysis is sufficiently extended. Specifically, we be analysing zero-shot performance across the following 8 datasets: Illness, wiki-weekdays, Monash (including ‘temperature rain’, London smart meters, weather, web traffic), UCI 'individual household electric power consumption', and the Azure 2019 function trace. We will keep the results for illness and wiki-weekdays (as in the current version of the paper), but also include results for PatchTST and DLinear when trained on these datasets. Since this is not strictly a fair comparison (i.e., zero-shot DAM against standard baslines), we will endeavour to make that clear in the revised version of the paper. It is likely that we will be removing Figure 1 in favour of additional space for this extension.

Regarding questions

1. See 1, above.

2. In the aforementioned comment we show how context selection is as simple as sampling indices (np.random.choice(np.arange(len(v)), p=phsr, replace=False)) and is thus extremely lightweight. Your question actually opens up an interesting discussion about how one might alter $p_{hsr}$ according to some intrinsic statistics about the context (which would be more costly) but currently the $p_{hsr}$ is static and independent of the query time series.

3. Hopefully our clarifications above explained the role of the initialisation coefficients, $\mathbf{\theta}_0$ , and how these are fit to context data (even when data is missing; hence being relevant for imputation). Figure 4 shows the basis initialisation for the past, and how well this fits to the context data. It is this that is used for imputation. So, effectively, you are correct: model training is not needed for imputation. Fitting the initial basis coefficients is perhaps an additional contribution of our paper, but there are other more important contributions we chose to focus on.

4. The high basis coefficients outside of the effective ‘visibility’ of the data is interesting and worth considering. One explanation might be that trends of that scale exist in the data even though the collection procedure could not have captured a full cycle of these trends, and that the DAM is capable of approximating those trends to improve forecasting. There are a number of potential alternative explanations, but it should be noted that this seems like a common behaviour across most datasets (Appendix L).

We believe that the combination of the already updated paper, additional writing and clarifications to be made, and additional zero-shot experiments cover the weaknesses and questions you raised. Please kindly let us know if there is anything else you would like seen done in order to cover your concerns or reticence.

Good day.

Please note that we have updated the paper as per your commentary and requests. There is a summary entitled "Rebuttal Summary - After addressing reviewers' comments and questions".

Of particular note for your consideration:

1. More diverse training datasets and extensive evaluations on held-out datasets, with improved performance throughout the paper.
2. Improved visualisations.
3. Description of inference process (Section 3.5).
4. A discussion on the high basis coefficients for ETTh2 (Section 5.1).

Please let us know if you have any more questions or requests that we can address before the end of the rebuttal period.

Thank you again for your review.

Good day,

Please take note we have heeded your and another reviewer's comments (PhNP) and decided to adjust the title and claim of our paper such that it is "towards foundational".

This and many of the changes made throughout this rebuttal period have been in responsive to your thought-provoking review. Thank you.

Do you believe that these changes go even a little way towards addressing some of your concerns? An answer from you, prior to the closing of the rebuttal period in a few hours, would be extremely appreciated. Moreover, if we have satisfactorily addressed your concerns, could you also consider changing your score? We firmly believe that the DAM is good work and deserves attention.

Many thanks once more for all of your hard work in reviewing this paper.

Thank you for your review. Please note that we have uploaded a revised version of the paper and have made a comment describing the changes thus far. In summary, we improved the model by adding more basis functions (437 instead of 219), trained it on 25 datasets (instead of 10), and improved the figures and results in the paper. In the coming days we will be uploading more updated versions of the paper that addresses concerns for all reviewers. That said, we would like to address some of your concerns prior to this, and clarify some points before making the necessary changes to the paper.

Regarding the weaknesses you mentioned:

• The model architecture is actually far closer to the standard transformer architecture (from NLP) than many transformer methods in the literature for forecasting. This is mentioned in the 'Model structure' paragraph in Section 3.1., but we have updated the architecture figure into a better digital format. We chose to be explicit about every component instead of hiding details in collective 'blocks' as is often the case for modern architectures. That said, we can add an additional paragraph in Section 3.1 to address the intended purposes of MHSA, Cross attention, and the feed forward (cross) block that acts across the basis dimension. The MHSA acts to share information across TV-tokens, and thus build a latent context for the model to adjust basis coefficients. The cross attention acts to make use of this latent context to update basis coefficients for prediction. The cross feed-forward block explicitly models patterns across basis coefficients in order to share information in the frequency domain. After seeing the new Figure 2, please confirm whether you think this will be necessary. ~~Running ablation studies is, unfortunately, not feasible within the rebuttal period because the DAM is expensive to train. Should this be an absolute necessity for you to increase your score, please let us know so that we can figure out how to do this in the time available.~~EDIT: we have performed an ablation study and discussed this in the next comment.

• DLinear is a linear forecasting model and a well-known strong baseline. Are you suggesting a less capable linear model should be included for comparison? We endeavoured to include results form SoTA methods to highlight the strength of the DAM, testing 14 methods (including Table 6 in Appendix I), instead of testing it against poorer performing baselines. However, we can certainly test a simpler linear method if you insist, and replace a method in Table 1 with these results.

• Could you please clarify what you mean by a normalized score? We have listed results in the same way that many papers in this field list them. Normalising the results (by whatever mechanism you suggest/clarify) is certainly possible, but this would also require additional space should we wish to keep unnormalised scores for easy comparison to other papers in the field. We prefer the latter, but are open to a discussion about this.

Regarding your question:

• The DAM is always univariate (although we are working on a version that can use information across variables). Inference simply involves sampling time-value pairs from the HSR using the inference time series, fitting initial coefficients to those context points, a forward pass through the DAM, and an assessment of the resultant basis composition for some query time. Figure 10 in Appendix G essentially answers your question regarding the reliance on context size. For the updated model (see the updated paper PDF), a longer context helps in most cases, but this seems to be highly dataset dependent. Thankfully the DAM can be tuned accordingly (HSR tuning) using some held out set (validation in the case of the paper results). It seems that the inference procedure is not immediately obvious (based on question and another), so we will endeavour to make this fully clear in an updated version of the paper.

We have performed an ablation study as per your recommendation. The results averaged across 4 horizons on the ETT datasets are listed below. We will include this table and a discussion in an update to the paper before the rebuttal period closes. Your suggestion was incredibly useful and enabled an insight into the model functionality that was missing - thank you.

Crucial components

Specifically, it is clear that $FF_{B,cross}$ is the most crucial component. This part of the architecture is a weighted connection that acts across B-tokens to update them at each step (i.e., it is not a transformer or attention mechanism). The second most impactful component is the cross-attention, which suggests that the DAM has learnt to extract additional information from TV-tokens that are useful in updating B-tokens, aside from that information that is captured in $\mathbf{\theta}_0$.

The impact of ToME

Finally, and perhaps most interestingly, removing token merging (ToME) is detrimental to performance, suggesting that a concise collection of TV-tokens is advantageous. This behaviour is the inverse of what was observed by the authors of ToME, although the technique was designed for image transformers.

Please note that we were able to perform this ablation study without requiring any additional training. We simply 'switched off' parts of the model to compute these results. This is possible because the latent dimension of the model, $D_{model}$, is consistent throughout the forward pass.

Good day.

Please note that we have updated the paper as per your commentary and requests. There is a summary entitled "Rebuttal Summary - After addressing reviewers' comments and questions".

Of particular note for your consideration:

1. The ablation study (now Section 5.2)
2. Normalised MSE and MAE (Appendix I.1.)
3. Description of inference process (Section 3.5)

Please let us know if you have any more questions or requests that we can address before the end of the rebuttal period.

Thank you again for your review.

Good day,

Please note that we have decided to adopt the term "toward a foundation model" throughout the paper. This is based on reviews and discussions with two other reviewers (wavr and PhNp), but worth pointing out to you. See the recent general comment for details of this.

We believe that we have addressed your concerns (particularly regarding the complex architecture and the new ablation study) and have improved the paper in a number of meaningful ways.

Do you believe that the changes we have made go even a little way towards addressing some of your concerns? An answer from you would be extremely appreciated. Since there is little over 1 hour left of the rebuttal period, we would very much appreciate it if you decided to alter your score at this stage, given the considerable effort we have put in during this period, and your statement of "I am willing to increase my score if the concerns below are addressed". We believe that the DAM is good work and deserves attention.

Many thanks again for all of your hard work and considerations.