Towards Neural Scaling Laws for Time Series Foundation Models

W4. The discussion on sample efficiency is insightful but would benefit from concrete examples or case studies that demonstrate how larger models achieve improved performance with fewer training steps. Including specific examples or empirical data to back up this claim would strengthen the credibility of the findings and provide practical guidance for researchers and practitioners looking to optimize their training processes!

To illustrate the sample efficiency of large models, we have included some examples from existing research to enrich the discussion on sample efficiency in $\underline {\rm Section\ D.2}$.

• In many fields, such as natural language processing (NLP), larger models have demonstrated significant sample efficiency, achieving lower evaluation loss with fewer training tokens [1]. Similarly, in computer vision (CV), larger Vision Transformers (ViTs) typically require fewer training steps to reach competitive accuracy [2].
• From a theoretical perspective, the Neural Tangent Kernel [3] framework provides insights into why this occurs. As neural networks become infinitely wide, their behavior approximates that of a kernel method, enabling them to effectively minimize the data distribution’s generalization error. This allows large models to generalize well with fewer samples, further supporting their observed sample efficiency in empirical settings.

[1] Kaplan J, McCandlish S, Henighan T, et al. Scaling laws for neural language models[J]. arXiv preprint arXiv:2001.08361, 2020.

[2] Zhai X, Kolesnikov A, Houlsby N, et al. Scaling vision transformers[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 12104-12113.

[3] Jacot A, Gabriel F, Hongler C. Neural tangent kernel: Convergence and generalization in neural networks[J]. Advances in neural information processing systems, 2018, 31.

W5. The dataset used for training and evaluation seems carefully chosen but lacks a detailed account of its diversity and representativeness. While the paper highlights the role of dataset size and variety in enhancing OOD performance, it provides limited insights into the dataset's specific characteristics, such as the types of time series it includes, their distributions, or any noise or anomalies present. A deeper examination of the dataset's attributes would offer a clearer picture of the models' performance and their relevance to real-world applications!

Following your suggestion, we have provided specific feature statistics for the pre-training dataset and the test dataset in $\underline {\rm Appendix\ C.2}$, including signal-to-noise ratio, shifting, stationarity, and transition.

W6. Another important limitation is the lack of attention to how scaling laws impact practical applications. The authors offer theoretical insights into scaling behaviour but do not go far enough in showing how these findings can be turned into useful strategies for practitioners. For example, guidance on balancing model size, computational resources, and dataset size in real-world settings would be valuable for those aiming to apply time series foundation models effectively. Including case studies or examples of successful implementations could also help demonstrate the research's relevance to industry professionals!

Thank you for this suggestion. We have revised $\underline {\rm Section\ 4}$ to include more actionable insights derived from our findings, to further guide practitioners in effectively developing time series foundation models.

• Based on Equation 4, we can expect that doubling the size of the pretraining dataset will result in an OOD MAPE reduction to approximately 90% of its previous value, and an ID MAPE reduction to approximately 97%.
• From the ratio of exponents in Equations 2 and 4, we infer that increasing the model size should be accompanied by a sublinear increase in dataset size, roughly following the relationship $D \propto N^{ \frac{\alpha_N}{\alpha_D}} \sim N^{0.8}$.
• As computational budgets grow, our findings suggest that resources should prioritize scaling up model size rather than extending training time for fixed models.
• The observed diminishing returns with extended training on fixed model sizes imply that larger models are increasingly sample efficient and can achieve better performance without requiring significantly longer training times.

We hope our responses provided above can adequately address the reviewer's concerns and questions.

We sincerely appreciate reviewer 5SSN for considering our work is significant, and we greatly appreciate the acknowledgement of our contributions. We have addressed the specific concerns raised by the reviewer as detailed below:

W1: One notable limitation in this study is its minimal exploration of how different model architectures impact scalability. Although the paper briefly acknowledges architecture's role in scaling, it falls short of offering a thorough analysis on how specific architectural choices, like attention mechanism, layer normalization, or residual connections, might influence scaling behaviour. A deeper, comparative study covering a broader set of architectures could reveal which design choices are most beneficial for performance across diverse data distributions!

We appreciate your constructive suggestions. We are conducting experiments to evaluate the impact of specific architectural choices on scalability. In this experiment, we focus on the time series embedding method, as it plays a critical role in determining how patterns are encoded within embedded sequences, and the types of operations that models can effectively learn. The results of this experiment will be included in the camera-ready version of the manuscript.

W2: he study's experiments center on log-likelihood loss and MAPE as key performance metrics. While these are certainly relevant, they do not fully capture the complexity of model performance in time series forecasting. A wider range of evaluation metrics, like the Continuous Ranked Probability Score (CRPS) or quantile loss, could offer a more comprehensive view of performance and its practical impact. Although the paper briefly suggests future research might investigate transformations between log-likelihood and forecasting metrics, including preliminary results or further discussion on this topic within the study would enhance the depth and clarity of the findings!

• Thanks for the valuable suggestion. In response, we have expanded our evaluation to include additional metrics such as CRPS, symmetric MAPE (sMAPE), and MASE (Mean Absolute Scaled Error). The results of these evaluations are provided in $\underline {\rm Appendix\ D.4}$ .
• Time series forecasting involves a diverse set of metrics, each measuring prediction error from a unique perspective. However, due to the lack of clear and definitive correlations between NLL loss and these metrics, it's difficult to understand model prediction behavior directly from NLL loss. Developing an estimation framework to bridge these metrics could greatly help us understand model behavior. To highlight its significance, we have included a detailed discussion on establishing such an estimation framework in $\underline {\rm Appendix\ E}$ .

W3. The paper would benefit from a deeper exploration of why performance tends to degrade on OOD data. Although the authors highlight that scaling the model improves OOD performance, they stop short of examining the mechanisms driving this effect. An in-depth look at how the training data distribution relates to OOD performance could offer valuable insights for practitioners applying these models in a real-world context. For example, investigating domain adaptation techniques or transfer learning strategies might help reduce the performance drop seen in OOD situations!

• OOD performance degradation may come from two reasons: 1) Domain gap: patterns learned during pre training may not translate well to unseen distributions, particularly when the characteristics of OOD datasets differ substantially from those of the training data. 2) Data quality: low-quality or noisy data in the pretraining dataset can negatively impact the model's ability to generalize, either by introducing biases that influence its preferences or by reducing its overall robustness.
• We fully agree that Investigating the impact of the domain gap on OOD performance could provide valuable insights into estimating OOD performance more cost-effectively, without requiring extensive retraining or additional fine-tuning. Additionally, given that pretrained models are often deployed in specific domains, techniques such as domain adaptation and transfer learning are crucial for improving their OOD performance.

Many thanks to Reviewer CUNM for providing the insightful review and comments.

W1. Line 046: provide citations for neural scaling laws that provide ground for believing that they exist.

Thanks for your suggestion. We have included relevant citations in $\underline{\rm Line\ 046}$ to substantiate the existence of neural scaling laws.

W2. Table 1: review the "proportion" row against the "Time points" row. Sales is 0.96% of dataset with 140M points, whereas Web is 0.40% of dataset with 600M points.

Thank you for pointing out this error. We have recomputed the proportion of different domains and corrected this mistake in the revised version.

W3. Line 149: "Given its proven effectiveness in improving time series forecasting performance (Woo et al., 2023), we adopt RoPE as a replacement for the original Transformer’s positional encoding." ==> my reading of Woo (2023) is that the evidence in favor of RoPE for time series forecasting is very weak (results presented without confidence bands, and do not appear statistically significant), and likely that better evidence should be presented than the statement of fact that RoPE is "proven" — I don't see that it is, especially in the context of a foundation model.

Thank you for pointing out this issue. Upon reviewing the experiments in [Woo et al., 2023], we realized that, as you mentioned, RoPE does not significantly improve forecasting performance. Experiments highlight that RoPE achieves lower pretraining loss and faster convergence speed compared to standard sinusoidal positional encoding and learned positional encoding. To correct our reason for choosing RoPE, we have revised the manuscript to state: “Given the improved pretraining efficiency observed with RoPE (Woo et al., 2023), we adopt it as a replacement for the original Transformer’s positional encoding.'"

W4. Line 190: The paper consistently talks about "log likelihood", for which higher values indicate a better fit. However, all results seem to be obtained under the "negative log-likelihood" (NLL, which is the loss counterpart), for which lower values indicate a better fit. This elementary terminology confusion should be clarified corrected.

Apologies for the confusion caused. We have revised the text throughout the manuscript to consistently use the term "negative log-likelihood" (NLL) in all relevant contexts.

W5. Section 3: The paper only studies Transformer-based architectures, without any baseline results with simple time series models (e.g. ETS-type exponential smoothing or ARIMA) — which could be shown as horizontal lines in the scaling law plots. In particular, the MAPE is around or below 1% for most of the studied scaling regime, which suggest that the datasets are very highly predictable: it is likely that simpler models would do well on these datasets too, bringing into question whether TSFMs' additional complexity is warranted.

Thank you for this great suggestion. To address this concern, we have incorporated the performance of ETS method as a baseline reference in the scaling law plot $\underline {\rm Fig\ 2}$.

• The results indicate that the pre-trained models consistently outperform ETS on ID data and progressively excel on OOD data as the model size increases. This suggests that pre-trained models must reach a certain scale, at least 3M parameters in this case, to demonstrate a level of superiority on OOD data that justifies their high pre-training cost.

W6. Appendix C.2: Since the training and test sets are taken from long-available public sources, is there a contamination analysis of the test set against the training set to ensure there's no overlap?

Yes, during the construction of both the pretraining and test datasets, we took care to ensure that there is no overlap between the test set and the training set. The detailed composition of the pretraining and test datasets is provided in $\underline {\rm Appendix\ A\ Table\ 2}$ and $\underline {\rm Appendix\ C\ Table\ 3}$, where we specify the data sources.

Q1. Section 3: The MAPE is consistently used as a metric in the main results, although Appendix C.3 outlines a well-known limitation, which is that it is unreliable when the time series takes values close to zero (and suggests using the sMAPE instead). It seems that for time series foundation models, which should be quite agnostic to shift and scale variations of the series, the MAPE should be the last metric to be considered. Why has not the sMAPE been used, or more robust scale-invariant methods like the MASE?

To address your concern about the reliability of MAPE, we have included evaluations of the scaling laws using symmetric MAPE (sMAPE), and MASE as an additional metric in the revised manuscript. The scaling behavior of sMAPE and MASE closely matches our observations on MAPE. These results are presented in $\underline {\rm Appendix\ D.4}$ .

Q2. Figures 2-4: Although the NLL difference between ID and OOD is of the expected sign (i.e. with OOD having greater NLL than ID), it is truly mystifying that the opposite is consistently true for the MAPE. This is not properly addressed by the authors. In my view, this is the main issue that is causing my "Soundness" assessment not to be higher.

• This inconsistency arises from the nature of the predicted mixture distribution. NLL measures the probability of the ground truth, while MAPE reflects the proportional distance between the predicted value (i.e., the expected value of the predicted distribution) and the ground truth. For a mixture distribution (e.g. a bimodal distribution), high-probability events may not align closely with the expected value, leading to cases where MAPE is small but NLL is large. Overall, NLL is not always positively correlated with MAPE when we use a mixture distribution to model real distribution.
• In Figures 2-4, OOD results show greater NLL but smaller MAPE compared to ID results. This suggests that the predicted OOD mixture distributions have lower probability density around the expected value, while the ID mixture distributions exhibit higher probability density near the expected value.

Q3. Figure 5: Although encoder-only clear beat decoder-only in distribution, this difference is almost insignificant out of distribution, in contrast to the claim in the caption. If the authors intend to make this claim, they should conduct a formal statistical test of the fit difference between encoder-only and decoder-only models.

• Sorry for the misleading, our intention is to highlight that model architecture influences scalability, rather than to claim that encoder-only models outperform decoder-only models. To avoid any misleading implications, we have revised the wording in $\underline {\rm Section\ 3.2}$ to clarify that the observed results “ Overall, the two architectures exhibit similar scalability, aside from a performance difference in ID data.”
• We fully agree that a formal statistical test is essential for a rigorous comparison of encoder-only and decoder-only architectures. Performance differences can be sensitive to several factors including history length, prediction length, and training strategies. Given these sensitivities, our results under the current evaluation setting suggest only a slight superiority of encoder-only models. This observation may not generalize to all scenarios.

Q4. Suggestion related to Figure 8: across these examples, it'd be instructive to plot (in appendix) what the actual forecasts look like as we transition from a regime of "relatively higher MAPE" to "relatively lower MAPE".

Following the reviewer's suggestion, we have added the intuitive forecasting results to demonsate "emergent phenomenon" in $\underline {\rm Appendix\ D.1}$.

We hope our responses provided above can adequately address the reviewer's concerns and questions.

Dear Reviewer CUNM,

Since the End of author/reviewer discussions is coming soon, we would like to know whether our response addressed your concerns. If so, we kindly request your reconsideration of the score. We eagerly await your feedback and are ready to respond to any further questions you may have.

Thank you so much for devoting time to improving our paper !

Dear Reviewer,

This is a kind reminder that three days remain until the end of the discussion. We kindly ask if our responses have addressed your concerns. Following your suggestions, we have made the following updates:

• Added the relevant citation in $\underline{\rm Line\ 46}$ to substantiate the existence of neural scaling laws.
• Corrected errors in $\underline{\rm Table\ 1}$ regarding the proportions of domains.
• Revised terminology throughout to consistently use "negative log-likelihood" (NLL) instead of "log-likelihood."
• Adjusted the justification for selecting RoPE in $\underline{\rm Line\ 154}$, emphasizing its improved pretraining efficiency rather than its performance in forecasting.
• Revised the conclusion in $\underline{\rm Section\ 3.2}$ regarding encoder-only vs. decoder-only models to ensure a more balanced and rigorous discussion.

Additionally, we have included more experiments and results:

• Incorporated ETS as a baseline for comparison in the scaling law plot ($\underline{\rm Fig.\ 2}$).
• Evaluated scaling laws using sMAPE, MASE, and CRPS, detailed in $\underline{\rm Appendix\ D.4}$.
• Provided intuitive forecasting results in $\underline{\rm Appendix\ D.1}$ to illustrate emergent abilities.

We sincerely hope these revisions address your main concerns. All updates have been incorporated into the $\underline{\rm revised\ paper}$ for your review.

Thank you again for your valuable feedback and dedication. We look forward to hearing your thoughts on our revisions.

Dear Reviewer CUNM,

Since only one day remaining until the discussion deadline, could you kindly confirm if our responses have addressed your concerns? Your further feedback is important to us, and we would be happy to address any follow-up questions or concerns you may have.

Thank you again for your constructive review !

Q5. "Scaling-laws for Large Time-series Models" suggests that model size has a big impact on the optimal learning rate. What learning rate did you use and how did you change it for different model sizes? Are the observations consistent for encoder-only and decoder-only models?

• For all model sizes, we use the same learning rate schedule: a maximum learning rate of 1e-3, with a linear warm-up for the first 10,000 steps, followed by cosine decay for the remaining 90,000 steps. To clarify our experimental details, we supplemented the information of learning rate in $\underline{\rm Section\ 2.3\ line\ 86}$.
• This choice of learning rate is based on the setup of time series foundation models Moirai and Chronos. Additionally, the specific results in "Scaling-laws for Large Time-series Models" also show that the optimal learning rate generally falls within the range of 1e-3 to 1e-2. Our setting is robust and good enough to make all the models converge well. The loss convergence process during training is provided in $\underline{ \rm Appendix\ C.1}$.
• We did not specifically investigate the impact of learning rate on encoder-only versus decoder-only models, as our focus was on maintaining a fair and unified experimental setup. However, we believe that exploring the relationship between optimal learning rates and model architecture is a valuable avenue for future research.

Minor

• "Lotsa" should be capitalized on line 94.
• "Sum" in Table 1 is a bit confusing. Consider replacing it with "Total" and use a vertical line to separate it from previous columns.
• In Figure 2 and 3, consider adding ID and OOD to the legend, in addition to using the colors.
• Line 794: "and Covid19_Energy datasets", the underscore here seems to be a typo?

We appreciate your careful reading. Following your suggestion, we have addressed these issues in the revised version.

• Scaling-laws for Large Time-series Models" suggests that using a separate head for each distribution parameter should stabilize training.

As same as "Scaling-laws for Large Time-series Models", we also adopt a separate head for each distribution parameter. To clarify this point, we have added more details in line $\underline {\rm Section\ 2.2\ line\ 75}$.

• Could you add an experiment to show the benefit of using student-T mixture versus gaussian mixture?

We have conducted experiments to compare the performance of Student-T mixture versus Gaussian mixture. The results are presented in $\underline{\rm Appendix\ B }$. Our results indicate that Gaussian mixture not only exhibits unstable convergence but also underperforms compared to Student-T mixture. This demonstrates that long-tail distributions are prevalent in real-world time series data, highlighting the critical importance of Student-T mixtures in effectively modeling these distributions.

We hope our responses above can adequately address the reviewer's concerns and questions.

Q2. Why does OOD have a larger exponent for MAPE than ID in Figure 2 and 3, but not for log-likelihood? It feels to me that it might be due to the type of data distribution you have.

• The faster decrease of OOD MAPE compared to ID MAPE is due to the larger reduction in the proportional error relative to the ground truth for OOD data . In addition, our statistics show that the average magnitude of the OOD data is 1198 while the average magnitude of the ID data is 1588. MAPE is more sensitive to performance improvements for data with relatively small magnitudes .
• NLL, on the other hand, depends on the absolute error and the parameters of the predicted distribution, such as variance and freedom degree. While both MAPE and NLL are functions of the absolute error, NLL is scale-invariant and is instead governed by the predicted distribution's characteristics. There is no definite correlation between the scaling behavior of these two metrics. That is why NLL does not exhibit the same trend as MAPE.

Q3. In Timer, the findings are decoder-only scales better than encoder-only. Could you explain why your finding is different? In Timer, the batch size is chosen to be 8192, but you only used a batch size of 128. Is it possible that using a larger batch size would change your observations and make the decoder-only model more favorable?

The difference between our findings and Timer’s conclusion stems from our different focus and experimental setups.

• Timer focuses on model transferability evaluation. It compares the performance of fine-tuned decoder-only and encoder-only models on OOD data, showing that the decoder-only model has better transferability. However, this does not directly indicate that the decoder-only model scales better than the encoder-only model.
• Our research focuses on zero-shot forecasting capability. Our experiments indicate that, for zero-shot forecasting on ID data, the encoder-only model has slightly better performance and scalability. Actually, our findings and Timer’s conclusion are not in conflict, as some studies [1] [2] have suggested that fine-tuned performance does not strongly correlate with the zero-shot performance.

[1] Lin H, et al. Selecting large language model to fine-tune via rectified scaling law[J]. arXiv preprint arXiv:2402.02314, 2024.

[2] Deshpande A,et al. A linearized framework and a new benchmark for model selection for fine-tuning[J]. arXiv preprint arXiv:2102.00084, 2021.

To investigate the influence of batch size on decoder-only models, we conducted additional experiments using batch size of 512 while training a 100M decoder-only Transformer.

• We observed that as the batch size increases, decoder-only Transformer converges to a lower level and shows a slightly lower NLL and MAPE in ID data. This suggests that bigger batch size is a better choice for releasing the potential of decoder-only models. We can infer that observations on the scalability of encoder-only and decoder-only models may vary with the degree of performance release. The decoder-only models may exhibit a higher upper bound when the computational cost continues to rise.
• In this paper, our focus is primarily on maintaining a fair and unified experimental setup for investigating the influence of model architecture on scalability. In the future, a formal statistical test will be necessary for rigorously comparing encoder-only and decoder-only architectures.

Q4. Did you try retraining MOIRAI/Timer/Chronos using your filtered dataset? The dataset mixture is often a very important factor. Not using the same pretraining datasets can be an unfair comparison in Section 3.2.

• Yes. We used the same pre-training dataset and mixture strategy across all models to ensure a fair comparison. To clarify our experimental details, we have included an introdcution of this setting in $\underline {\rm the\ first\ paragraph\ of\ Section\ 3.2 }$.

We thank the reviewer for offering the valuable feedback. We have addressed each of the concerns raised by the reviewer as outlined below.

W1. Evaluation setup could be improved. On line 130, "This subset includes test data from the ETTh1-2, ETTm1-2, electricity, and weather datasets." Are these the only datasets that are used as OOD test data throughout all the experiments? To strengthen your findings, I suggest varying the OOD datasets (maybe include a confidence interval), and see if your findings still hold.

• Our evaluation of OOD performance is not limited to the LSF subset ( ETTh1-2, ETTm1-2, electricity, and weather datasets). We also conducted assessments on a subset of the Monash dataset. The results from the Monash subset align with our observations from the LSF subset, confirming that OOD performance follows a power law relationship with model size, compute, and training data size.
• Following the reviewer’s suggestion, we have included the experiment results of Monash subset in $\underline{\rm Appendix \ D.3}$ and repeated the experiments five times to draw confidence intervals , ensuring the reliability of our findings.

W2. The paper could benefit from explaining why its findings are different from those of the existing works.

• Prior research has not explored the scaling laws of time series foundation models in OOD scenarios. We address the gap and show that the power law trend persists in OOD time series forecasting .
• Previous work has not examined scalability differences across model architectures. Our experiments indicate that the scalability slightly varies between encoder-only and decoder-only architectures, while encoder-only models shows superior scalability in ID data.
• Prior research has not focused specifically on the scalability of SOTA models. Our findings reveal that while improvements in model design enhance performance on ID data, these gains do not generalize effectively to OOD data.

To better highlight these distinctions, we have revised the summaries of our contributions in the updated version.

Q1.1. For the training details on line 183, I am not sure if I understood correctly. If one dataset is much larger than the other datasets, are we much more likely to sample from this dataset? Is there a cap to this likelihood? In existing works such as MOIRAI and Timer, they apply some threshold that prevents a large dataset from dominating from the training distribution.

• Yes, we assign a sampling probability to each dataset based on the proportion of its time points relative to the total. Consequently, datasets with more time points are more likely to be sampled during training.
• Meantime, we follow the approach used inMoirai and Timer by capping the sampling probability at 0.05 to ensure a more balanced contribution from each dataset.

To clarify the sampling method further, we have added the details of our sampling method to $\underline{ \rm Section\ 2.3\ line\ 190}$ in the revised manuscript.

Q1.2. In addition to "Domain Balancing" as mentioned in Appendix A in your paper, I think balancing the datasets within each domain should also be helpful. Could you compare the different data mixing strategies?

• We agree that balancing datasets within each domain can help mitigate capability bias and improve the performance of time series foundation models. However, the primary focus of our research is to investigate scaling laws, where maintaining a fair and unified experimental setup is crucial. Therefore, we adhere to the experimental settings established in [1] and [2], ensuring consistent pre-training datasets across evaluations.
• Given that time series pre-training datasets typically include data from over ten or even one hundred sources, the potential combinations for dataset mixing are exponentially large. Thus, it is more feasible to develop an extrapolable analytical framework under controlled experimental complexity [3][4]. Due to resource and time constraints, we put our focus on our primary objectives by using the same settings in [1][2]. We add further discussion on data mixing strategies in the revised manuscript $\underline{ \rm Appendix\ E}$. We hope to explore this topic more comprehensively in future work.

[1] Edwards T D P, et al. Scaling-laws for Large Time-series Models[J]. arXiv preprint arXiv:2405.13867, 2024.

[2] Kaplan J, et al. Scaling laws for neural language models[J]. arXiv preprint arXiv:2001.08361, 2020.

[3] Hashimoto T. Model performance scaling with multiple data sources[C]//International Conference on Machine Learning. PMLR, 2021: 4107-4116.

[4] Ye J, et al. Data mixing laws: Optimizing data mixtures by predicting language modeling performance[J]. arXiv preprint arXiv:2403.16952, 2024

Dear Reviewer oPfC,

Since the End of author/reviewer discussions is coming soon, may we know if our response addresses your main concerns? If so, we kindly ask for your reconsideration of the score. Should you have any further advice on the paper and our rebuttal, please let us know and we will be more than happy to engage in more discussion and paper improvements.

Thank you so much for devoting time to improving our paper!

W4. Absence of computational efficiency analysis: While the paper considers compute budgets as a scaling factor, it does not provide a detailed analysis of the computational efficiency of the investigated models. As TSFMs scale up in size, computational efficiency becomes increasingly important. The paper could be strengthened by including metrics such as training time, inference time, and memory consumption for the different model architectures and sizes. This would provide valuable insights into the practical trade-offs between performance and computational cost.

Q4. Computational efficiency considerations: While the paper considers compute budgets as a scaling factor, it does not provide a detailed analysis of the computational efficiency of the investigated models. Can the authors comment on the practical trade-offs between performance and computational cost for the different model architectures and sizes? Including metrics such as training time, inference time, and memory consumption could provide valuable insights for practitioners looking to deploy TSFMs in resource-constrained environments.

• In figure 3, we follow the setting in [1][2] and focus on FLOPs as the metric for computational cost, as it provides a more device-independent measure of model complexity compared to training time, inference time, and memory consumption. The latter three metrics can vary significantly depending on the hardware, training strategy, and batch size.
• For training time and inference time, these can be estimated using the amount of FLOPs and the type of hardware. Memory consumption depends more on batch size, sequence length and the distributed training techniques like Fully Sharded Data Parallel (FSDP) or Data Parallel (DDP).
• In the trade-off between performance and computational cost, our results show that within a fixed computational budget, scaling up model size tends to be more beneficial for improving performance compared to spending the same amount of compute time on training a smaller model for a longer time.

[1] Kaplan J, McCandlish S, Henighan T, et al. Scaling laws for neural language models[J]. arXiv preprint arXiv:2001.08361, 2020.

[2] Edwards T D P, Alvey J, Alsing J, et al. Scaling-laws for Large Time-series Models[J]. arXiv preprint arXiv:2405.13867, 2024.

Q5. Generalizability to multivariate time series: The paper focuses exclusively on univariate time series forecasting to avoid the confounding effects introduced by multivariate time series. Can the authors discuss the potential challenges and opportunities in extending their findings to multivariate TSFMs? What additional factors (e.g., variable interactions, correlations) might need to be considered when investigating scaling laws in the multivariate setting?

Thank you for raising this important question. Extending our findings to multivariate time series forecasting still faces some challenges, primarily in three key areas:

Multivariate Pre-training Dataset Availability: Large, diverse multivariate datasets are limited, unlike the more standardized univariate time series datasets. This scarcity makes it difficult to conduct experiments that scale both model size and data volume in a controlled manner.

Variable Correlations and Interactions: As you mentioned, correlations and interactions between variables are key factors in multivariate time series modeling. These relationships must be captured explicitly to ensure model performance and scalability. When investigating the scaling laws of multivariate TSFMs, we need to independently investigate the effects of the number of variables and the strength of correlations on forecasting performance. This adds another variable dimension to the experiment and makes the analysis more complex compared to a univariate setup.

Multivariate Modeling Approaches: Compared to univariate models, multivariate time series foundation models are still at an early stage, and face the challenges in designing architectures that can flexibly handle varying numbers of variables and capturing complex inter-variable dependencies.

In the revised version, we have added more discussion on scaling laws for multivariate time series forecasting in $\underline {\rm Appendix\ E}$ .

We hope our responses provided above can adequately address the reviewer's concerns and questions.

W2. Lack of theoretical analysis: The paper primarily relies on empirical observations to establish the scaling laws for TSFMs. While the experimental results are valuable, the work could be enhanced by providing a theoretical foundation for the observed scaling behaviors. Theoretical analysis could help explain why certain scaling patterns emerge and offer insights into the underlying mechanisms that govern the relationship between model size, compute, data, and performance. For example, the authors could explore the connection between the power-law scaling and the model's capacity to learn and generalize from the training data.

Q2. Potential for theoretical analysis: The paper primarily relies on empirical observations to establish the scaling laws for TSFMs. Have the authors considered complementing their empirical findings with a theoretical analysis? If so, what challenges do they anticipate in developing a theoretical framework for the observed scaling behaviors, and how might such a framework enhance our understanding of the underlying mechanisms governing the relationship between model size, compute, data, and performance?

Our scaling laws, similar to the scaling laws for natural language, are empirical findings. We believe a theoretical understanding of the training dynamics that form the laws provides a more solid justification. A potential perspective is understanding the optimization process through dynamics modeling. Specifically, based on the dynamical mean field theory [1], we can derive the infinite limit statistical description of the scaling behavior of training factors. By analyzing the response function in the infinite limit, which gauges the correlation between training factors and test error, we can obtain insights into how the scaling laws manifest. In future work, we will strengthen the research on theoretical framework. And we have added more discussion on the theory for scaling laws in $\underline {\rm Appendix\ E}$.

W3. Limited discussion on the trade-offs between ID and OOD performance: The paper highlights that architectural modifications in Moirai and Chronos improve ID performance but compromise OOD scalability. However, the discussion on this trade-off is relatively brief. A more in-depth analysis of the factors contributing to this trade-off and the potential implications for TSFM design would be valuable. For instance, the authors could investigate whether certain architectural choices (e.g., attention mechanisms, embedding strategies) are more prone to overfitting on ID data, leading to reduced OOD scalability.

Q3. Factors contributing to the ID-OOD performance trade-off: The paper highlights that architectural modifications in Moirai and Chronos improve ID performance but compromise OOD scalability. Can the authors provide more insights into the factors that might contribute to this trade-off? For example, are there specific architectural choices (e.g., attention mechanisms, embedding strategies) that could be more prone to overfitting on ID data, leading to reduced OOD scalability? Understanding these factors could help guide the design of future TSFMs.

Thanks for your constructive feedback. Following the reviewer’s suggestion, we are conducting experiments to evaluate the impact of specific architectural choices on OOD scalability. In this experiment, we focus on the time series embedding layer, as it plays a critical role in determining how patterns are encoded within embedded sequences, and the types of operations that models can effectively learn. The results of this experiment will be included in the camera-ready version of the manuscript.

We express our gratitude to the reviewer for providing constructive feedback on our paper, and we greatly appreciate the acknowledgement of our contributions. We have addressed the specific concerns raised by the reviewer as detailed below.

W1. Limited scope of model architectures: The study focuses on two main model architectures: encoder-only and decoder-only Transformers. While these are indeed widely used in TSFMs, the paper could benefit from including a broader range of architectures, such as encoder-decoder models (e.g., Seq2Seq) or hybrid models that combine Transformers with other neural network components (e.g., CNN, RNN). Expanding the scope of the investigated architectures would provide a more comprehensive understanding of how architectural choices affect scalability.

Q1. Justification for the chosen model architectures: Can the authors provide a more detailed explanation for why they selected encoder-only and decoder-only Transformers as the main focus of their study? While these architectures are indeed popular in TSFMs, it would be helpful to understand the rationale behind this choice and whether the authors believe their findings would generalize to other architectures, such as encoder-decoder models or hybrid models.

Most Time Series Foundation Models (TSFMs), including Timer, TimesFM, Moirai, Chronos, Lag-Llama, and Moment, are built upon Transformer architectures. Our primary objective in this study is to explore the scaling laws for these TSFMs, which motivates our selection of encoder-only and decoder-only architectures. Additionally, the below two reasons are also important:

• Scalability: TSFMs require architectures that can effectively leverage large-scale pretraining, making scalability a critical consideration. Previous study [1] has shown that Transformer-based architectures outperform MLPs and CNNs in scalability for time series modeling. Additionally, the success of large language models (LLMs) has demonstrated that Transformers can scale to billions of parameters, further emphasizing their suitability as the backbone for time series models.
• Forecasting Paradigms: Transformer-based time series models primarily adopt two forecasting paradigms: 1) One-forward masked prediction: Typically implemented with encoder-only Transformers. 2) Auto-regressive next-token prediction: Represented by decoder-only Transformers (Encoder-decoder Transformer also adopts this forecasting paradigm but uses a independent encoder to encode context). Encoder-only and decoder-only architectures thus serve as the canonical representatives for these widely adopted methods in current TSFMs .

Our research comparing the scalability of encoder-only and decoder-only Transformers can serve as the groundwork for future exploration of other architectures. CNN-based models [2], RNN-based models [3], and hybrid models [4] each demonstrate unique strengths in sequence modeling. Investigating their scalability within the context of time series modeling is a promising direction for future research, and we hope our findings will inspire further work in this area. We have added more discussion on the impact of architectures on scalability in $\underline{ \rm Appendix\ E}$.

[1] Liu Y, et al. Timer: Generative Pre-trained Transformers Are Large Time Series Models[C]//Forty-first International Conference on Machine Learning.

[2] Wu H, et al. Timesnet: Temporal 2d-variation modeling for general time series analysis[J]. arXiv preprint arXiv:2210.02186, 2022.

[3] Gu A, et al. Efficiently modeling long sequences with structured state spaces[J]. arXiv preprint arXiv:2111.00396, 2021.

[4] Lieber O, et al. Jamba: A hybrid transformer-mamba language model[J]. arXiv preprint arXiv:2403.19887, 2024.

Dear Reviewer,

Thanks for your valuable review, which has inspired us to improve our paper further. This is a kind reminder that the end of author/reviewer discussions is coming soon. We kindly ask if our responses have addressed your concerns. We look forward to hearing your thoughts on our revisions.

Thank you so much for devoting time to improving our paper!